2024-03-19T02:41:42.906 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-19T02:41:42.907 DEBUG:teuthology.run:Teuthology command: teuthology --name pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi --verbose --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} --archive /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609460 --owner scheduled_pdonnell@teuthology -- /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609460/orig.config.yaml 2024-03-19T02:41:42.959 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-19T02:41:43.054 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609460 branch: wip-batrick-testing-20240318.181317 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} email: pdonnell@redhat.com first_in_suite: false job_id: '7609460' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240318.181317 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - PG_AVAILABILITY - PG_DEGRADED - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.dir_split install: ceph: flavor: default sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240318.181317 sha1: e48b1f6c0608b2615e1890cc46286c3877d6e910 owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3005 sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 sleep_before_teardown: 0 subset: 433/512 suite: fs suite_branch: wip-batrick-testing-20240318.181317 suite_path: /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: e48b1f6c0608b2615e1890cc46286c3877d6e910 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi117.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJxM4ZQBmQKCjhr6XygHCic/wYzpzLj5NY6vPMZ159xdqkwGnJimG6q1LiQvc24Zm2XuqsGiqWQyEzk0c6baW/A= smithi119.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCKrJY4oZavoFwutvIcgDonTGOpMinb+rD5M9GAzejj7pUNZs7ElO/bXEtR/QMgOFupenJhSmcRDkF7Q3n/9TPw= smithi178.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCb7+ZVOTQgtPYhzbO/uemTtpEbApH0EnJo1LXCybwiksC0fg3jmBvjKY3oIJnGw9Gb9vS5VMe60EsmO0o5tEYQ= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - workunit: clients: all: - suites/blogbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-18_21:05:49 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-19T02:41:43.054 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa; will attempt to use it 2024-03-19T02:41:43.071 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks 2024-03-19T02:41:43.072 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-19T02:41:43.073 INFO:teuthology.task.internal:Checking packages... 2024-03-19T02:41:43.097 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b6ae24918b03b1b8a19c9263a50655bba80bb810' 2024-03-19T02:41:43.097 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-19T02:41:43.097 INFO:teuthology.packaging:ref: None 2024-03-19T02:41:43.097 INFO:teuthology.packaging:tag: None 2024-03-19T02:41:43.097 INFO:teuthology.packaging:branch: wip-batrick-testing-20240318.181317 2024-03-19T02:41:43.097 INFO:teuthology.packaging:sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T02:41:43.097 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240318.181317 2024-03-19T02:41:43.244 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2206.gb6ae2491 2024-03-19T02:41:43.245 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-19T02:41:43.251 INFO:teuthology.task.internal:no buildpackages task found 2024-03-19T02:41:43.251 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-19T02:41:43.261 INFO:teuthology.task.internal:Saving configuration 2024-03-19T02:41:43.285 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-19T02:41:43.293 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T02:41:43.318 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609460', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 02:36:25.658961', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJxM4ZQBmQKCjhr6XygHCic/wYzpzLj5NY6vPMZ159xdqkwGnJimG6q1LiQvc24Zm2XuqsGiqWQyEzk0c6baW/A='} 2024-03-19T02:41:43.342 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609460', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 02:36:25.657241', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCKrJY4oZavoFwutvIcgDonTGOpMinb+rD5M9GAzejj7pUNZs7ElO/bXEtR/QMgOFupenJhSmcRDkF7Q3n/9TPw='} 2024-03-19T02:41:43.364 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609460', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 02:36:25.660491', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCb7+ZVOTQgtPYhzbO/uemTtpEbApH0EnJo1LXCybwiksC0fg3jmBvjKY3oIJnGw9Gb9vS5VMe60EsmO0o5tEYQ='} 2024-03-19T02:41:43.364 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-19T02:41:43.372 INFO:teuthology.task.internal:roles: ubuntu@smithi117.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-19T02:41:43.372 INFO:teuthology.task.internal:roles: ubuntu@smithi119.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-19T02:41:43.372 INFO:teuthology.task.internal:roles: ubuntu@smithi178.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-19T02:41:43.372 INFO:teuthology.run_tasks:Running task console_log... 2024-03-19T02:41:43.485 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5da617c160>, signals=[15]) 2024-03-19T02:41:43.485 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-19T02:41:43.492 INFO:teuthology.task.internal:Opening connections... 2024-03-19T02:41:43.493 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi117.front.sepia.ceph.com 2024-03-19T02:41:43.494 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:41:43.585 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi119.front.sepia.ceph.com 2024-03-19T02:41:43.586 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:41:43.677 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi178.front.sepia.ceph.com 2024-03-19T02:41:43.678 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:41:43.767 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-19T02:41:43.774 DEBUG:teuthology.orchestra.run.smithi117:> uname -m 2024-03-19T02:41:43.792 INFO:teuthology.orchestra.run.smithi117.stdout:x86_64 2024-03-19T02:41:43.792 DEBUG:teuthology.orchestra.run.smithi117:> cat /etc/os-release 2024-03-19T02:41:43.849 INFO:teuthology.orchestra.run.smithi117.stdout:NAME="CentOS Stream" 2024-03-19T02:41:43.849 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION="9" 2024-03-19T02:41:43.849 INFO:teuthology.orchestra.run.smithi117.stdout:ID="centos" 2024-03-19T02:41:43.849 INFO:teuthology.orchestra.run.smithi117.stdout:ID_LIKE="rhel fedora" 2024-03-19T02:41:43.849 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION_ID="9" 2024-03-19T02:41:43.849 INFO:teuthology.orchestra.run.smithi117.stdout:PLATFORM_ID="platform:el9" 2024-03-19T02:41:43.849 INFO:teuthology.orchestra.run.smithi117.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-19T02:41:43.850 INFO:teuthology.orchestra.run.smithi117.stdout:ANSI_COLOR="0;31" 2024-03-19T02:41:43.850 INFO:teuthology.orchestra.run.smithi117.stdout:LOGO="fedora-logo-icon" 2024-03-19T02:41:43.850 INFO:teuthology.orchestra.run.smithi117.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-19T02:41:43.850 INFO:teuthology.orchestra.run.smithi117.stdout:HOME_URL="https://centos.org/" 2024-03-19T02:41:43.850 INFO:teuthology.orchestra.run.smithi117.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-19T02:41:43.850 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-19T02:41:43.850 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-19T02:41:43.851 INFO:teuthology.lock.ops:Updating smithi117.front.sepia.ceph.com on lock server 2024-03-19T02:41:43.872 DEBUG:teuthology.orchestra.run.smithi119:> uname -m 2024-03-19T02:41:43.890 INFO:teuthology.orchestra.run.smithi119.stdout:x86_64 2024-03-19T02:41:43.891 DEBUG:teuthology.orchestra.run.smithi119:> cat /etc/os-release 2024-03-19T02:41:43.948 INFO:teuthology.orchestra.run.smithi119.stdout:NAME="CentOS Stream" 2024-03-19T02:41:43.948 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION="9" 2024-03-19T02:41:43.948 INFO:teuthology.orchestra.run.smithi119.stdout:ID="centos" 2024-03-19T02:41:43.948 INFO:teuthology.orchestra.run.smithi119.stdout:ID_LIKE="rhel fedora" 2024-03-19T02:41:43.948 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_ID="9" 2024-03-19T02:41:43.948 INFO:teuthology.orchestra.run.smithi119.stdout:PLATFORM_ID="platform:el9" 2024-03-19T02:41:43.948 INFO:teuthology.orchestra.run.smithi119.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-19T02:41:43.948 INFO:teuthology.orchestra.run.smithi119.stdout:ANSI_COLOR="0;31" 2024-03-19T02:41:43.948 INFO:teuthology.orchestra.run.smithi119.stdout:LOGO="fedora-logo-icon" 2024-03-19T02:41:43.948 INFO:teuthology.orchestra.run.smithi119.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-19T02:41:43.948 INFO:teuthology.orchestra.run.smithi119.stdout:HOME_URL="https://centos.org/" 2024-03-19T02:41:43.948 INFO:teuthology.orchestra.run.smithi119.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-19T02:41:43.949 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-19T02:41:43.949 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-19T02:41:43.949 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server 2024-03-19T02:41:43.972 DEBUG:teuthology.orchestra.run.smithi178:> uname -m 2024-03-19T02:41:43.990 INFO:teuthology.orchestra.run.smithi178.stdout:x86_64 2024-03-19T02:41:43.990 DEBUG:teuthology.orchestra.run.smithi178:> cat /etc/os-release 2024-03-19T02:41:44.048 INFO:teuthology.orchestra.run.smithi178.stdout:NAME="CentOS Stream" 2024-03-19T02:41:44.048 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION="9" 2024-03-19T02:41:44.048 INFO:teuthology.orchestra.run.smithi178.stdout:ID="centos" 2024-03-19T02:41:44.048 INFO:teuthology.orchestra.run.smithi178.stdout:ID_LIKE="rhel fedora" 2024-03-19T02:41:44.048 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION_ID="9" 2024-03-19T02:41:44.048 INFO:teuthology.orchestra.run.smithi178.stdout:PLATFORM_ID="platform:el9" 2024-03-19T02:41:44.048 INFO:teuthology.orchestra.run.smithi178.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-19T02:41:44.048 INFO:teuthology.orchestra.run.smithi178.stdout:ANSI_COLOR="0;31" 2024-03-19T02:41:44.048 INFO:teuthology.orchestra.run.smithi178.stdout:LOGO="fedora-logo-icon" 2024-03-19T02:41:44.048 INFO:teuthology.orchestra.run.smithi178.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-19T02:41:44.048 INFO:teuthology.orchestra.run.smithi178.stdout:HOME_URL="https://centos.org/" 2024-03-19T02:41:44.049 INFO:teuthology.orchestra.run.smithi178.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-19T02:41:44.049 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-19T02:41:44.049 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-19T02:41:44.049 INFO:teuthology.lock.ops:Updating smithi178.front.sepia.ceph.com on lock server 2024-03-19T02:41:44.072 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-19T02:41:44.083 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-19T02:41:44.094 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-19T02:41:44.094 DEBUG:teuthology.orchestra.run.smithi117:> test '!' -e /home/ubuntu/cephtest 2024-03-19T02:41:44.096 DEBUG:teuthology.orchestra.run.smithi119:> test '!' -e /home/ubuntu/cephtest 2024-03-19T02:41:44.098 DEBUG:teuthology.orchestra.run.smithi178:> test '!' -e /home/ubuntu/cephtest 2024-03-19T02:41:44.114 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-19T02:41:44.124 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-19T02:41:44.125 DEBUG:teuthology.orchestra.run.smithi117:> test -z $(ls -A /var/lib/ceph) 2024-03-19T02:41:44.153 DEBUG:teuthology.orchestra.run.smithi119:> test -z $(ls -A /var/lib/ceph) 2024-03-19T02:41:44.156 DEBUG:teuthology.orchestra.run.smithi178:> test -z $(ls -A /var/lib/ceph) 2024-03-19T02:41:44.262 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-19T02:41:44.341 INFO:teuthology.run_tasks:Running task kernel... 2024-03-19T02:41:44.361 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-03-19T02:41:44.361 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-19T02:41:44.361 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-19T02:41:44.362 DEBUG:teuthology.orchestra.run.smithi117:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-19T02:41:44.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T02:41:44.379 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-19T02:41:44.615 DEBUG:teuthology.task.kernel:sha1 for client.0 is 908b6b276e8f14e16032b32a9770077cd37b6bd4 2024-03-19T02:41:44.615 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "908b6b276e8f14e16032b32a9770077cd37b6bd4"... 2024-03-19T02:41:44.615 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-03-19T02:41:44.633 INFO:teuthology.orchestra.run.smithi117.stdout:5.14.0-361.el9.x86_64 2024-03-19T02:41:44.633 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 908b6b276e8f14e16032b32a9770077cd37b6bd4 2024-03-19T02:41:44.633 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-19T02:41:44.634 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum upgrade -y linux-firmware 2024-03-19T02:41:46.819 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-19T02:41:48.445 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-03-19T02:41:49.443 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-19T02:41:51.144 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-03-19T02:46:25.854 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - CRB 23 kB/s | 6.1 MB 04:31 2024-03-19T02:46:27.604 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 16 kB 00:00 2024-03-19T02:50:00.430 INFO:teuthology.orchestra.run.smithi117.stdout:Extra Packages for Enterprise Linux 99 kB/s | 21 MB 03:32 2024-03-19T02:50:05.287 INFO:teuthology.orchestra.run.smithi117.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-19T02:50:07.483 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-19T02:50:07.493 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-19T02:50:07.493 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-19T02:50:07.493 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-19T02:50:07.493 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrading: 2024-03-19T02:50:07.493 INFO:teuthology.orchestra.run.smithi117.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-19T02:50:07.493 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T02:50:07.493 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-19T02:50:07.493 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-19T02:50:07.494 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrade 1 Package 2024-03-19T02:50:07.494 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T02:50:07.494 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 375 M 2024-03-19T02:50:07.494 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-19T02:50:18.111 INFO:teuthology.orchestra.run.smithi117.stdout:linux-firmware-20240219-143.el9.noarch.rpm 35 MB/s | 375 MB 00:10 2024-03-19T02:50:18.112 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-19T02:50:18.113 INFO:teuthology.orchestra.run.smithi117.stdout:Total 35 MB/s | 375 MB 00:10 2024-03-19T02:50:18.113 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-19T02:50:18.199 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-19T02:50:18.199 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-19T02:50:20.665 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-19T02:50:20.666 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-19T02:50:27.039 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-19T02:50:27.270 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-19T02:50:27.352 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-19T02:50:28.563 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-19T02:50:28.563 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-19T02:50:28.950 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-19T02:50:28.950 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T02:50:28.950 INFO:teuthology.orchestra.run.smithi117.stdout:Upgraded: 2024-03-19T02:50:28.951 INFO:teuthology.orchestra.run.smithi117.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-19T02:50:28.951 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T02:50:28.951 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-19T02:50:29.257 INFO:teuthology.task.kernel:Downloading kernel 908b6b276e8f14e16032b32a9770077cd37b6bd4 on client.0... 2024-03-19T02:50:29.258 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=908b6b276e8f14e16032b32a9770077cd37b6bd4 2024-03-19T02:50:29.408 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/908b6b276e8f14e16032b32a9770077cd37b6bd4/centos/9/flavors/default/x86_64/ 2024-03-19T02:50:29.409 DEBUG:teuthology.orchestra.run.smithi117:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g908b6b276e8f-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/908b6b276e8f14e16032b32a9770077cd37b6bd4/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-19T02:50:29.411 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi117.front.sepia.ceph.com to complete... 2024-03-19T02:50:40.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19 02:50:40 URL:https://2.chacra.ceph.com/r/kernel/testing/908b6b276e8f14e16032b32a9770077cd37b6bd4/centos/9/flavors/default/x86_64/kernel-6.8.0_g908b6b276e8f-1.x86_64.rpm [336362676/336362676] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-19T02:50:40.827 INFO:teuthology.orchestra.run.smithi117.stderr:FINISHED --2024-03-19 02:50:40-- 2024-03-19T02:50:40.827 INFO:teuthology.orchestra.run.smithi117.stderr:Total wall clock time: 11s 2024-03-19T02:50:40.827 INFO:teuthology.orchestra.run.smithi117.stderr:Downloaded: 1 files, 321M in 11s (28.9 MB/s) 2024-03-19T02:50:40.829 INFO:teuthology.task.kernel:Installing kernel 908b6b276e8f14e16032b32a9770077cd37b6bd4 on client.0... 2024-03-19T02:50:40.830 DEBUG:teuthology.orchestra.run.smithi117:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-19T02:50:42.765 INFO:teuthology.orchestra.run.smithi117.stdout:Verifying... ######################################## 2024-03-19T02:50:42.900 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing... ######################################## 2024-03-19T02:50:42.980 INFO:teuthology.orchestra.run.smithi117.stdout:Updating / installing... 2024-03-19T02:50:50.569 INFO:teuthology.orchestra.run.smithi117.stdout:kernel-6.8.0_g908b6b276e8f-1 ######################################## 2024-03-19T02:51:06.614 INFO:teuthology.orchestra.run.smithi117.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-19T02:51:29.111 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi117.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-19T02:51:29.112 DEBUG:teuthology.orchestra.run.smithi117:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-19T02:51:29.206 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f 2024-03-19T02:51:29.206 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/System.map 2024-03-19T02:51:29.207 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/config 2024-03-19T02:51:29.207 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel 2024-03-19T02:51:29.207 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch 2024-03-19T02:51:29.207 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86 2024-03-19T02:51:29.207 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto 2024-03-19T02:51:29.207 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-19T02:51:29.208 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-19T02:51:29.208 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-19T02:51:29.208 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-19T02:51:29.208 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-19T02:51:29.208 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-19T02:51:29.209 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-19T02:51:29.209 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-19T02:51:29.209 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-19T02:51:29.209 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-19T02:51:29.209 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-19T02:51:29.209 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-19T02:51:29.210 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-19T02:51:29.210 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-19T02:51:29.210 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-19T02:51:29.210 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-19T02:51:29.210 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-19T02:51:29.210 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-19T02:51:29.211 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-19T02:51:29.211 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-19T02:51:29.211 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/kernel 2024-03-19T02:51:29.211 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/kernel/cpu 2024-03-19T02:51:29.211 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/kernel/cpu/mce 2024-03-19T02:51:29.211 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-19T02:51:29.212 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/kvm 2024-03-19T02:51:29.212 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-19T02:51:29.212 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-19T02:51:29.212 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/arch/x86/kvm/kvm.ko 2024-03-19T02:51:29.212 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto 2024-03-19T02:51:29.212 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/ansi_cprng.ko 2024-03-19T02:51:29.213 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/anubis.ko 2024-03-19T02:51:29.213 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/arc4.ko 2024-03-19T02:51:29.213 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/async_tx 2024-03-19T02:51:29.213 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/async_tx/async_memcpy.ko 2024-03-19T02:51:29.213 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/async_tx/async_pq.ko 2024-03-19T02:51:29.213 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-19T02:51:29.214 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/async_tx/async_tx.ko 2024-03-19T02:51:29.214 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/async_tx/async_xor.ko 2024-03-19T02:51:29.214 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/async_tx/raid6test.ko 2024-03-19T02:51:29.214 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/authenc.ko 2024-03-19T02:51:29.214 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/authencesn.ko 2024-03-19T02:51:29.215 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/blake2b_generic.ko 2024-03-19T02:51:29.215 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/blowfish_common.ko 2024-03-19T02:51:29.215 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/blowfish_generic.ko 2024-03-19T02:51:29.215 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/camellia_generic.ko 2024-03-19T02:51:29.215 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/cast5_generic.ko 2024-03-19T02:51:29.215 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/cast6_generic.ko 2024-03-19T02:51:29.216 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/cast_common.ko 2024-03-19T02:51:29.216 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/ccm.ko 2024-03-19T02:51:29.216 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/cmac.ko 2024-03-19T02:51:29.216 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/crc32_generic.ko 2024-03-19T02:51:29.216 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/crypto_engine.ko 2024-03-19T02:51:29.216 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/crypto_user.ko 2024-03-19T02:51:29.217 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/cts.ko 2024-03-19T02:51:29.217 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/des_generic.ko 2024-03-19T02:51:29.217 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/dh_generic.ko 2024-03-19T02:51:29.217 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/echainiv.ko 2024-03-19T02:51:29.217 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/essiv.ko 2024-03-19T02:51:29.217 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/fcrypt.ko 2024-03-19T02:51:29.217 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/khazad.ko 2024-03-19T02:51:29.218 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/lz4.ko 2024-03-19T02:51:29.218 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/lz4hc.ko 2024-03-19T02:51:29.218 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/md4.ko 2024-03-19T02:51:29.218 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/michael_mic.ko 2024-03-19T02:51:29.218 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/pcbc.ko 2024-03-19T02:51:29.219 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/pcrypt.ko 2024-03-19T02:51:29.219 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/rmd160.ko 2024-03-19T02:51:29.219 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/seed.ko 2024-03-19T02:51:29.219 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/serpent_generic.ko 2024-03-19T02:51:29.219 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/tcrypt.ko 2024-03-19T02:51:29.219 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/tea.ko 2024-03-19T02:51:29.220 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/twofish_common.ko 2024-03-19T02:51:29.220 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/twofish_generic.ko 2024-03-19T02:51:29.220 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/vmac.ko 2024-03-19T02:51:29.220 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/wp512.ko 2024-03-19T02:51:29.220 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/xcbc.ko 2024-03-19T02:51:29.220 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/xor.ko 2024-03-19T02:51:29.221 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/crypto/xxhash_generic.ko 2024-03-19T02:51:29.221 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers 2024-03-19T02:51:29.221 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/acpi 2024-03-19T02:51:29.221 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-19T02:51:29.221 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/acpi/acpi_pad.ko 2024-03-19T02:51:29.221 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/acpi/custom_method.ko 2024-03-19T02:51:29.221 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/acpi/ec_sys.ko 2024-03-19T02:51:29.222 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/acpi/platform_profile.ko 2024-03-19T02:51:29.222 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/acpi/sbs.ko 2024-03-19T02:51:29.222 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/acpi/sbshc.ko 2024-03-19T02:51:29.222 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/acpi/video.ko 2024-03-19T02:51:29.222 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata 2024-03-19T02:51:29.223 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/acard-ahci.ko 2024-03-19T02:51:29.223 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/ahci_platform.ko 2024-03-19T02:51:29.223 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/ata_generic.ko 2024-03-19T02:51:29.223 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/libahci_platform.ko 2024-03-19T02:51:29.223 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_acpi.ko 2024-03-19T02:51:29.223 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_ali.ko 2024-03-19T02:51:29.224 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_amd.ko 2024-03-19T02:51:29.224 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_artop.ko 2024-03-19T02:51:29.224 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_atiixp.ko 2024-03-19T02:51:29.224 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_atp867x.ko 2024-03-19T02:51:29.224 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_cmd640.ko 2024-03-19T02:51:29.224 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_cmd64x.ko 2024-03-19T02:51:29.225 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_cypress.ko 2024-03-19T02:51:29.225 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_efar.ko 2024-03-19T02:51:29.225 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_hpt366.ko 2024-03-19T02:51:29.225 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_hpt37x.ko 2024-03-19T02:51:29.225 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-19T02:51:29.225 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-19T02:51:29.226 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_it8213.ko 2024-03-19T02:51:29.226 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_it821x.ko 2024-03-19T02:51:29.226 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_jmicron.ko 2024-03-19T02:51:29.226 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_marvell.ko 2024-03-19T02:51:29.226 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_mpiix.ko 2024-03-19T02:51:29.226 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_netcell.ko 2024-03-19T02:51:29.227 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_ninja32.ko 2024-03-19T02:51:29.227 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_ns87410.ko 2024-03-19T02:51:29.227 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_ns87415.ko 2024-03-19T02:51:29.227 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_oldpiix.ko 2024-03-19T02:51:29.227 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_opti.ko 2024-03-19T02:51:29.227 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_optidma.ko 2024-03-19T02:51:29.227 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_pcmcia.ko 2024-03-19T02:51:29.228 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-19T02:51:29.228 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-19T02:51:29.228 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_piccolo.ko 2024-03-19T02:51:29.228 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_rdc.ko 2024-03-19T02:51:29.228 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_sch.ko 2024-03-19T02:51:29.228 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_serverworks.ko 2024-03-19T02:51:29.229 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_sil680.ko 2024-03-19T02:51:29.229 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_sis.ko 2024-03-19T02:51:29.229 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_sl82c105.ko 2024-03-19T02:51:29.229 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_triflex.ko 2024-03-19T02:51:29.229 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pata_via.ko 2024-03-19T02:51:29.229 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/pdc_adma.ko 2024-03-19T02:51:29.229 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_inic162x.ko 2024-03-19T02:51:29.230 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_mv.ko 2024-03-19T02:51:29.230 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_nv.ko 2024-03-19T02:51:29.230 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_promise.ko 2024-03-19T02:51:29.230 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_qstor.ko 2024-03-19T02:51:29.230 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_sil.ko 2024-03-19T02:51:29.230 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_sil24.ko 2024-03-19T02:51:29.231 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_sis.ko 2024-03-19T02:51:29.231 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_svw.ko 2024-03-19T02:51:29.231 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_sx4.ko 2024-03-19T02:51:29.231 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_uli.ko 2024-03-19T02:51:29.231 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_via.ko 2024-03-19T02:51:29.231 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ata/sata_vsc.ko 2024-03-19T02:51:29.231 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/auxdisplay 2024-03-19T02:51:29.232 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/auxdisplay/ks0108.ko 2024-03-19T02:51:29.232 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/base 2024-03-19T02:51:29.232 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/base/regmap 2024-03-19T02:51:29.232 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-19T02:51:29.232 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-19T02:51:29.232 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/bcma 2024-03-19T02:51:29.232 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/bcma/bcma.ko 2024-03-19T02:51:29.232 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block 2024-03-19T02:51:29.232 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/aoe 2024-03-19T02:51:29.232 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/aoe/aoe.ko 2024-03-19T02:51:29.232 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/brd.ko 2024-03-19T02:51:29.232 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/drbd 2024-03-19T02:51:29.232 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/drbd/drbd.ko 2024-03-19T02:51:29.232 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/floppy.ko 2024-03-19T02:51:29.233 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/loop.ko 2024-03-19T02:51:29.233 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/mtip32xx 2024-03-19T02:51:29.233 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-19T02:51:29.233 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/nbd.ko 2024-03-19T02:51:29.233 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/null_blk 2024-03-19T02:51:29.233 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/null_blk/null_blk.ko 2024-03-19T02:51:29.233 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/pktcdvd.ko 2024-03-19T02:51:29.233 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/rbd.ko 2024-03-19T02:51:29.233 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/virtio_blk.ko 2024-03-19T02:51:29.233 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/xen-blkback 2024-03-19T02:51:29.233 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-19T02:51:29.233 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/xen-blkfront.ko 2024-03-19T02:51:29.233 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/zram 2024-03-19T02:51:29.233 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/block/zram/zram.ko 2024-03-19T02:51:29.233 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char 2024-03-19T02:51:29.234 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/hangcheck-timer.ko 2024-03-19T02:51:29.234 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/hw_random 2024-03-19T02:51:29.234 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-19T02:51:29.234 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-19T02:51:29.234 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-19T02:51:29.234 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/hw_random/via-rng.ko 2024-03-19T02:51:29.234 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-19T02:51:29.234 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/ipmi 2024-03-19T02:51:29.234 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-19T02:51:29.234 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-19T02:51:29.234 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-19T02:51:29.234 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-19T02:51:29.234 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-19T02:51:29.235 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-19T02:51:29.235 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/lp.ko 2024-03-19T02:51:29.235 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/mwave 2024-03-19T02:51:29.235 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/mwave/mwave.ko 2024-03-19T02:51:29.235 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/ppdev.ko 2024-03-19T02:51:29.235 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/tlclk.ko 2024-03-19T02:51:29.235 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/tpm 2024-03-19T02:51:29.235 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/tpm/tpm.ko 2024-03-19T02:51:29.235 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-19T02:51:29.235 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-19T02:51:29.235 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-19T02:51:29.235 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-19T02:51:29.235 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-19T02:51:29.235 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/uv_mmtimer.ko 2024-03-19T02:51:29.235 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/char/virtio_console.ko 2024-03-19T02:51:29.236 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/cpufreq 2024-03-19T02:51:29.236 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-19T02:51:29.236 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-19T02:51:29.236 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-19T02:51:29.236 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-19T02:51:29.236 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-19T02:51:29.236 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-19T02:51:29.236 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto 2024-03-19T02:51:29.236 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/ccp 2024-03-19T02:51:29.236 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-19T02:51:29.236 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/ccp/ccp.ko 2024-03-19T02:51:29.236 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/intel 2024-03-19T02:51:29.236 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/intel/qat 2024-03-19T02:51:29.236 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/intel/qat/qat_common 2024-03-19T02:51:29.236 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-19T02:51:29.237 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-19T02:51:29.237 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-19T02:51:29.237 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/padlock-aes.ko 2024-03-19T02:51:29.237 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/padlock-sha.ko 2024-03-19T02:51:29.237 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/virtio 2024-03-19T02:51:29.237 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-19T02:51:29.237 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/dca 2024-03-19T02:51:29.237 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/dca/dca.ko 2024-03-19T02:51:29.237 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/devfreq 2024-03-19T02:51:29.237 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-19T02:51:29.237 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/dma 2024-03-19T02:51:29.237 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/dma/dw 2024-03-19T02:51:29.237 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-19T02:51:29.237 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/dma/ioat 2024-03-19T02:51:29.237 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-19T02:51:29.238 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac 2024-03-19T02:51:29.238 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/amd64_edac.ko 2024-03-19T02:51:29.238 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/e752x_edac.ko 2024-03-19T02:51:29.238 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/edac_mce_amd.ko 2024-03-19T02:51:29.238 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/i3000_edac.ko 2024-03-19T02:51:29.238 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/i3200_edac.ko 2024-03-19T02:51:29.238 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/i5100_edac.ko 2024-03-19T02:51:29.238 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/i5400_edac.ko 2024-03-19T02:51:29.238 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/i7300_edac.ko 2024-03-19T02:51:29.238 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/i7core_edac.ko 2024-03-19T02:51:29.238 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/i82975x_edac.ko 2024-03-19T02:51:29.238 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/ie31200_edac.ko 2024-03-19T02:51:29.238 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/sb_edac.ko 2024-03-19T02:51:29.238 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/edac/x38_edac.ko 2024-03-19T02:51:29.239 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/firmware 2024-03-19T02:51:29.239 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/firmware/edd.ko 2024-03-19T02:51:29.239 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-19T02:51:29.239 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/gpio 2024-03-19T02:51:29.239 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/gpio/gpio-ich.ko 2024-03-19T02:51:29.239 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-19T02:51:29.239 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid 2024-03-19T02:51:29.239 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-appleir.ko 2024-03-19T02:51:29.239 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-aureal.ko 2024-03-19T02:51:29.239 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-axff.ko 2024-03-19T02:51:29.239 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-dr.ko 2024-03-19T02:51:29.239 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-elecom.ko 2024-03-19T02:51:29.239 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-elo.ko 2024-03-19T02:51:29.239 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-emsff.ko 2024-03-19T02:51:29.239 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-gaff.ko 2024-03-19T02:51:29.240 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-gt683r.ko 2024-03-19T02:51:29.240 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-gyration.ko 2024-03-19T02:51:29.240 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-19T02:51:29.240 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-19T02:51:29.240 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-holtekff.ko 2024-03-19T02:51:29.240 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-hyperv.ko 2024-03-19T02:51:29.240 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-icade.ko 2024-03-19T02:51:29.240 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-keytouch.ko 2024-03-19T02:51:29.240 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-kye.ko 2024-03-19T02:51:29.240 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-lcpower.ko 2024-03-19T02:51:29.240 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-led.ko 2024-03-19T02:51:29.240 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-lenovo.ko 2024-03-19T02:51:29.240 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-19T02:51:29.240 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-19T02:51:29.241 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-multitouch.ko 2024-03-19T02:51:29.241 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-ortek.ko 2024-03-19T02:51:29.241 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-penmount.ko 2024-03-19T02:51:29.241 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-petalynx.ko 2024-03-19T02:51:29.241 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-picolcd.ko 2024-03-19T02:51:29.241 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-pl.ko 2024-03-19T02:51:29.241 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-plantronics.ko 2024-03-19T02:51:29.241 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-primax.ko 2024-03-19T02:51:29.241 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-rmi.ko 2024-03-19T02:51:29.241 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-19T02:51:29.241 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-roccat-common.ko 2024-03-19T02:51:29.241 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-19T02:51:29.241 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-19T02:51:29.241 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-19T02:51:29.241 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-19T02:51:29.242 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-19T02:51:29.242 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-19T02:51:29.242 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-19T02:51:29.242 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-19T02:51:29.242 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-19T02:51:29.242 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-roccat.ko 2024-03-19T02:51:29.242 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-saitek.ko 2024-03-19T02:51:29.242 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-samsung.ko 2024-03-19T02:51:29.242 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-19T02:51:29.242 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-sjoy.ko 2024-03-19T02:51:29.242 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-sony.ko 2024-03-19T02:51:29.242 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-speedlink.ko 2024-03-19T02:51:29.242 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-steelseries.ko 2024-03-19T02:51:29.243 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-sunplus.ko 2024-03-19T02:51:29.243 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-19T02:51:29.243 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-tivo.ko 2024-03-19T02:51:29.243 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-tmff.ko 2024-03-19T02:51:29.243 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-topseed.ko 2024-03-19T02:51:29.243 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-twinhan.ko 2024-03-19T02:51:29.243 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-uclogic.ko 2024-03-19T02:51:29.243 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-waltop.ko 2024-03-19T02:51:29.243 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-wiimote.ko 2024-03-19T02:51:29.243 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-xinmo.ko 2024-03-19T02:51:29.243 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-zpff.ko 2024-03-19T02:51:29.243 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/hid-zydacron.ko 2024-03-19T02:51:29.243 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/uhid.ko 2024-03-19T02:51:29.243 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hid/wacom.ko 2024-03-19T02:51:29.243 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hv 2024-03-19T02:51:29.244 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hv/hv_balloon.ko 2024-03-19T02:51:29.244 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hv/hv_utils.ko 2024-03-19T02:51:29.244 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hv/hv_vmbus.ko 2024-03-19T02:51:29.244 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon 2024-03-19T02:51:29.244 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/abituguru.ko 2024-03-19T02:51:29.244 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/abituguru3.ko 2024-03-19T02:51:29.244 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-19T02:51:29.244 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ad7414.ko 2024-03-19T02:51:29.244 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ad7418.ko 2024-03-19T02:51:29.244 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/adc128d818.ko 2024-03-19T02:51:29.244 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/adm1025.ko 2024-03-19T02:51:29.244 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/adm1026.ko 2024-03-19T02:51:29.244 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/adm1029.ko 2024-03-19T02:51:29.244 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/adm1031.ko 2024-03-19T02:51:29.245 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/adm9240.ko 2024-03-19T02:51:29.245 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ads7828.ko 2024-03-19T02:51:29.245 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/adt7410.ko 2024-03-19T02:51:29.245 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/adt7411.ko 2024-03-19T02:51:29.245 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/adt7462.ko 2024-03-19T02:51:29.245 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/adt7470.ko 2024-03-19T02:51:29.245 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/adt7475.ko 2024-03-19T02:51:29.245 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/adt7x10.ko 2024-03-19T02:51:29.245 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/amc6821.ko 2024-03-19T02:51:29.245 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/applesmc.ko 2024-03-19T02:51:29.245 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/asb100.ko 2024-03-19T02:51:29.245 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/asc7621.ko 2024-03-19T02:51:29.245 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-19T02:51:29.245 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/atxp1.ko 2024-03-19T02:51:29.245 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/coretemp.ko 2024-03-19T02:51:29.246 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-19T02:51:29.246 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/dme1737.ko 2024-03-19T02:51:29.246 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ds1621.ko 2024-03-19T02:51:29.246 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ds620.ko 2024-03-19T02:51:29.246 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/emc1403.ko 2024-03-19T02:51:29.246 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/emc6w201.ko 2024-03-19T02:51:29.246 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/f71805f.ko 2024-03-19T02:51:29.246 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/f71882fg.ko 2024-03-19T02:51:29.246 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/f75375s.ko 2024-03-19T02:51:29.246 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/fam15h_power.ko 2024-03-19T02:51:29.246 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/fschmd.ko 2024-03-19T02:51:29.246 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/g760a.ko 2024-03-19T02:51:29.246 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/g762.ko 2024-03-19T02:51:29.246 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/gl518sm.ko 2024-03-19T02:51:29.246 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/gl520sm.ko 2024-03-19T02:51:29.247 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-19T02:51:29.247 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/i5500_temp.ko 2024-03-19T02:51:29.247 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/i5k_amb.ko 2024-03-19T02:51:29.247 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ibmaem.ko 2024-03-19T02:51:29.247 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ibmpex.ko 2024-03-19T02:51:29.247 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ina209.ko 2024-03-19T02:51:29.247 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ina2xx.ko 2024-03-19T02:51:29.247 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/it87.ko 2024-03-19T02:51:29.247 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/k10temp.ko 2024-03-19T02:51:29.247 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/k8temp.ko 2024-03-19T02:51:29.247 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lineage-pem.ko 2024-03-19T02:51:29.247 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm63.ko 2024-03-19T02:51:29.247 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm73.ko 2024-03-19T02:51:29.247 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm75.ko 2024-03-19T02:51:29.248 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm77.ko 2024-03-19T02:51:29.248 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm78.ko 2024-03-19T02:51:29.248 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm80.ko 2024-03-19T02:51:29.248 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm83.ko 2024-03-19T02:51:29.248 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm85.ko 2024-03-19T02:51:29.248 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm87.ko 2024-03-19T02:51:29.248 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm90.ko 2024-03-19T02:51:29.248 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm92.ko 2024-03-19T02:51:29.248 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm93.ko 2024-03-19T02:51:29.248 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm95234.ko 2024-03-19T02:51:29.248 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm95241.ko 2024-03-19T02:51:29.248 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/lm95245.ko 2024-03-19T02:51:29.248 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ltc2945.ko 2024-03-19T02:51:29.248 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ltc4151.ko 2024-03-19T02:51:29.248 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ltc4215.ko 2024-03-19T02:51:29.249 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ltc4222.ko 2024-03-19T02:51:29.249 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ltc4245.ko 2024-03-19T02:51:29.249 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ltc4260.ko 2024-03-19T02:51:29.249 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/ltc4261.ko 2024-03-19T02:51:29.249 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/max16065.ko 2024-03-19T02:51:29.249 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/max1619.ko 2024-03-19T02:51:29.249 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/max1668.ko 2024-03-19T02:51:29.249 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/max197.ko 2024-03-19T02:51:29.249 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/max6639.ko 2024-03-19T02:51:29.249 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/max6650.ko 2024-03-19T02:51:29.249 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/max6697.ko 2024-03-19T02:51:29.249 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/mcp3021.ko 2024-03-19T02:51:29.249 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/nct6683.ko 2024-03-19T02:51:29.249 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/nct6775-core.ko 2024-03-19T02:51:29.249 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/nct6775.ko 2024-03-19T02:51:29.250 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/nct7802.ko 2024-03-19T02:51:29.250 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pc87360.ko 2024-03-19T02:51:29.250 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pc87427.ko 2024-03-19T02:51:29.250 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pcf8591.ko 2024-03-19T02:51:29.250 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus 2024-03-19T02:51:29.250 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-19T02:51:29.250 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-19T02:51:29.250 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-19T02:51:29.250 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-19T02:51:29.250 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-19T02:51:29.250 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-19T02:51:29.250 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-19T02:51:29.250 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-19T02:51:29.250 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-19T02:51:29.251 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-19T02:51:29.251 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-19T02:51:29.251 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-19T02:51:29.251 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/powr1220.ko 2024-03-19T02:51:29.251 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/sch5627.ko 2024-03-19T02:51:29.251 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/sch5636.ko 2024-03-19T02:51:29.251 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-19T02:51:29.251 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/sht15.ko 2024-03-19T02:51:29.251 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/sht21.ko 2024-03-19T02:51:29.251 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/shtc1.ko 2024-03-19T02:51:29.251 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/sis5595.ko 2024-03-19T02:51:29.251 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/smsc47b397.ko 2024-03-19T02:51:29.252 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/smsc47m1.ko 2024-03-19T02:51:29.252 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/smsc47m192.ko 2024-03-19T02:51:29.252 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/thmc50.ko 2024-03-19T02:51:29.252 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/tmp102.ko 2024-03-19T02:51:29.252 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/tmp103.ko 2024-03-19T02:51:29.252 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/tmp401.ko 2024-03-19T02:51:29.252 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/tmp421.ko 2024-03-19T02:51:29.252 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/via-cputemp.ko 2024-03-19T02:51:29.252 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/via686a.ko 2024-03-19T02:51:29.252 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/vt1211.ko 2024-03-19T02:51:29.252 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/vt8231.ko 2024-03-19T02:51:29.252 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/w83627ehf.ko 2024-03-19T02:51:29.252 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/w83627hf.ko 2024-03-19T02:51:29.252 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/w83781d.ko 2024-03-19T02:51:29.252 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/w83791d.ko 2024-03-19T02:51:29.253 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/w83792d.ko 2024-03-19T02:51:29.253 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/w83793.ko 2024-03-19T02:51:29.253 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/w83795.ko 2024-03-19T02:51:29.253 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/w83l785ts.ko 2024-03-19T02:51:29.253 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/hwmon/w83l786ng.ko 2024-03-19T02:51:29.253 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c 2024-03-19T02:51:29.253 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/algos 2024-03-19T02:51:29.253 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-19T02:51:29.253 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-19T02:51:29.253 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses 2024-03-19T02:51:29.253 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-19T02:51:29.253 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-19T02:51:29.253 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-19T02:51:29.253 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-19T02:51:29.253 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-19T02:51:29.254 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-19T02:51:29.254 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-19T02:51:29.254 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-19T02:51:29.254 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-19T02:51:29.254 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-19T02:51:29.254 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-19T02:51:29.254 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-19T02:51:29.254 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-19T02:51:29.254 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-19T02:51:29.254 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-19T02:51:29.254 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-19T02:51:29.254 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-19T02:51:29.254 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-19T02:51:29.254 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-19T02:51:29.254 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-19T02:51:29.255 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-19T02:51:29.255 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-19T02:51:29.255 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-19T02:51:29.255 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/i2c-dev.ko 2024-03-19T02:51:29.255 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/i2c-mux.ko 2024-03-19T02:51:29.255 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-19T02:51:29.255 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/i2c-smbus.ko 2024-03-19T02:51:29.255 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/i2c/i2c-stub.ko 2024-03-19T02:51:29.255 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband 2024-03-19T02:51:29.255 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/core 2024-03-19T02:51:29.255 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-19T02:51:29.255 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/core/ib_core.ko 2024-03-19T02:51:29.255 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-19T02:51:29.256 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-19T02:51:29.256 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-19T02:51:29.256 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-19T02:51:29.256 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-19T02:51:29.256 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/hw 2024-03-19T02:51:29.256 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/hw/cxgb4 2024-03-19T02:51:29.256 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-19T02:51:29.256 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/hw/mlx4 2024-03-19T02:51:29.256 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-19T02:51:29.256 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/hw/mlx5 2024-03-19T02:51:29.256 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-19T02:51:29.256 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/hw/mthca 2024-03-19T02:51:29.256 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-19T02:51:29.256 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/ulp 2024-03-19T02:51:29.256 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/ulp/ipoib 2024-03-19T02:51:29.257 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-19T02:51:29.257 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/ulp/iser 2024-03-19T02:51:29.257 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-19T02:51:29.257 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/ulp/isert 2024-03-19T02:51:29.257 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-19T02:51:29.257 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/ulp/srp 2024-03-19T02:51:29.257 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-19T02:51:29.257 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/ulp/srpt 2024-03-19T02:51:29.257 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-19T02:51:29.257 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input 2024-03-19T02:51:29.257 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/gameport 2024-03-19T02:51:29.257 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-19T02:51:29.257 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-19T02:51:29.257 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/gameport/gameport.ko 2024-03-19T02:51:29.258 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/joydev.ko 2024-03-19T02:51:29.258 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/keyboard 2024-03-19T02:51:29.258 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-19T02:51:29.258 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/mouse 2024-03-19T02:51:29.258 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/mouse/appletouch.ko 2024-03-19T02:51:29.258 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/mouse/bcm5974.ko 2024-03-19T02:51:29.258 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/mouse/cyapatp.ko 2024-03-19T02:51:29.258 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-19T02:51:29.258 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/mouse/sermouse.ko 2024-03-19T02:51:29.258 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-19T02:51:29.258 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-19T02:51:29.258 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-19T02:51:29.258 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/rmi4 2024-03-19T02:51:29.258 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-19T02:51:29.259 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/serio 2024-03-19T02:51:29.259 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/serio/altera_ps2.ko 2024-03-19T02:51:29.259 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/serio/arc_ps2.ko 2024-03-19T02:51:29.259 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-19T02:51:29.259 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/serio/serio_raw.ko 2024-03-19T02:51:29.259 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/input/sparse-keymap.ko 2024-03-19T02:51:29.259 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds 2024-03-19T02:51:29.259 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/leds-blinkm.ko 2024-03-19T02:51:29.259 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/leds-lm3530.ko 2024-03-19T02:51:29.259 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/leds-lp3944.ko 2024-03-19T02:51:29.259 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/leds-ss4200.ko 2024-03-19T02:51:29.259 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/trigger 2024-03-19T02:51:29.259 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-19T02:51:29.259 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-19T02:51:29.259 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-19T02:51:29.260 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-19T02:51:29.260 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-19T02:51:29.260 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-19T02:51:29.260 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-19T02:51:29.260 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-19T02:51:29.260 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-19T02:51:29.260 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md 2024-03-19T02:51:29.260 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/bcache 2024-03-19T02:51:29.260 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/bcache/bcache.ko 2024-03-19T02:51:29.260 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-bio-prison.ko 2024-03-19T02:51:29.260 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-cache-smq.ko 2024-03-19T02:51:29.260 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-cache.ko 2024-03-19T02:51:29.260 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-crypt.ko 2024-03-19T02:51:29.260 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-delay.ko 2024-03-19T02:51:29.260 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-flakey.ko 2024-03-19T02:51:29.261 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-log-userspace.ko 2024-03-19T02:51:29.261 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-multipath.ko 2024-03-19T02:51:29.261 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-queue-length.ko 2024-03-19T02:51:29.261 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-raid.ko 2024-03-19T02:51:29.261 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-round-robin.ko 2024-03-19T02:51:29.261 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-service-time.ko 2024-03-19T02:51:29.261 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-switch.ko 2024-03-19T02:51:29.261 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-thin-pool.ko 2024-03-19T02:51:29.261 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/dm-verity.ko 2024-03-19T02:51:29.261 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/persistent-data 2024-03-19T02:51:29.261 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-19T02:51:29.261 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/raid0.ko 2024-03-19T02:51:29.261 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/raid1.ko 2024-03-19T02:51:29.261 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/raid10.ko 2024-03-19T02:51:29.261 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/md/raid456.ko 2024-03-19T02:51:29.262 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/message 2024-03-19T02:51:29.262 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/message/fusion 2024-03-19T02:51:29.262 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/message/fusion/mptbase.ko 2024-03-19T02:51:29.262 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/message/fusion/mptctl.ko 2024-03-19T02:51:29.262 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/message/fusion/mptfc.ko 2024-03-19T02:51:29.262 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/message/fusion/mptlan.ko 2024-03-19T02:51:29.262 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/message/fusion/mptsas.ko 2024-03-19T02:51:29.262 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/message/fusion/mptscsih.ko 2024-03-19T02:51:29.262 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/message/fusion/mptspi.ko 2024-03-19T02:51:29.262 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/mfd 2024-03-19T02:51:29.262 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/mfd/lpc_ich.ko 2024-03-19T02:51:29.262 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/mfd/lpc_sch.ko 2024-03-19T02:51:29.262 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/mfd/mfd-core.ko 2024-03-19T02:51:29.262 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/mfd/sm501.ko 2024-03-19T02:51:29.263 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/mfd/viperboard.ko 2024-03-19T02:51:29.263 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/mfd/vx855.ko 2024-03-19T02:51:29.263 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/mfd/wl1273-core.ko 2024-03-19T02:51:29.263 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc 2024-03-19T02:51:29.263 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/altera-stapl 2024-03-19T02:51:29.263 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-19T02:51:29.263 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/apds9802als.ko 2024-03-19T02:51:29.263 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/apds990x.ko 2024-03-19T02:51:29.263 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/bh1770glc.ko 2024-03-19T02:51:29.263 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/cb710 2024-03-19T02:51:29.263 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/cb710/cb710.ko 2024-03-19T02:51:29.263 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/eeprom 2024-03-19T02:51:29.263 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/eeprom/at24.ko 2024-03-19T02:51:29.263 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-19T02:51:29.263 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/eeprom/max6875.ko 2024-03-19T02:51:29.264 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/enclosure.ko 2024-03-19T02:51:29.264 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/hpilo.ko 2024-03-19T02:51:29.264 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/isl29003.ko 2024-03-19T02:51:29.264 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/isl29020.ko 2024-03-19T02:51:29.264 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/lis3lv02d 2024-03-19T02:51:29.264 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-19T02:51:29.264 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-19T02:51:29.264 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/mei 2024-03-19T02:51:29.264 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/mei/mei-me.ko 2024-03-19T02:51:29.264 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/mei/mei-txe.ko 2024-03-19T02:51:29.264 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/mei/mei.ko 2024-03-19T02:51:29.264 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/sgi-gru 2024-03-19T02:51:29.264 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-19T02:51:29.264 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/sgi-xp 2024-03-19T02:51:29.264 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-19T02:51:29.265 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-19T02:51:29.265 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-19T02:51:29.265 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/tifm_7xx1.ko 2024-03-19T02:51:29.265 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/tifm_core.ko 2024-03-19T02:51:29.265 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/tsl2550.ko 2024-03-19T02:51:29.265 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/vmw_balloon.ko 2024-03-19T02:51:29.265 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/vmw_vmci 2024-03-19T02:51:29.265 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-19T02:51:29.265 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net 2024-03-19T02:51:29.265 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/bonding 2024-03-19T02:51:29.265 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/bonding/bonding.ko 2024-03-19T02:51:29.265 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/dsa 2024-03-19T02:51:29.265 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/dsa/b53 2024-03-19T02:51:29.265 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-19T02:51:29.265 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-19T02:51:29.266 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-19T02:51:29.266 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/dsa/mv88e6xxx 2024-03-19T02:51:29.266 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-19T02:51:29.266 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/dummy.ko 2024-03-19T02:51:29.266 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/eql.ko 2024-03-19T02:51:29.266 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet 2024-03-19T02:51:29.266 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/3com 2024-03-19T02:51:29.266 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-19T02:51:29.266 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-19T02:51:29.266 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-19T02:51:29.266 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-19T02:51:29.266 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/8390 2024-03-19T02:51:29.266 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-19T02:51:29.266 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-19T02:51:29.267 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-19T02:51:29.267 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-19T02:51:29.267 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/adaptec 2024-03-19T02:51:29.267 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-19T02:51:29.267 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/agere 2024-03-19T02:51:29.267 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-19T02:51:29.267 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/alteon 2024-03-19T02:51:29.267 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-19T02:51:29.267 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/altera 2024-03-19T02:51:29.267 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-19T02:51:29.267 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/amd 2024-03-19T02:51:29.267 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-19T02:51:29.267 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-19T02:51:29.267 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-19T02:51:29.267 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/atheros 2024-03-19T02:51:29.268 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/atheros/alx 2024-03-19T02:51:29.268 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-19T02:51:29.268 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-19T02:51:29.268 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-19T02:51:29.268 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-19T02:51:29.268 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-19T02:51:29.268 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/atheros/atlx 2024-03-19T02:51:29.268 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-19T02:51:29.268 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-19T02:51:29.268 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/broadcom 2024-03-19T02:51:29.268 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-19T02:51:29.268 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-19T02:51:29.268 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-19T02:51:29.268 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-19T02:51:29.268 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-19T02:51:29.269 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/broadcom/genet 2024-03-19T02:51:29.269 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-19T02:51:29.269 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-19T02:51:29.269 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/brocade 2024-03-19T02:51:29.269 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/brocade/bna 2024-03-19T02:51:29.269 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-19T02:51:29.269 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/chelsio 2024-03-19T02:51:29.269 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-19T02:51:29.269 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-19T02:51:29.269 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-19T02:51:29.269 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-19T02:51:29.269 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-19T02:51:29.269 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-19T02:51:29.269 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-19T02:51:29.269 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-19T02:51:29.270 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-19T02:51:29.270 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-19T02:51:29.270 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/cisco 2024-03-19T02:51:29.270 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/cisco/enic 2024-03-19T02:51:29.270 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-19T02:51:29.270 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/dec 2024-03-19T02:51:29.270 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/dec/tulip 2024-03-19T02:51:29.270 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-19T02:51:29.270 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-19T02:51:29.270 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-19T02:51:29.270 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-19T02:51:29.270 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-19T02:51:29.270 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-19T02:51:29.270 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/dlink 2024-03-19T02:51:29.271 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-19T02:51:29.271 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-19T02:51:29.271 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/dnet.ko 2024-03-19T02:51:29.271 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/emulex 2024-03-19T02:51:29.271 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/emulex/benet 2024-03-19T02:51:29.271 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-19T02:51:29.271 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/ethoc.ko 2024-03-19T02:51:29.271 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/fealnx.ko 2024-03-19T02:51:29.271 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel 2024-03-19T02:51:29.271 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-19T02:51:29.271 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/e1000 2024-03-19T02:51:29.271 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-19T02:51:29.271 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/e1000e 2024-03-19T02:51:29.271 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-19T02:51:29.271 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/fm10k 2024-03-19T02:51:29.272 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-19T02:51:29.272 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/i40e 2024-03-19T02:51:29.272 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-19T02:51:29.272 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/iavf 2024-03-19T02:51:29.272 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-19T02:51:29.272 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/igb 2024-03-19T02:51:29.272 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-19T02:51:29.272 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/igbvf 2024-03-19T02:51:29.272 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-19T02:51:29.272 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-19T02:51:29.272 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-19T02:51:29.272 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-19T02:51:29.272 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-19T02:51:29.272 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/jme.ko 2024-03-19T02:51:29.272 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/marvell 2024-03-19T02:51:29.273 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-19T02:51:29.273 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-19T02:51:29.273 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-19T02:51:29.273 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/mellanox 2024-03-19T02:51:29.273 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-19T02:51:29.273 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-19T02:51:29.273 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-19T02:51:29.273 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-19T02:51:29.273 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-19T02:51:29.273 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-19T02:51:29.273 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/micrel 2024-03-19T02:51:29.273 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-19T02:51:29.273 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/myricom 2024-03-19T02:51:29.273 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-19T02:51:29.273 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-19T02:51:29.274 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/natsemi 2024-03-19T02:51:29.274 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-19T02:51:29.274 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-19T02:51:29.274 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/neterion 2024-03-19T02:51:29.274 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-19T02:51:29.274 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/nvidia 2024-03-19T02:51:29.274 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-19T02:51:29.274 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/packetengines 2024-03-19T02:51:29.274 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-19T02:51:29.274 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-19T02:51:29.274 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/qlogic 2024-03-19T02:51:29.274 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-19T02:51:29.274 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-19T02:51:29.274 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-19T02:51:29.275 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-19T02:51:29.275 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-19T02:51:29.275 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/rdc 2024-03-19T02:51:29.275 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-19T02:51:29.275 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/realtek 2024-03-19T02:51:29.275 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-19T02:51:29.275 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-19T02:51:29.275 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-19T02:51:29.275 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-19T02:51:29.275 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/rocker 2024-03-19T02:51:29.275 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-19T02:51:29.275 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/sfc 2024-03-19T02:51:29.275 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-19T02:51:29.275 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/silan 2024-03-19T02:51:29.275 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-19T02:51:29.276 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/sis 2024-03-19T02:51:29.276 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-19T02:51:29.276 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-19T02:51:29.276 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/smsc 2024-03-19T02:51:29.276 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-19T02:51:29.276 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-19T02:51:29.276 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-19T02:51:29.276 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-19T02:51:29.276 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/stmicro 2024-03-19T02:51:29.276 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-19T02:51:29.276 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-19T02:51:29.276 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-19T02:51:29.276 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/sun 2024-03-19T02:51:29.276 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-19T02:51:29.277 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-19T02:51:29.277 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-19T02:51:29.277 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-19T02:51:29.277 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/tehuti 2024-03-19T02:51:29.277 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-19T02:51:29.277 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/ti 2024-03-19T02:51:29.277 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-19T02:51:29.277 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/via 2024-03-19T02:51:29.277 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-19T02:51:29.277 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-19T02:51:29.277 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/wiznet 2024-03-19T02:51:29.277 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-19T02:51:29.277 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-19T02:51:29.277 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/xircom 2024-03-19T02:51:29.277 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-19T02:51:29.277 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/geneve.ko 2024-03-19T02:51:29.278 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/hyperv 2024-03-19T02:51:29.278 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-19T02:51:29.278 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ieee802154 2024-03-19T02:51:29.278 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-19T02:51:29.278 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ifb.ko 2024-03-19T02:51:29.278 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ipvlan 2024-03-19T02:51:29.278 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-19T02:51:29.278 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/macvlan.ko 2024-03-19T02:51:29.278 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/macvtap.ko 2024-03-19T02:51:29.278 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/mdio 2024-03-19T02:51:29.278 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/mdio.ko 2024-03-19T02:51:29.278 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-19T02:51:29.278 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-19T02:51:29.278 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-19T02:51:29.279 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/mii.ko 2024-03-19T02:51:29.279 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/net_failover.ko 2024-03-19T02:51:29.279 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/netconsole.ko 2024-03-19T02:51:29.279 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/nlmon.ko 2024-03-19T02:51:29.279 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/pcs 2024-03-19T02:51:29.279 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-19T02:51:29.279 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-19T02:51:29.279 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy 2024-03-19T02:51:29.279 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/amd.ko 2024-03-19T02:51:29.279 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-19T02:51:29.279 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-19T02:51:29.279 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-19T02:51:29.279 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/bcm87xx.ko 2024-03-19T02:51:29.279 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/broadcom.ko 2024-03-19T02:51:29.280 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/cicada.ko 2024-03-19T02:51:29.280 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/davicom.ko 2024-03-19T02:51:29.280 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/dp83640.ko 2024-03-19T02:51:29.280 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/et1011c.ko 2024-03-19T02:51:29.280 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/icplus.ko 2024-03-19T02:51:29.280 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/lxt.ko 2024-03-19T02:51:29.280 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/marvell.ko 2024-03-19T02:51:29.280 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/micrel.ko 2024-03-19T02:51:29.280 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/national.ko 2024-03-19T02:51:29.280 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/phylink.ko 2024-03-19T02:51:29.280 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/qsemi.ko 2024-03-19T02:51:29.280 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/realtek.ko 2024-03-19T02:51:29.280 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/smsc.ko 2024-03-19T02:51:29.280 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/ste10Xp.ko 2024-03-19T02:51:29.281 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/phy/vitesse.ko 2024-03-19T02:51:29.281 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ppp 2024-03-19T02:51:29.281 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-19T02:51:29.281 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ppp/ppp_async.ko 2024-03-19T02:51:29.281 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-19T02:51:29.281 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-19T02:51:29.281 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-19T02:51:29.281 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-19T02:51:29.281 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ppp/pppoe.ko 2024-03-19T02:51:29.281 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ppp/pppox.ko 2024-03-19T02:51:29.281 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/ppp/pptp.ko 2024-03-19T02:51:29.281 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/slip 2024-03-19T02:51:29.281 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/slip/slhc.ko 2024-03-19T02:51:29.281 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/slip/slip.ko 2024-03-19T02:51:29.281 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/sungem_phy.ko 2024-03-19T02:51:29.281 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/tap.ko 2024-03-19T02:51:29.282 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/team 2024-03-19T02:51:29.282 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/team/team.ko 2024-03-19T02:51:29.282 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-19T02:51:29.282 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-19T02:51:29.282 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-19T02:51:29.282 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/team/team_mode_random.ko 2024-03-19T02:51:29.282 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-19T02:51:29.282 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/tun.ko 2024-03-19T02:51:29.282 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/veth.ko 2024-03-19T02:51:29.282 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/virtio_net.ko 2024-03-19T02:51:29.282 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/vmxnet3 2024-03-19T02:51:29.282 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-19T02:51:29.282 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/vxlan 2024-03-19T02:51:29.282 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/vxlan/vxlan.ko 2024-03-19T02:51:29.283 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/xen-netback 2024-03-19T02:51:29.283 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-19T02:51:29.283 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/net/xen-netfront.ko 2024-03-19T02:51:29.283 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ntb 2024-03-19T02:51:29.283 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ntb/ntb.ko 2024-03-19T02:51:29.283 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/nvme 2024-03-19T02:51:29.283 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/nvme/host 2024-03-19T02:51:29.283 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/nvme/host/nvme-core.ko 2024-03-19T02:51:29.283 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/nvme/host/nvme.ko 2024-03-19T02:51:29.283 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/parport 2024-03-19T02:51:29.283 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/parport/parport.ko 2024-03-19T02:51:29.283 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/parport/parport_cs.ko 2024-03-19T02:51:29.283 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/parport/parport_pc.ko 2024-03-19T02:51:29.283 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/parport/parport_serial.ko 2024-03-19T02:51:29.283 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pci 2024-03-19T02:51:29.284 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pci/hotplug 2024-03-19T02:51:29.284 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-19T02:51:29.284 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pci/pcie 2024-03-19T02:51:29.284 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-19T02:51:29.284 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pci/xen-pcifront.ko 2024-03-19T02:51:29.284 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pcmcia 2024-03-19T02:51:29.284 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pcmcia/i82092.ko 2024-03-19T02:51:29.284 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pcmcia/pd6729.ko 2024-03-19T02:51:29.284 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-19T02:51:29.284 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pinctrl 2024-03-19T02:51:29.284 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pinctrl/intel 2024-03-19T02:51:29.284 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-19T02:51:29.284 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform 2024-03-19T02:51:29.284 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86 2024-03-19T02:51:29.284 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-19T02:51:29.285 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/acerhdf.ko 2024-03-19T02:51:29.285 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-19T02:51:29.285 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-19T02:51:29.285 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-19T02:51:29.285 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-19T02:51:29.285 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-19T02:51:29.285 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-19T02:51:29.285 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-19T02:51:29.285 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-19T02:51:29.285 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-19T02:51:29.285 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/hdaps.ko 2024-03-19T02:51:29.285 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/intel 2024-03-19T02:51:29.285 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-19T02:51:29.285 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/intel_ips.ko 2024-03-19T02:51:29.285 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-19T02:51:29.285 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-19T02:51:29.286 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-19T02:51:29.286 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-19T02:51:29.286 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-19T02:51:29.286 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-19T02:51:29.286 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-19T02:51:29.286 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-19T02:51:29.286 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-19T02:51:29.286 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/platform/x86/wmi.ko 2024-03-19T02:51:29.286 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/powercap 2024-03-19T02:51:29.286 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-19T02:51:29.286 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-19T02:51:29.286 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pps 2024-03-19T02:51:29.286 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pps/clients 2024-03-19T02:51:29.286 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-19T02:51:29.287 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-19T02:51:29.287 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pps/clients/pps_parport.ko 2024-03-19T02:51:29.287 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/pps/pps_core.ko 2024-03-19T02:51:29.287 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ptp 2024-03-19T02:51:29.287 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ptp/ptp.ko 2024-03-19T02:51:29.287 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ptp/ptp_kvm.ko 2024-03-19T02:51:29.287 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc 2024-03-19T02:51:29.287 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-19T02:51:29.287 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-19T02:51:29.287 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-19T02:51:29.287 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-19T02:51:29.287 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-19T02:51:29.287 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-19T02:51:29.287 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-19T02:51:29.287 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-19T02:51:29.288 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-19T02:51:29.288 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-19T02:51:29.288 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-em3027.ko 2024-03-19T02:51:29.288 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-19T02:51:29.288 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-19T02:51:29.288 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-19T02:51:29.288 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-19T02:51:29.288 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-19T02:51:29.288 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-19T02:51:29.288 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-max6900.ko 2024-03-19T02:51:29.288 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-19T02:51:29.288 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-19T02:51:29.288 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-19T02:51:29.288 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-19T02:51:29.288 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-19T02:51:29.289 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-19T02:51:29.289 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-19T02:51:29.289 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-19T02:51:29.289 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-19T02:51:29.289 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-19T02:51:29.289 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-19T02:51:29.289 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-19T02:51:29.289 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/rtc/rtc-x1205.ko 2024-03-19T02:51:29.289 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi 2024-03-19T02:51:29.289 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/3w-9xxx.ko 2024-03-19T02:51:29.289 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/3w-sas.ko 2024-03-19T02:51:29.289 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/3w-xxxx.ko 2024-03-19T02:51:29.289 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/BusLogic.ko 2024-03-19T02:51:29.289 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/a100u2w.ko 2024-03-19T02:51:29.289 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/aacraid 2024-03-19T02:51:29.289 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-19T02:51:29.290 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/advansys.ko 2024-03-19T02:51:29.290 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/aic7xxx 2024-03-19T02:51:29.290 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-19T02:51:29.290 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-19T02:51:29.290 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/am53c974.ko 2024-03-19T02:51:29.290 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/arcmsr 2024-03-19T02:51:29.290 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-19T02:51:29.290 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/atp870u.ko 2024-03-19T02:51:29.290 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/be2iscsi 2024-03-19T02:51:29.290 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-19T02:51:29.290 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/bfa 2024-03-19T02:51:29.290 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/bfa/bfa.ko 2024-03-19T02:51:29.290 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/bnx2fc 2024-03-19T02:51:29.290 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-19T02:51:29.291 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/bnx2i 2024-03-19T02:51:29.291 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-19T02:51:29.291 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/ch.ko 2024-03-19T02:51:29.291 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/csiostor 2024-03-19T02:51:29.291 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-19T02:51:29.291 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/cxgbi 2024-03-19T02:51:29.291 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-19T02:51:29.291 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-19T02:51:29.291 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-19T02:51:29.291 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-19T02:51:29.291 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-19T02:51:29.291 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/dc395x.ko 2024-03-19T02:51:29.291 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/device_handler 2024-03-19T02:51:29.291 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-19T02:51:29.291 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-19T02:51:29.292 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-19T02:51:29.292 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-19T02:51:29.292 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/dmx3191d.ko 2024-03-19T02:51:29.292 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/esas2r 2024-03-19T02:51:29.292 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-19T02:51:29.292 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/esp_scsi.ko 2024-03-19T02:51:29.292 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/fcoe 2024-03-19T02:51:29.292 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-19T02:51:29.292 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-19T02:51:29.292 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/fnic 2024-03-19T02:51:29.292 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/fnic/fnic.ko 2024-03-19T02:51:29.292 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/hpsa.ko 2024-03-19T02:51:29.292 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/hptiop.ko 2024-03-19T02:51:29.292 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/hv_storvsc.ko 2024-03-19T02:51:29.292 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/initio.ko 2024-03-19T02:51:29.293 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/ipr.ko 2024-03-19T02:51:29.293 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/ips.ko 2024-03-19T02:51:29.293 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/isci 2024-03-19T02:51:29.293 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/isci/isci.ko 2024-03-19T02:51:29.293 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-19T02:51:29.293 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-19T02:51:29.293 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/libfc 2024-03-19T02:51:29.293 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/libfc/libfc.ko 2024-03-19T02:51:29.293 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/libiscsi.ko 2024-03-19T02:51:29.293 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-19T02:51:29.293 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/libsas 2024-03-19T02:51:29.293 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/libsas/libsas.ko 2024-03-19T02:51:29.293 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/lpfc 2024-03-19T02:51:29.293 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-19T02:51:29.293 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/megaraid 2024-03-19T02:51:29.294 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/megaraid.ko 2024-03-19T02:51:29.294 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-19T02:51:29.294 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-19T02:51:29.294 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-19T02:51:29.294 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/mpt3sas 2024-03-19T02:51:29.294 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-19T02:51:29.294 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/mvsas 2024-03-19T02:51:29.294 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-19T02:51:29.294 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/mvumi.ko 2024-03-19T02:51:29.294 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/pm8001 2024-03-19T02:51:29.294 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-19T02:51:29.294 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/pmcraid.ko 2024-03-19T02:51:29.294 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/qla1280.ko 2024-03-19T02:51:29.294 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/qla2xxx 2024-03-19T02:51:29.295 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-19T02:51:29.295 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-19T02:51:29.295 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/qla4xxx 2024-03-19T02:51:29.295 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-19T02:51:29.295 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/raid_class.ko 2024-03-19T02:51:29.295 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/scsi_debug.ko 2024-03-19T02:51:29.295 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-19T02:51:29.295 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-19T02:51:29.295 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-19T02:51:29.295 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-19T02:51:29.295 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-19T02:51:29.295 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/ses.ko 2024-03-19T02:51:29.295 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/st.ko 2024-03-19T02:51:29.295 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/stex.ko 2024-03-19T02:51:29.295 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/sym53c8xx_2 2024-03-19T02:51:29.296 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-19T02:51:29.296 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/virtio_scsi.ko 2024-03-19T02:51:29.296 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-19T02:51:29.296 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/scsi/wd719x.ko 2024-03-19T02:51:29.296 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ssb 2024-03-19T02:51:29.296 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ssb/ssb.ko 2024-03-19T02:51:29.296 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/target 2024-03-19T02:51:29.296 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/target/iscsi 2024-03-19T02:51:29.296 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-19T02:51:29.296 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/target/loopback 2024-03-19T02:51:29.296 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-19T02:51:29.296 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/target/target_core_file.ko 2024-03-19T02:51:29.296 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/target/target_core_iblock.ko 2024-03-19T02:51:29.296 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/target/target_core_mod.ko 2024-03-19T02:51:29.297 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/target/target_core_pscsi.ko 2024-03-19T02:51:29.297 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/target/target_core_user.ko 2024-03-19T02:51:29.297 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/target/tcm_fc 2024-03-19T02:51:29.297 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-19T02:51:29.297 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal 2024-03-19T02:51:29.297 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel 2024-03-19T02:51:29.297 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal 2024-03-19T02:51:29.297 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-19T02:51:29.297 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-19T02:51:29.297 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-19T02:51:29.297 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-19T02:51:29.297 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-19T02:51:29.297 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-19T02:51:29.297 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-19T02:51:29.297 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-19T02:51:29.297 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-19T02:51:29.298 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-19T02:51:29.298 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-19T02:51:29.298 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-19T02:51:29.298 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-19T02:51:29.298 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-19T02:51:29.298 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-19T02:51:29.298 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-19T02:51:29.298 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-19T02:51:29.298 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-19T02:51:29.298 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/tty 2024-03-19T02:51:29.298 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/tty/ipwireless 2024-03-19T02:51:29.298 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-19T02:51:29.298 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/tty/n_gsm.ko 2024-03-19T02:51:29.298 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/tty/n_hdlc.ko 2024-03-19T02:51:29.299 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/tty/nozomi.ko 2024-03-19T02:51:29.299 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/tty/serial 2024-03-19T02:51:29.299 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/tty/serial/8250 2024-03-19T02:51:29.299 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-19T02:51:29.299 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/tty/serial/arc_uart.ko 2024-03-19T02:51:29.299 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/tty/serial/jsm 2024-03-19T02:51:29.299 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-19T02:51:29.299 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ufs 2024-03-19T02:51:29.299 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ufs/core 2024-03-19T02:51:29.299 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-19T02:51:29.299 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ufs/host 2024-03-19T02:51:29.299 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-19T02:51:29.299 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/uio 2024-03-19T02:51:29.299 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/uio/uio.ko 2024-03-19T02:51:29.299 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/uio/uio_aec.ko 2024-03-19T02:51:29.300 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/uio/uio_cif.ko 2024-03-19T02:51:29.300 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/uio/uio_pci_generic.ko 2024-03-19T02:51:29.300 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/uio/uio_sercos3.ko 2024-03-19T02:51:29.300 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb 2024-03-19T02:51:29.300 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/class 2024-03-19T02:51:29.300 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/class/cdc-acm.ko 2024-03-19T02:51:29.300 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-19T02:51:29.300 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/class/usblp.ko 2024-03-19T02:51:29.300 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/class/usbtmc.ko 2024-03-19T02:51:29.300 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/host 2024-03-19T02:51:29.300 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-19T02:51:29.300 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/image 2024-03-19T02:51:29.300 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/image/mdc800.ko 2024-03-19T02:51:29.300 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/image/microtek.ko 2024-03-19T02:51:29.301 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc 2024-03-19T02:51:29.301 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/adutux.ko 2024-03-19T02:51:29.301 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/appledisplay.ko 2024-03-19T02:51:29.301 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/emi26.ko 2024-03-19T02:51:29.301 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/emi62.ko 2024-03-19T02:51:29.301 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/ezusb.ko 2024-03-19T02:51:29.301 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/idmouse.ko 2024-03-19T02:51:29.301 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/iowarrior.ko 2024-03-19T02:51:29.301 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-19T02:51:29.301 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/ldusb.ko 2024-03-19T02:51:29.301 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/legousbtower.ko 2024-03-19T02:51:29.301 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/sisusbvga 2024-03-19T02:51:29.301 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-19T02:51:29.301 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-19T02:51:29.301 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/usb3503.ko 2024-03-19T02:51:29.301 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/usblcd.ko 2024-03-19T02:51:29.302 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-19T02:51:29.302 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/uss720.ko 2024-03-19T02:51:29.302 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/misc/yurex.ko 2024-03-19T02:51:29.302 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/phy 2024-03-19T02:51:29.302 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/phy/phy-generic.ko 2024-03-19T02:51:29.302 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial 2024-03-19T02:51:29.302 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/aircable.ko 2024-03-19T02:51:29.302 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/ark3116.ko 2024-03-19T02:51:29.302 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-19T02:51:29.302 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/ch341.ko 2024-03-19T02:51:29.302 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/cp210x.ko 2024-03-19T02:51:29.302 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/cyberjack.ko 2024-03-19T02:51:29.302 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-19T02:51:29.302 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-19T02:51:29.303 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/empeg.ko 2024-03-19T02:51:29.303 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-19T02:51:29.303 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-19T02:51:29.303 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-19T02:51:29.303 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/io_ti.ko 2024-03-19T02:51:29.303 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/ipaq.ko 2024-03-19T02:51:29.303 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/ipw.ko 2024-03-19T02:51:29.303 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/ir-usb.ko 2024-03-19T02:51:29.303 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-19T02:51:29.303 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/keyspan.ko 2024-03-19T02:51:29.303 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-19T02:51:29.303 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-19T02:51:29.303 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-19T02:51:29.303 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/mct_u232.ko 2024-03-19T02:51:29.303 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/mos7720.ko 2024-03-19T02:51:29.304 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/mos7840.ko 2024-03-19T02:51:29.304 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/navman.ko 2024-03-19T02:51:29.304 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/omninet.ko 2024-03-19T02:51:29.304 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/opticon.ko 2024-03-19T02:51:29.304 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/option.ko 2024-03-19T02:51:29.304 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/oti6858.ko 2024-03-19T02:51:29.304 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/pl2303.ko 2024-03-19T02:51:29.304 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/qcaux.ko 2024-03-19T02:51:29.304 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/qcserial.ko 2024-03-19T02:51:29.304 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/quatech2.ko 2024-03-19T02:51:29.304 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/safe_serial.ko 2024-03-19T02:51:29.304 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/sierra.ko 2024-03-19T02:51:29.304 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-19T02:51:29.304 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/ssu100.ko 2024-03-19T02:51:29.304 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/symbolserial.ko 2024-03-19T02:51:29.305 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-19T02:51:29.305 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-19T02:51:29.305 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/usb_debug.ko 2024-03-19T02:51:29.305 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-19T02:51:29.305 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/visor.ko 2024-03-19T02:51:29.305 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/whiteheat.ko 2024-03-19T02:51:29.305 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-19T02:51:29.305 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage 2024-03-19T02:51:29.305 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/uas.ko 2024-03-19T02:51:29.305 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-19T02:51:29.305 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-19T02:51:29.305 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-19T02:51:29.305 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-19T02:51:29.306 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-19T02:51:29.306 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-19T02:51:29.306 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-19T02:51:29.306 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-karma.ko 2024-03-19T02:51:29.306 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-19T02:51:29.306 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-19T02:51:29.306 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-19T02:51:29.306 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-19T02:51:29.306 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-19T02:51:29.306 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/storage/usb-storage.ko 2024-03-19T02:51:29.306 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/usbip 2024-03-19T02:51:29.306 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-19T02:51:29.306 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-19T02:51:29.306 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-19T02:51:29.306 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/vfio 2024-03-19T02:51:29.307 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/vfio/pci 2024-03-19T02:51:29.307 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-19T02:51:29.307 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-19T02:51:29.307 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/vfio/vfio.ko 2024-03-19T02:51:29.307 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-19T02:51:29.307 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/vhost 2024-03-19T02:51:29.307 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/vhost/vhost.ko 2024-03-19T02:51:29.307 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-19T02:51:29.307 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/vhost/vhost_net.ko 2024-03-19T02:51:29.307 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/vhost/vhost_scsi.ko 2024-03-19T02:51:29.307 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/video 2024-03-19T02:51:29.307 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/video/backlight 2024-03-19T02:51:29.307 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/video/backlight/apple_bl.ko 2024-03-19T02:51:29.307 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/video/backlight/lcd.ko 2024-03-19T02:51:29.307 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-19T02:51:29.308 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/virtio 2024-03-19T02:51:29.308 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/virtio/virtio.ko 2024-03-19T02:51:29.308 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/virtio/virtio_balloon.ko 2024-03-19T02:51:29.308 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/virtio/virtio_mmio.ko 2024-03-19T02:51:29.308 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/virtio/virtio_pci.ko 2024-03-19T02:51:29.308 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-19T02:51:29.308 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-19T02:51:29.308 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/virtio/virtio_ring.ko 2024-03-19T02:51:29.308 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1 2024-03-19T02:51:29.308 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/masters 2024-03-19T02:51:29.308 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/masters/ds2482.ko 2024-03-19T02:51:29.308 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/masters/ds2490.ko 2024-03-19T02:51:29.308 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/slaves 2024-03-19T02:51:29.308 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-19T02:51:29.308 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-19T02:51:29.309 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-19T02:51:29.309 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-19T02:51:29.309 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-19T02:51:29.309 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-19T02:51:29.309 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-19T02:51:29.309 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-19T02:51:29.309 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-19T02:51:29.309 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-19T02:51:29.309 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-19T02:51:29.309 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/w1/wire.ko 2024-03-19T02:51:29.309 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog 2024-03-19T02:51:29.309 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-19T02:51:29.309 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-19T02:51:29.309 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-19T02:51:29.309 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/hpwdt.ko 2024-03-19T02:51:29.310 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/i6300esb.ko 2024-03-19T02:51:29.310 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-19T02:51:29.310 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-19T02:51:29.310 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/ib700wdt.ko 2024-03-19T02:51:29.310 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/ibmasr.ko 2024-03-19T02:51:29.310 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-19T02:51:29.310 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-19T02:51:29.310 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/it87_wdt.ko 2024-03-19T02:51:29.310 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/machzwd.ko 2024-03-19T02:51:29.310 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/nv_tco.ko 2024-03-19T02:51:29.310 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-19T02:51:29.310 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-19T02:51:29.310 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-19T02:51:29.310 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-19T02:51:29.310 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/softdog.ko 2024-03-19T02:51:29.311 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-19T02:51:29.311 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/via_wdt.ko 2024-03-19T02:51:29.311 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-19T02:51:29.311 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-19T02:51:29.311 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-19T02:51:29.311 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/wdt_pci.ko 2024-03-19T02:51:29.311 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/watchdog/xen_wdt.ko 2024-03-19T02:51:29.311 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/xen 2024-03-19T02:51:29.311 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-19T02:51:29.311 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/xen/xen-evtchn.ko 2024-03-19T02:51:29.311 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/xen/xen-gntalloc.ko 2024-03-19T02:51:29.311 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/xen/xen-gntdev.ko 2024-03-19T02:51:29.311 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/xen/xen-pciback 2024-03-19T02:51:29.311 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-19T02:51:29.311 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/xen/xen-privcmd.ko 2024-03-19T02:51:29.312 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/xen/xenfs 2024-03-19T02:51:29.312 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-19T02:51:29.312 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs 2024-03-19T02:51:29.312 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/9p 2024-03-19T02:51:29.312 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/9p/9p.ko 2024-03-19T02:51:29.312 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/affs 2024-03-19T02:51:29.312 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/affs/affs.ko 2024-03-19T02:51:29.312 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/befs 2024-03-19T02:51:29.312 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/befs/befs.ko 2024-03-19T02:51:29.312 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/binfmt_misc.ko 2024-03-19T02:51:29.312 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/btrfs 2024-03-19T02:51:29.312 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/btrfs/btrfs.ko 2024-03-19T02:51:29.312 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/cachefiles 2024-03-19T02:51:29.312 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/cachefiles/cachefiles.ko 2024-03-19T02:51:29.312 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ceph 2024-03-19T02:51:29.313 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ceph/ceph.ko 2024-03-19T02:51:29.313 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/coda 2024-03-19T02:51:29.313 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/coda/coda.ko 2024-03-19T02:51:29.313 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/cramfs 2024-03-19T02:51:29.313 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/cramfs/cramfs.ko 2024-03-19T02:51:29.313 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/dlm 2024-03-19T02:51:29.313 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/dlm/dlm.ko 2024-03-19T02:51:29.313 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ecryptfs 2024-03-19T02:51:29.313 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-19T02:51:29.313 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/f2fs 2024-03-19T02:51:29.313 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/f2fs/f2fs.ko 2024-03-19T02:51:29.313 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/fat 2024-03-19T02:51:29.313 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/fat/fat.ko 2024-03-19T02:51:29.313 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/fat/msdos.ko 2024-03-19T02:51:29.313 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/fat/vfat.ko 2024-03-19T02:51:29.314 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/fuse 2024-03-19T02:51:29.314 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/fuse/cuse.ko 2024-03-19T02:51:29.314 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/fuse/fuse.ko 2024-03-19T02:51:29.314 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/gfs2 2024-03-19T02:51:29.314 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/gfs2/gfs2.ko 2024-03-19T02:51:29.314 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/hfs 2024-03-19T02:51:29.314 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/hfs/hfs.ko 2024-03-19T02:51:29.314 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/hfsplus 2024-03-19T02:51:29.314 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/hfsplus/hfsplus.ko 2024-03-19T02:51:29.314 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/isofs 2024-03-19T02:51:29.314 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/isofs/isofs.ko 2024-03-19T02:51:29.314 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/jfs 2024-03-19T02:51:29.314 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/jfs/jfs.ko 2024-03-19T02:51:29.314 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/lockd 2024-03-19T02:51:29.315 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/lockd/lockd.ko 2024-03-19T02:51:29.315 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/minix 2024-03-19T02:51:29.315 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/minix/minix.ko 2024-03-19T02:51:29.315 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/netfs 2024-03-19T02:51:29.315 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/netfs/netfs.ko 2024-03-19T02:51:29.315 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs 2024-03-19T02:51:29.315 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs/blocklayout 2024-03-19T02:51:29.315 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-19T02:51:29.315 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs/filelayout 2024-03-19T02:51:29.315 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-19T02:51:29.315 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs/flexfilelayout 2024-03-19T02:51:29.315 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-19T02:51:29.315 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs/nfs.ko 2024-03-19T02:51:29.315 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs/nfsv3.ko 2024-03-19T02:51:29.315 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs/nfsv4.ko 2024-03-19T02:51:29.316 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs_common 2024-03-19T02:51:29.316 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs_common/grace.ko 2024-03-19T02:51:29.316 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfs_common/nfs_acl.ko 2024-03-19T02:51:29.316 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfsd 2024-03-19T02:51:29.316 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nfsd/nfsd.ko 2024-03-19T02:51:29.316 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nilfs2 2024-03-19T02:51:29.316 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nilfs2/nilfs2.ko 2024-03-19T02:51:29.316 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls 2024-03-19T02:51:29.316 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/mac-celtic.ko 2024-03-19T02:51:29.316 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/mac-centeuro.ko 2024-03-19T02:51:29.316 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/mac-croatian.ko 2024-03-19T02:51:29.316 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/mac-cyrillic.ko 2024-03-19T02:51:29.316 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/mac-gaelic.ko 2024-03-19T02:51:29.316 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/mac-greek.ko 2024-03-19T02:51:29.316 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/mac-iceland.ko 2024-03-19T02:51:29.316 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/mac-inuit.ko 2024-03-19T02:51:29.317 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/mac-roman.ko 2024-03-19T02:51:29.317 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/mac-romanian.ko 2024-03-19T02:51:29.317 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/mac-turkish.ko 2024-03-19T02:51:29.317 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp1250.ko 2024-03-19T02:51:29.317 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp1251.ko 2024-03-19T02:51:29.317 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp1255.ko 2024-03-19T02:51:29.317 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp737.ko 2024-03-19T02:51:29.317 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp775.ko 2024-03-19T02:51:29.317 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp850.ko 2024-03-19T02:51:29.317 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp852.ko 2024-03-19T02:51:29.317 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp855.ko 2024-03-19T02:51:29.317 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp857.ko 2024-03-19T02:51:29.317 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp860.ko 2024-03-19T02:51:29.317 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp861.ko 2024-03-19T02:51:29.317 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp862.ko 2024-03-19T02:51:29.318 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp863.ko 2024-03-19T02:51:29.318 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp864.ko 2024-03-19T02:51:29.318 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp865.ko 2024-03-19T02:51:29.318 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp866.ko 2024-03-19T02:51:29.318 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp869.ko 2024-03-19T02:51:29.318 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp874.ko 2024-03-19T02:51:29.318 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp932.ko 2024-03-19T02:51:29.318 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp936.ko 2024-03-19T02:51:29.318 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp949.ko 2024-03-19T02:51:29.318 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_cp950.ko 2024-03-19T02:51:29.318 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_euc-jp.ko 2024-03-19T02:51:29.318 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_iso8859-1.ko 2024-03-19T02:51:29.318 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_iso8859-13.ko 2024-03-19T02:51:29.318 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_iso8859-14.ko 2024-03-19T02:51:29.318 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_iso8859-15.ko 2024-03-19T02:51:29.319 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_iso8859-2.ko 2024-03-19T02:51:29.319 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_iso8859-3.ko 2024-03-19T02:51:29.319 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_iso8859-4.ko 2024-03-19T02:51:29.319 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_iso8859-5.ko 2024-03-19T02:51:29.319 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_iso8859-6.ko 2024-03-19T02:51:29.319 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_iso8859-7.ko 2024-03-19T02:51:29.319 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_iso8859-9.ko 2024-03-19T02:51:29.319 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_koi8-r.ko 2024-03-19T02:51:29.319 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_koi8-ru.ko 2024-03-19T02:51:29.319 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_koi8-u.ko 2024-03-19T02:51:29.319 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-19T02:51:29.319 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/nls/nls_utf8.ko 2024-03-19T02:51:29.319 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ocfs2 2024-03-19T02:51:29.319 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ocfs2/cluster 2024-03-19T02:51:29.319 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-19T02:51:29.320 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ocfs2/dlm 2024-03-19T02:51:29.320 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-19T02:51:29.320 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ocfs2/dlmfs 2024-03-19T02:51:29.320 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-19T02:51:29.320 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ocfs2/ocfs2.ko 2024-03-19T02:51:29.320 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-19T02:51:29.320 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-19T02:51:29.321 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-19T02:51:29.321 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/overlayfs 2024-03-19T02:51:29.322 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/overlayfs/overlay.ko 2024-03-19T02:51:29.322 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/pstore 2024-03-19T02:51:29.322 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/pstore/ramoops.ko 2024-03-19T02:51:29.322 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/reiserfs 2024-03-19T02:51:29.322 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/reiserfs/reiserfs.ko 2024-03-19T02:51:29.322 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/romfs 2024-03-19T02:51:29.322 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/romfs/romfs.ko 2024-03-19T02:51:29.322 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/smb 2024-03-19T02:51:29.322 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/smb/client 2024-03-19T02:51:29.322 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/smb/client/cifs.ko 2024-03-19T02:51:29.322 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/smb/common 2024-03-19T02:51:29.322 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/smb/common/cifs_arc4.ko 2024-03-19T02:51:29.322 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/smb/common/cifs_md4.ko 2024-03-19T02:51:29.322 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/squashfs 2024-03-19T02:51:29.322 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/squashfs/squashfs.ko 2024-03-19T02:51:29.323 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/sysv 2024-03-19T02:51:29.323 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/sysv/sysv.ko 2024-03-19T02:51:29.323 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/udf 2024-03-19T02:51:29.323 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/udf/udf.ko 2024-03-19T02:51:29.323 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ufs 2024-03-19T02:51:29.323 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/ufs/ufs.ko 2024-03-19T02:51:29.323 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/xfs 2024-03-19T02:51:29.323 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/fs/xfs/xfs.ko 2024-03-19T02:51:29.323 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/kernel 2024-03-19T02:51:29.323 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/kernel/rcu 2024-03-19T02:51:29.323 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/kernel/rcu/rcutorture.ko 2024-03-19T02:51:29.323 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/kernel/torture.ko 2024-03-19T02:51:29.323 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/kernel/trace 2024-03-19T02:51:29.323 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-19T02:51:29.323 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib 2024-03-19T02:51:29.324 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/asn1_encoder.ko 2024-03-19T02:51:29.324 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/crc-itu-t.ko 2024-03-19T02:51:29.324 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/crc8.ko 2024-03-19T02:51:29.324 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/crypto 2024-03-19T02:51:29.324 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/crypto/libarc4.ko 2024-03-19T02:51:29.324 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/crypto/libdes.ko 2024-03-19T02:51:29.324 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/libcrc32c.ko 2024-03-19T02:51:29.324 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/lru_cache.ko 2024-03-19T02:51:29.324 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/lz4 2024-03-19T02:51:29.324 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/lz4/lz4_compress.ko 2024-03-19T02:51:29.324 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/lz4/lz4hc_compress.ko 2024-03-19T02:51:29.324 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/math 2024-03-19T02:51:29.324 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/math/cordic.ko 2024-03-19T02:51:29.324 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/raid6 2024-03-19T02:51:29.325 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/raid6/raid6_pq.ko 2024-03-19T02:51:29.325 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/reed_solomon 2024-03-19T02:51:29.325 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-19T02:51:29.325 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/ts_bm.ko 2024-03-19T02:51:29.325 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/ts_fsm.ko 2024-03-19T02:51:29.325 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/ts_kmp.ko 2024-03-19T02:51:29.325 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/zstd 2024-03-19T02:51:29.325 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/lib/zstd/zstd_compress.ko 2024-03-19T02:51:29.325 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/mm 2024-03-19T02:51:29.325 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/mm/hwpoison-inject.ko 2024-03-19T02:51:29.325 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net 2024-03-19T02:51:29.325 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/6lowpan 2024-03-19T02:51:29.325 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/6lowpan/6lowpan.ko 2024-03-19T02:51:29.325 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/6lowpan/nhc_dest.ko 2024-03-19T02:51:29.325 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/6lowpan/nhc_fragment.ko 2024-03-19T02:51:29.326 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/6lowpan/nhc_hop.ko 2024-03-19T02:51:29.326 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-19T02:51:29.326 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/6lowpan/nhc_mobility.ko 2024-03-19T02:51:29.326 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/6lowpan/nhc_routing.ko 2024-03-19T02:51:29.326 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/6lowpan/nhc_udp.ko 2024-03-19T02:51:29.326 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/802 2024-03-19T02:51:29.326 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/802/garp.ko 2024-03-19T02:51:29.326 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/802/mrp.ko 2024-03-19T02:51:29.326 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/802/p8022.ko 2024-03-19T02:51:29.326 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/802/psnap.ko 2024-03-19T02:51:29.326 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/802/stp.ko 2024-03-19T02:51:29.326 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/8021q 2024-03-19T02:51:29.326 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/8021q/8021q.ko 2024-03-19T02:51:29.326 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/9p 2024-03-19T02:51:29.326 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/9p/9pnet.ko 2024-03-19T02:51:29.327 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/9p/9pnet_fd.ko 2024-03-19T02:51:29.327 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/9p/9pnet_rdma.ko 2024-03-19T02:51:29.327 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/9p/9pnet_virtio.ko 2024-03-19T02:51:29.327 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/batman-adv 2024-03-19T02:51:29.327 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/batman-adv/batman-adv.ko 2024-03-19T02:51:29.327 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge 2024-03-19T02:51:29.327 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/br_netfilter.ko 2024-03-19T02:51:29.327 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/bridge.ko 2024-03-19T02:51:29.327 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter 2024-03-19T02:51:29.327 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-19T02:51:29.327 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-19T02:51:29.327 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-19T02:51:29.327 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-19T02:51:29.327 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-19T02:51:29.327 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-19T02:51:29.328 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-19T02:51:29.328 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-19T02:51:29.328 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-19T02:51:29.328 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-19T02:51:29.328 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-19T02:51:29.328 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-19T02:51:29.328 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-19T02:51:29.328 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-19T02:51:29.328 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-19T02:51:29.328 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-19T02:51:29.328 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-19T02:51:29.328 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-19T02:51:29.328 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-19T02:51:29.328 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-19T02:51:29.328 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/ebtables.ko 2024-03-19T02:51:29.329 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-19T02:51:29.329 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-19T02:51:29.329 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-19T02:51:29.329 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ceph 2024-03-19T02:51:29.329 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ceph/libceph.ko 2024-03-19T02:51:29.329 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/core 2024-03-19T02:51:29.329 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/core/failover.ko 2024-03-19T02:51:29.329 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/core/pktgen.ko 2024-03-19T02:51:29.329 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dccp 2024-03-19T02:51:29.329 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dccp/dccp.ko 2024-03-19T02:51:29.329 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dccp/dccp_diag.ko 2024-03-19T02:51:29.329 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dccp/dccp_ipv4.ko 2024-03-19T02:51:29.329 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dccp/dccp_ipv6.ko 2024-03-19T02:51:29.329 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dns_resolver 2024-03-19T02:51:29.329 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dns_resolver/dns_resolver.ko 2024-03-19T02:51:29.330 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dsa 2024-03-19T02:51:29.330 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dsa/dsa_core.ko 2024-03-19T02:51:29.330 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dsa/tag_brcm.ko 2024-03-19T02:51:29.330 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dsa/tag_dsa.ko 2024-03-19T02:51:29.330 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dsa/tag_none.ko 2024-03-19T02:51:29.330 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/dsa/tag_trailer.ko 2024-03-19T02:51:29.330 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ieee802154 2024-03-19T02:51:29.330 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ieee802154/6lowpan 2024-03-19T02:51:29.330 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-19T02:51:29.330 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ieee802154/ieee802154.ko 2024-03-19T02:51:29.330 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-19T02:51:29.330 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4 2024-03-19T02:51:29.330 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/ah4.ko 2024-03-19T02:51:29.330 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/esp4.ko 2024-03-19T02:51:29.330 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/fou.ko 2024-03-19T02:51:29.331 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/gre.ko 2024-03-19T02:51:29.331 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/inet_diag.ko 2024-03-19T02:51:29.331 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/ip_gre.ko 2024-03-19T02:51:29.331 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/ip_tunnel.ko 2024-03-19T02:51:29.331 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/ip_vti.ko 2024-03-19T02:51:29.331 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/ipcomp.ko 2024-03-19T02:51:29.331 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/ipip.ko 2024-03-19T02:51:29.331 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter 2024-03-19T02:51:29.331 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-19T02:51:29.331 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-19T02:51:29.331 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-19T02:51:29.331 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-19T02:51:29.331 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-19T02:51:29.331 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-19T02:51:29.331 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-19T02:51:29.332 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-19T02:51:29.332 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-19T02:51:29.332 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-19T02:51:29.332 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-19T02:51:29.332 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-19T02:51:29.332 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-19T02:51:29.332 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-19T02:51:29.332 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-19T02:51:29.332 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-19T02:51:29.332 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-19T02:51:29.332 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-19T02:51:29.332 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-19T02:51:29.332 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-19T02:51:29.332 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-19T02:51:29.332 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_bic.ko 2024-03-19T02:51:29.333 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_dctcp.ko 2024-03-19T02:51:29.333 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_diag.ko 2024-03-19T02:51:29.333 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_highspeed.ko 2024-03-19T02:51:29.333 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_htcp.ko 2024-03-19T02:51:29.333 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_hybla.ko 2024-03-19T02:51:29.333 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_illinois.ko 2024-03-19T02:51:29.333 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_lp.ko 2024-03-19T02:51:29.333 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_scalable.ko 2024-03-19T02:51:29.333 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_vegas.ko 2024-03-19T02:51:29.333 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_veno.ko 2024-03-19T02:51:29.333 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_westwood.ko 2024-03-19T02:51:29.333 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tcp_yeah.ko 2024-03-19T02:51:29.333 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/tunnel4.ko 2024-03-19T02:51:29.333 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/udp_diag.ko 2024-03-19T02:51:29.334 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/udp_tunnel.ko 2024-03-19T02:51:29.334 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-19T02:51:29.334 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6 2024-03-19T02:51:29.334 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/ah6.ko 2024-03-19T02:51:29.334 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/esp6.ko 2024-03-19T02:51:29.334 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/fou6.ko 2024-03-19T02:51:29.334 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/ip6_tunnel.ko 2024-03-19T02:51:29.334 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-19T02:51:29.334 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/ip6_vti.ko 2024-03-19T02:51:29.334 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/ipcomp6.ko 2024-03-19T02:51:29.334 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter 2024-03-19T02:51:29.334 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-19T02:51:29.334 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-19T02:51:29.334 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-19T02:51:29.335 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-19T02:51:29.335 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-19T02:51:29.335 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-19T02:51:29.335 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-19T02:51:29.335 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-19T02:51:29.335 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-19T02:51:29.335 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-19T02:51:29.335 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-19T02:51:29.335 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-19T02:51:29.335 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-19T02:51:29.335 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-19T02:51:29.335 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-19T02:51:29.335 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-19T02:51:29.335 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-19T02:51:29.335 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-19T02:51:29.336 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-19T02:51:29.336 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-19T02:51:29.336 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-19T02:51:29.336 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-19T02:51:29.336 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-19T02:51:29.336 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-19T02:51:29.336 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/sit.ko 2024-03-19T02:51:29.336 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/tunnel6.ko 2024-03-19T02:51:29.336 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-19T02:51:29.336 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/key 2024-03-19T02:51:29.336 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/key/af_key.ko 2024-03-19T02:51:29.336 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/l2tp 2024-03-19T02:51:29.336 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/l2tp/l2tp_core.ko 2024-03-19T02:51:29.336 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-19T02:51:29.336 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/l2tp/l2tp_eth.ko 2024-03-19T02:51:29.337 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/l2tp/l2tp_ip.ko 2024-03-19T02:51:29.337 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/l2tp/l2tp_ip6.ko 2024-03-19T02:51:29.337 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/l2tp/l2tp_netlink.ko 2024-03-19T02:51:29.337 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/l2tp/l2tp_ppp.ko 2024-03-19T02:51:29.337 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/llc 2024-03-19T02:51:29.337 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/llc/llc.ko 2024-03-19T02:51:29.337 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/mac802154 2024-03-19T02:51:29.337 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/mac802154/mac802154.ko 2024-03-19T02:51:29.337 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/mpls 2024-03-19T02:51:29.337 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/mpls/mpls_gso.ko 2024-03-19T02:51:29.337 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter 2024-03-19T02:51:29.337 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset 2024-03-19T02:51:29.337 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set.ko 2024-03-19T02:51:29.337 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-19T02:51:29.338 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-19T02:51:29.338 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-19T02:51:29.338 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-19T02:51:29.338 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-19T02:51:29.338 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-19T02:51:29.338 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-19T02:51:29.338 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-19T02:51:29.338 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-19T02:51:29.338 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-19T02:51:29.338 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-19T02:51:29.338 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-19T02:51:29.338 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-19T02:51:29.338 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-19T02:51:29.338 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-19T02:51:29.339 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs 2024-03-19T02:51:29.339 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-19T02:51:29.339 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-19T02:51:29.339 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-19T02:51:29.339 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-19T02:51:29.339 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-19T02:51:29.339 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-19T02:51:29.339 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-19T02:51:29.339 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-19T02:51:29.339 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-19T02:51:29.339 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-19T02:51:29.339 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-19T02:51:29.339 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-19T02:51:29.339 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-19T02:51:29.340 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-19T02:51:29.340 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conncount.ko 2024-03-19T02:51:29.340 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack.ko 2024-03-19T02:51:29.340 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-19T02:51:29.340 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-19T02:51:29.340 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-19T02:51:29.340 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-19T02:51:29.340 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-19T02:51:29.340 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-19T02:51:29.340 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-19T02:51:29.340 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-19T02:51:29.340 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-19T02:51:29.340 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-19T02:51:29.340 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-19T02:51:29.340 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-19T02:51:29.341 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-19T02:51:29.341 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_flow_table.ko 2024-03-19T02:51:29.341 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-19T02:51:29.341 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_log_syslog.ko 2024-03-19T02:51:29.341 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_nat.ko 2024-03-19T02:51:29.341 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-19T02:51:29.341 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-19T02:51:29.341 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_nat_irc.ko 2024-03-19T02:51:29.341 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_nat_sip.ko 2024-03-19T02:51:29.341 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-19T02:51:29.341 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-19T02:51:29.341 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nf_tables.ko 2024-03-19T02:51:29.341 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nfnetlink.ko 2024-03-19T02:51:29.341 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-19T02:51:29.341 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nfnetlink_log.ko 2024-03-19T02:51:29.342 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-19T02:51:29.342 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-19T02:51:29.342 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_chain_nat.ko 2024-03-19T02:51:29.342 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_compat.ko 2024-03-19T02:51:29.342 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_ct.ko 2024-03-19T02:51:29.342 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-19T02:51:29.342 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_fib.ko 2024-03-19T02:51:29.342 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-19T02:51:29.342 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_hash.ko 2024-03-19T02:51:29.342 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_limit.ko 2024-03-19T02:51:29.342 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_log.ko 2024-03-19T02:51:29.342 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_masq.ko 2024-03-19T02:51:29.342 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_nat.ko 2024-03-19T02:51:29.342 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_numgen.ko 2024-03-19T02:51:29.342 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_queue.ko 2024-03-19T02:51:29.343 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_quota.ko 2024-03-19T02:51:29.343 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_redir.ko 2024-03-19T02:51:29.343 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_reject.ko 2024-03-19T02:51:29.343 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_reject_inet.ko 2024-03-19T02:51:29.343 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_socket.ko 2024-03-19T02:51:29.343 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/nft_tunnel.ko 2024-03-19T02:51:29.343 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_AUDIT.ko 2024-03-19T02:51:29.343 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-19T02:51:29.343 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-19T02:51:29.343 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-19T02:51:29.343 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_CT.ko 2024-03-19T02:51:29.343 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_DSCP.ko 2024-03-19T02:51:29.343 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_HL.ko 2024-03-19T02:51:29.343 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_HMARK.ko 2024-03-19T02:51:29.344 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-19T02:51:29.344 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_LED.ko 2024-03-19T02:51:29.344 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_LOG.ko 2024-03-19T02:51:29.344 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-19T02:51:29.344 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_NETMAP.ko 2024-03-19T02:51:29.344 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_NFLOG.ko 2024-03-19T02:51:29.344 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-19T02:51:29.344 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_RATEEST.ko 2024-03-19T02:51:29.344 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-19T02:51:29.344 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_SECMARK.ko 2024-03-19T02:51:29.344 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-19T02:51:29.344 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-19T02:51:29.344 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_TEE.ko 2024-03-19T02:51:29.344 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_TPROXY.ko 2024-03-19T02:51:29.344 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_TRACE.ko 2024-03-19T02:51:29.345 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_addrtype.ko 2024-03-19T02:51:29.345 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_bpf.ko 2024-03-19T02:51:29.345 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_cgroup.ko 2024-03-19T02:51:29.345 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_cluster.ko 2024-03-19T02:51:29.345 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_comment.ko 2024-03-19T02:51:29.345 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_connbytes.ko 2024-03-19T02:51:29.345 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_connlabel.ko 2024-03-19T02:51:29.345 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_connlimit.ko 2024-03-19T02:51:29.345 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_connmark.ko 2024-03-19T02:51:29.345 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_conntrack.ko 2024-03-19T02:51:29.345 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_cpu.ko 2024-03-19T02:51:29.345 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_dccp.ko 2024-03-19T02:51:29.345 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_devgroup.ko 2024-03-19T02:51:29.345 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_dscp.ko 2024-03-19T02:51:29.345 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_ecn.ko 2024-03-19T02:51:29.346 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_esp.ko 2024-03-19T02:51:29.346 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_hashlimit.ko 2024-03-19T02:51:29.346 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_helper.ko 2024-03-19T02:51:29.346 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_hl.ko 2024-03-19T02:51:29.346 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_ipcomp.ko 2024-03-19T02:51:29.346 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_iprange.ko 2024-03-19T02:51:29.346 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_ipvs.ko 2024-03-19T02:51:29.346 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_l2tp.ko 2024-03-19T02:51:29.346 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_length.ko 2024-03-19T02:51:29.346 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_limit.ko 2024-03-19T02:51:29.346 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_mac.ko 2024-03-19T02:51:29.346 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_mark.ko 2024-03-19T02:51:29.346 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_multiport.ko 2024-03-19T02:51:29.346 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_nat.ko 2024-03-19T02:51:29.346 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_nfacct.ko 2024-03-19T02:51:29.347 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_osf.ko 2024-03-19T02:51:29.347 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_owner.ko 2024-03-19T02:51:29.347 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_physdev.ko 2024-03-19T02:51:29.347 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_pkttype.ko 2024-03-19T02:51:29.347 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_policy.ko 2024-03-19T02:51:29.347 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_quota.ko 2024-03-19T02:51:29.347 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_rateest.ko 2024-03-19T02:51:29.347 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_realm.ko 2024-03-19T02:51:29.347 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_recent.ko 2024-03-19T02:51:29.347 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_sctp.ko 2024-03-19T02:51:29.347 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_set.ko 2024-03-19T02:51:29.347 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_socket.ko 2024-03-19T02:51:29.347 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_state.ko 2024-03-19T02:51:29.347 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_statistic.ko 2024-03-19T02:51:29.348 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_string.ko 2024-03-19T02:51:29.348 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_tcpmss.ko 2024-03-19T02:51:29.348 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_time.ko 2024-03-19T02:51:29.348 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netfilter/xt_u32.ko 2024-03-19T02:51:29.348 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netlink 2024-03-19T02:51:29.348 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/netlink/netlink_diag.ko 2024-03-19T02:51:29.348 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/nsh 2024-03-19T02:51:29.348 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/nsh/nsh.ko 2024-03-19T02:51:29.348 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/openvswitch 2024-03-19T02:51:29.348 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/openvswitch/openvswitch.ko 2024-03-19T02:51:29.348 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/openvswitch/vport-geneve.ko 2024-03-19T02:51:29.348 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/openvswitch/vport-gre.ko 2024-03-19T02:51:29.348 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/openvswitch/vport-vxlan.ko 2024-03-19T02:51:29.348 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/packet 2024-03-19T02:51:29.349 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/packet/af_packet_diag.ko 2024-03-19T02:51:29.349 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/rds 2024-03-19T02:51:29.349 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/rds/rds.ko 2024-03-19T02:51:29.349 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/rds/rds_rdma.ko 2024-03-19T02:51:29.349 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/rds/rds_tcp.ko 2024-03-19T02:51:29.349 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched 2024-03-19T02:51:29.349 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/act_csum.ko 2024-03-19T02:51:29.349 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/act_gact.ko 2024-03-19T02:51:29.349 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/act_mirred.ko 2024-03-19T02:51:29.349 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/act_nat.ko 2024-03-19T02:51:29.349 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/act_pedit.ko 2024-03-19T02:51:29.349 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/act_police.ko 2024-03-19T02:51:29.349 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/act_simple.ko 2024-03-19T02:51:29.349 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/act_skbedit.ko 2024-03-19T02:51:29.349 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/act_vlan.ko 2024-03-19T02:51:29.350 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/cls_basic.ko 2024-03-19T02:51:29.350 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/cls_bpf.ko 2024-03-19T02:51:29.350 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/cls_flow.ko 2024-03-19T02:51:29.350 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/cls_fw.ko 2024-03-19T02:51:29.350 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/cls_route.ko 2024-03-19T02:51:29.350 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/cls_u32.ko 2024-03-19T02:51:29.350 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/em_cmp.ko 2024-03-19T02:51:29.350 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/em_ipset.ko 2024-03-19T02:51:29.350 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/em_meta.ko 2024-03-19T02:51:29.350 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/em_nbyte.ko 2024-03-19T02:51:29.350 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/em_text.ko 2024-03-19T02:51:29.350 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/em_u32.ko 2024-03-19T02:51:29.350 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_choke.ko 2024-03-19T02:51:29.350 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_codel.ko 2024-03-19T02:51:29.351 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_drr.ko 2024-03-19T02:51:29.351 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_fq.ko 2024-03-19T02:51:29.351 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_gred.ko 2024-03-19T02:51:29.351 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_hfsc.ko 2024-03-19T02:51:29.351 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_hhf.ko 2024-03-19T02:51:29.351 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_htb.ko 2024-03-19T02:51:29.351 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_ingress.ko 2024-03-19T02:51:29.351 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_mqprio.ko 2024-03-19T02:51:29.351 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_mqprio_lib.ko 2024-03-19T02:51:29.351 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_multiq.ko 2024-03-19T02:51:29.351 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_netem.ko 2024-03-19T02:51:29.351 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_pie.ko 2024-03-19T02:51:29.351 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_plug.ko 2024-03-19T02:51:29.351 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_prio.ko 2024-03-19T02:51:29.351 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_qfq.ko 2024-03-19T02:51:29.352 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_red.ko 2024-03-19T02:51:29.352 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_sfb.ko 2024-03-19T02:51:29.352 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_sfq.ko 2024-03-19T02:51:29.352 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_tbf.ko 2024-03-19T02:51:29.352 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sched/sch_teql.ko 2024-03-19T02:51:29.352 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sctp 2024-03-19T02:51:29.352 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sctp/sctp.ko 2024-03-19T02:51:29.352 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sctp/sctp_diag.ko 2024-03-19T02:51:29.352 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sunrpc 2024-03-19T02:51:29.352 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sunrpc/auth_gss 2024-03-19T02:51:29.352 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-19T02:51:29.352 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-19T02:51:29.352 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sunrpc/sunrpc.ko 2024-03-19T02:51:29.352 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sunrpc/xprtrdma 2024-03-19T02:51:29.352 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-19T02:51:29.353 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/tipc 2024-03-19T02:51:29.353 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/tipc/diag.ko 2024-03-19T02:51:29.353 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/tipc/tipc.ko 2024-03-19T02:51:29.353 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/unix 2024-03-19T02:51:29.353 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/unix/unix_diag.ko 2024-03-19T02:51:29.353 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/vmw_vsock 2024-03-19T02:51:29.353 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-19T02:51:29.353 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-19T02:51:29.353 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/vmw_vsock/vsock.ko 2024-03-19T02:51:29.353 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-19T02:51:29.353 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-19T02:51:29.353 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/xfrm 2024-03-19T02:51:29.353 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-19T02:51:29.353 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/security 2024-03-19T02:51:29.353 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/security/keys 2024-03-19T02:51:29.354 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/security/keys/encrypted-keys 2024-03-19T02:51:29.354 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-19T02:51:29.354 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/security/keys/trusted-keys 2024-03-19T02:51:29.354 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/security/keys/trusted-keys/trusted.ko 2024-03-19T02:51:29.354 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/virt 2024-03-19T02:51:29.354 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/virt/lib 2024-03-19T02:51:29.354 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/kernel/virt/lib/irqbypass.ko 2024-03-19T02:51:29.354 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.alias 2024-03-19T02:51:29.354 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.alias.bin 2024-03-19T02:51:29.354 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.builtin 2024-03-19T02:51:29.354 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.builtin.alias.bin 2024-03-19T02:51:29.354 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.builtin.bin 2024-03-19T02:51:29.354 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.builtin.modinfo 2024-03-19T02:51:29.354 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.dep 2024-03-19T02:51:29.355 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.dep.bin 2024-03-19T02:51:29.355 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.devname 2024-03-19T02:51:29.355 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.order 2024-03-19T02:51:29.355 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.softdep 2024-03-19T02:51:29.355 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.symbols 2024-03-19T02:51:29.355 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/modules.symbols.bin 2024-03-19T02:51:29.355 INFO:teuthology.orchestra.run.smithi117.stdout:/lib/modules/6.8.0-g908b6b276e8f/vmlinuz 2024-03-19T02:51:29.359 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g908b6b276e8f 2024-03-19T02:51:29.359 DEBUG:teuthology.orchestra.run.smithi117:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-19T02:51:29.380 INFO:teuthology.orchestra.run.smithi117.stdout:postinstall scriptlet (using /bin/sh): 2024-03-19T02:51:29.381 INFO:teuthology.orchestra.run.smithi117.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-19T02:51:29.381 INFO:teuthology.orchestra.run.smithi117.stdout: /usr/bin/kernel-install add 6.8.0-g908b6b276e8f /lib/modules/6.8.0-g908b6b276e8f/vmlinuz 2024-03-19T02:51:29.381 INFO:teuthology.orchestra.run.smithi117.stdout:fi 2024-03-19T02:51:29.381 INFO:teuthology.orchestra.run.smithi117.stdout:for file in vmlinuz System.map config; do 2024-03-19T02:51:29.381 INFO:teuthology.orchestra.run.smithi117.stdout: if ! cmp --silent "/lib/modules/6.8.0-g908b6b276e8f/${file}" "/boot/${file}-6.8.0-g908b6b276e8f"; then 2024-03-19T02:51:29.381 INFO:teuthology.orchestra.run.smithi117.stdout: cp "/lib/modules/6.8.0-g908b6b276e8f/${file}" "/boot/${file}-6.8.0-g908b6b276e8f" 2024-03-19T02:51:29.381 INFO:teuthology.orchestra.run.smithi117.stdout: fi 2024-03-19T02:51:29.381 INFO:teuthology.orchestra.run.smithi117.stdout:done 2024-03-19T02:51:29.381 INFO:teuthology.orchestra.run.smithi117.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-19T02:51:29.381 INFO:teuthology.orchestra.run.smithi117.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-19T02:51:29.381 INFO:teuthology.orchestra.run.smithi117.stdout:new-kernel-pkg --remove 6.8.0-g908b6b276e8f --rminitrd --initrdfile=/boot/initramfs-6.8.0-g908b6b276e8f.img 2024-03-19T02:51:29.381 INFO:teuthology.orchestra.run.smithi117.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-19T02:51:29.382 INFO:teuthology.orchestra.run.smithi117.stdout:kernel-install remove 6.8.0-g908b6b276e8f 2024-03-19T02:51:29.382 INFO:teuthology.orchestra.run.smithi117.stdout:fi 2024-03-19T02:51:29.382 INFO:teuthology.orchestra.run.smithi117.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-19T02:51:29.382 INFO:teuthology.orchestra.run.smithi117.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-19T02:51:29.382 INFO:teuthology.orchestra.run.smithi117.stdout:/sbin/update-bootloader --remove 6.8.0-g908b6b276e8f 2024-03-19T02:51:29.382 INFO:teuthology.orchestra.run.smithi117.stdout:fi 2024-03-19T02:51:29.383 DEBUG:teuthology.orchestra.run.smithi117:> sudo rpm -qi grub2-tools 2024-03-19T02:51:29.458 INFO:teuthology.orchestra.run.smithi117.stdout:Name : grub2-tools 2024-03-19T02:51:29.458 INFO:teuthology.orchestra.run.smithi117.stdout:Epoch : 1 2024-03-19T02:51:29.458 INFO:teuthology.orchestra.run.smithi117.stdout:Version : 2.06 2024-03-19T02:51:29.458 INFO:teuthology.orchestra.run.smithi117.stdout:Release : 68.el9 2024-03-19T02:51:29.458 INFO:teuthology.orchestra.run.smithi117.stdout:Architecture: x86_64 2024-03-19T02:51:29.458 INFO:teuthology.orchestra.run.smithi117.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-19T02:51:29.458 INFO:teuthology.orchestra.run.smithi117.stdout:Group : Unspecified 2024-03-19T02:51:29.459 INFO:teuthology.orchestra.run.smithi117.stdout:Size : 8264254 2024-03-19T02:51:29.459 INFO:teuthology.orchestra.run.smithi117.stdout:License : GPLv3+ 2024-03-19T02:51:29.459 INFO:teuthology.orchestra.run.smithi117.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-19T02:51:29.459 INFO:teuthology.orchestra.run.smithi117.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-19T02:51:29.459 INFO:teuthology.orchestra.run.smithi117.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-19T02:51:29.459 INFO:teuthology.orchestra.run.smithi117.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-19T02:51:29.459 INFO:teuthology.orchestra.run.smithi117.stdout:Packager : builder@centos.org 2024-03-19T02:51:29.459 INFO:teuthology.orchestra.run.smithi117.stdout:Vendor : CentOS 2024-03-19T02:51:29.459 INFO:teuthology.orchestra.run.smithi117.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-19T02:51:29.459 INFO:teuthology.orchestra.run.smithi117.stdout:Summary : Support tools for GRUB. 2024-03-19T02:51:29.459 INFO:teuthology.orchestra.run.smithi117.stdout:Description : 2024-03-19T02:51:29.459 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T02:51:29.459 INFO:teuthology.orchestra.run.smithi117.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-19T02:51:29.459 INFO:teuthology.orchestra.run.smithi117.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-19T02:51:29.459 INFO:teuthology.orchestra.run.smithi117.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-19T02:51:29.460 INFO:teuthology.orchestra.run.smithi117.stdout:hardware devices. 2024-03-19T02:51:29.460 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T02:51:29.460 INFO:teuthology.orchestra.run.smithi117.stdout:This subpackage provides tools for support of all platforms. 2024-03-19T02:51:29.461 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-19T02:51:29.461 INFO:teuthology.task.kernel:Updating grub on smithi117 to boot 6.8.0-g908b6b276e8f 2024-03-19T02:51:29.461 DEBUG:teuthology.orchestra.run.smithi117:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-19T02:51:30.055 INFO:teuthology.orchestra.run.smithi117.stderr:Generating grub configuration file ... 2024-03-19T02:51:31.396 INFO:teuthology.orchestra.run.smithi117.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-19T02:51:31.449 INFO:teuthology.orchestra.run.smithi117.stderr:done 2024-03-19T02:51:31.451 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-03-19T02:51:31.466 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.RBO6jDLIGE 2024-03-19T02:51:31.466 DEBUG:teuthology.orchestra.run.smithi117:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.RBO6jDLIGE 2024-03-19T02:51:31.534 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.RBO6jDLIGE 2024-03-19T02:51:31.668 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.RBO6jDLIGE is 6KB 2024-03-19T02:51:31.716 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.RBO6jDLIGE 2024-03-19T02:51:31.730 DEBUG:teuthology.orchestra.run.smithi117:> sudo /bin/ls /boot/loader/entries || true 2024-03-19T02:51:31.794 INFO:teuthology.orchestra.run.smithi117.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-19T02:51:31.795 INFO:teuthology.orchestra.run.smithi117.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-19T02:51:31.795 INFO:teuthology.orchestra.run.smithi117.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-19T02:51:31.795 INFO:teuthology.orchestra.run.smithi117.stdout:15949e3c88704300882482eddfd7baae-6.8.0-g908b6b276e8f.conf 2024-03-19T02:51:31.796 DEBUG:teuthology.orchestra.run.smithi117:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-g908b6b276e8f 2024-03-19T02:51:31.947 DEBUG:teuthology.orchestra.run.smithi117:> sudo shutdown -r now 2024-03-19T02:52:01.949 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-19T02:52:01.950 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-19T02:52:01.951 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:53:01.954 DEBUG:teuthology.orchestra.remote:timed out 2024-03-19T02:53:10.955 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-19T02:53:10.956 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T02:53:18.657 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-03-19T02:53:19.313 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-19T02:53:19.313 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-g908b6b276e8f"... 2024-03-19T02:53:19.313 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-03-19T02:53:19.352 INFO:teuthology.orchestra.run.smithi117.stdout:6.8.0-g908b6b276e8f 2024-03-19T02:53:19.353 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g908b6b276e8f vs 6.8.0-g908b6b276e8f 2024-03-19T02:53:19.353 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-19T02:53:19.353 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-19T02:53:20.356 DEBUG:teuthology.parallel:result is None 2024-03-19T02:53:20.356 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-19T02:53:20.364 INFO:teuthology.task.internal:Creating test directory... 2024-03-19T02:53:20.364 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-19T02:53:20.369 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-19T02:53:20.372 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-19T02:53:20.412 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-19T02:53:20.419 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-19T02:53:20.431 INFO:teuthology.task.internal:Creating archive directory... 2024-03-19T02:53:20.431 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-19T02:53:20.435 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-19T02:53:20.437 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-19T02:53:20.730 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-19T02:53:20.737 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-19T02:53:20.737 DEBUG:teuthology.orchestra.run.smithi119:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-19T02:53:20.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T02:53:20.754 DEBUG:teuthology.orchestra.run.smithi178:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-19T02:53:20.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T02:53:20.770 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-19T02:53:20.776 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-19T02:53:20.796 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-19T02:53:20.833 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T02:53:20.853 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T02:53:20.880 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T02:53:20.898 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T02:53:20.904 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T02:53:20.990 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T02:53:20.995 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-19T02:53:21.003 INFO:teuthology.task.internal:Configuring sudo... 2024-03-19T02:53:21.004 DEBUG:teuthology.orchestra.run.smithi117:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-19T02:53:21.013 DEBUG:teuthology.orchestra.run.smithi119:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-19T02:53:21.015 DEBUG:teuthology.orchestra.run.smithi178:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-19T02:53:21.086 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-19T02:53:21.096 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-19T02:53:21.096 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-19T02:53:21.100 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-19T02:53:21.102 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-19T02:53:21.137 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T02:53:21.248 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T02:53:21.348 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T02:53:21.419 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T02:53:21.506 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-19T02:53:21.507 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-19T02:53:21.595 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T02:53:21.640 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T02:53:21.698 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T02:53:21.784 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T02:53:21.828 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-19T02:53:21.828 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-19T02:53:21.899 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T02:53:21.944 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T02:53:22.001 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T02:53:22.082 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T02:53:22.131 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-19T02:53:22.131 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-19T02:53:22.201 DEBUG:teuthology.orchestra.run.smithi117:> sudo service rsyslog restart 2024-03-19T02:53:22.207 DEBUG:teuthology.orchestra.run.smithi119:> sudo service rsyslog restart 2024-03-19T02:53:22.209 DEBUG:teuthology.orchestra.run.smithi178:> sudo service rsyslog restart 2024-03-19T02:53:22.245 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T02:53:22.288 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T02:53:22.314 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T02:53:22.857 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-19T02:53:22.865 INFO:teuthology.task.internal:Starting timer... 2024-03-19T02:53:22.866 INFO:teuthology.run_tasks:Running task pcp... 2024-03-19T02:53:22.882 INFO:teuthology.run_tasks:Running task selinux... 2024-03-19T02:53:22.890 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-19T02:53:22.891 DEBUG:teuthology.orchestra.run.smithi117:> sudo service auditd rotate 2024-03-19T02:53:22.991 INFO:teuthology.orchestra.run.smithi117.stdout:Rotating logs: 2024-03-19T02:53:22.995 DEBUG:teuthology.orchestra.run.smithi119:> sudo service auditd rotate 2024-03-19T02:53:23.054 INFO:teuthology.orchestra.run.smithi119.stdout:Rotating logs: 2024-03-19T02:53:23.056 DEBUG:teuthology.orchestra.run.smithi178:> sudo service auditd rotate 2024-03-19T02:53:23.123 INFO:teuthology.orchestra.run.smithi178.stdout:Rotating logs: 2024-03-19T02:53:23.125 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-19T02:53:23.125 DEBUG:teuthology.orchestra.run.smithi117:> /usr/sbin/getenforce 2024-03-19T02:53:23.166 INFO:teuthology.orchestra.run.smithi117.stdout:Permissive 2024-03-19T02:53:23.166 DEBUG:teuthology.orchestra.run.smithi119:> /usr/sbin/getenforce 2024-03-19T02:53:23.200 INFO:teuthology.orchestra.run.smithi119.stdout:Permissive 2024-03-19T02:53:23.200 DEBUG:teuthology.orchestra.run.smithi178:> /usr/sbin/getenforce 2024-03-19T02:53:23.220 INFO:teuthology.orchestra.run.smithi178.stdout:Permissive 2024-03-19T02:53:23.220 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi117.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi119.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi178.front.sepia.ceph.com': 'permissive'} 2024-03-19T02:53:23.221 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T02:53:23.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T02:53:23.291 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T02:53:23.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T02:53:23.322 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T02:53:23.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T02:53:23.350 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-19T02:53:23.351 DEBUG:teuthology.orchestra.run.smithi117:> sudo /usr/sbin/setenforce permissive 2024-03-19T02:53:23.481 DEBUG:teuthology.orchestra.run.smithi119:> sudo /usr/sbin/setenforce permissive 2024-03-19T02:53:23.511 DEBUG:teuthology.orchestra.run.smithi178:> sudo /usr/sbin/setenforce permissive 2024-03-19T02:53:23.564 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-19T02:53:23.573 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-19T02:53:23.591 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-19T02:53:23.592 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi117.front.sepia.ceph.com,smithi119.front.sepia.ceph.com,smithi178.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-19T03:04:39.654 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi117.front.sepia.ceph.com'), Remote(name='ubuntu@smithi119.front.sepia.ceph.com'), Remote(name='ubuntu@smithi178.front.sepia.ceph.com')] 2024-03-19T03:04:39.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-19T03:04:39.656 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T03:04:39.803 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-03-19T03:04:39.955 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-19T03:04:39.955 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-19T03:04:39.956 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T03:04:40.046 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-03-19T03:04:40.115 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-19T03:04:40.115 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-03-19T03:04:40.116 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T03:04:40.207 DEBUG:teuthology.orchestra.run.smithi178:> true 2024-03-19T03:04:40.281 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-03-19T03:04:40.282 INFO:teuthology.run_tasks:Running task clock... 2024-03-19T03:04:40.293 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-19T03:04:40.294 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-19T03:04:40.294 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T03:04:40.302 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-19T03:04:40.302 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T03:04:40.304 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-19T03:04:40.304 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T03:04:40.337 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-19T03:04:40.354 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-19T03:04:40.355 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-19T03:04:40.374 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-19T03:04:40.386 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: ntpd: command not found 2024-03-19T03:04:40.399 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-19T03:04:40.400 INFO:teuthology.orchestra.run.smithi119.stdout:506 Cannot talk to daemon 2024-03-19T03:04:40.402 INFO:teuthology.orchestra.run.smithi178.stderr:sudo: ntpd: command not found 2024-03-19T03:04:40.416 INFO:teuthology.orchestra.run.smithi178.stdout:506 Cannot talk to daemon 2024-03-19T03:04:40.418 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-19T03:04:40.432 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-19T03:04:40.434 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-19T03:04:40.448 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-19T03:04:40.452 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-19T03:04:40.478 INFO:teuthology.orchestra.run.smithi119.stderr:bash: line 1: ntpq: command not found 2024-03-19T03:04:40.483 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T03:04:40.483 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-03-19T03:04:40.483 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T03:04:40.483 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T03:04:40.483 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T03:04:40.483 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T03:04:40.492 INFO:teuthology.orchestra.run.smithi178.stderr:bash: line 1: ntpq: command not found 2024-03-19T03:04:40.496 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T03:04:40.496 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2024-03-19T03:04:40.496 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T03:04:40.496 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T03:04:40.496 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T03:04:40.496 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T03:04:40.541 INFO:teuthology.orchestra.run.smithi117.stderr:sudo: ntpd: command not found 2024-03-19T03:04:40.581 INFO:teuthology.orchestra.run.smithi117.stdout:506 Cannot talk to daemon 2024-03-19T03:04:40.630 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-19T03:04:40.683 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-19T03:04:40.804 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-03-19T03:04:40.811 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T03:04:40.811 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-03-19T03:04:40.811 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T03:04:40.811 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T03:04:40.811 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T03:04:40.812 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T03:04:40.813 INFO:teuthology.run_tasks:Running task install... 2024-03-19T03:04:40.821 DEBUG:teuthology.task.install:project ceph 2024-03-19T03:04:40.821 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'}} 2024-03-19T03:04:40.821 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'} 2024-03-19T03:04:40.821 INFO:teuthology.task.install:Using flavor: default 2024-03-19T03:04:40.829 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-19T03:04:40.829 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-19T03:04:40.830 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'tag': None, 'wait_for_package': False} 2024-03-19T03:04:40.830 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T03:04:40.832 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'tag': None, 'wait_for_package': False} 2024-03-19T03:04:40.833 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T03:04:40.833 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'tag': None, 'wait_for_package': False} 2024-03-19T03:04:40.834 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T03:04:40.983 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/ 2024-03-19T03:04:40.983 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2206.gb6ae2491 2024-03-19T03:04:40.994 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/ 2024-03-19T03:04:40.994 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2206.gb6ae2491 2024-03-19T03:04:41.055 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/ 2024-03-19T03:04:41.055 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2206.gb6ae2491 2024-03-19T03:04:41.095 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-19T03:04:41.095 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-19T03:04:41.095 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-19T03:04:41.111 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-19T03:04:41.111 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-19T03:04:41.111 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-19T03:04:41.138 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-19T03:04:41.139 DEBUG:teuthology.orchestra.run.smithi178:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b6ae24918b03b1b8a19c9263a50655bba80bb810/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-19T03:04:41.203 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-19T03:04:41.203 DEBUG:teuthology.orchestra.run.smithi117:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b6ae24918b03b1b8a19c9263a50655bba80bb810/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-19T03:04:41.207 DEBUG:teuthology.orchestra.run.smithi178:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-19T03:04:41.232 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-19T03:04:41.232 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-19T03:04:41.232 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-19T03:04:41.261 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-19T03:04:41.262 DEBUG:teuthology.orchestra.run.smithi119:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b6ae24918b03b1b8a19c9263a50655bba80bb810/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-19T03:04:41.286 DEBUG:teuthology.orchestra.run.smithi178:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-19T03:04:41.330 DEBUG:teuthology.orchestra.run.smithi119:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-19T03:04:41.334 DEBUG:teuthology.orchestra.run.smithi117:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-19T03:04:41.352 INFO:teuthology.orchestra.run.smithi178.stdout:check_obsoletes = 1 2024-03-19T03:04:41.353 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum clean all 2024-03-19T03:04:41.411 DEBUG:teuthology.orchestra.run.smithi119:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-19T03:04:41.458 DEBUG:teuthology.orchestra.run.smithi117:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-19T03:04:41.477 INFO:teuthology.orchestra.run.smithi119.stdout:check_obsoletes = 1 2024-03-19T03:04:41.479 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum clean all 2024-03-19T03:04:41.545 INFO:teuthology.orchestra.run.smithi117.stdout:check_obsoletes = 1 2024-03-19T03:04:41.549 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum clean all 2024-03-19T03:04:41.592 INFO:teuthology.orchestra.run.smithi178.stdout:74 files removed 2024-03-19T03:04:41.620 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-19T03:04:41.714 INFO:teuthology.orchestra.run.smithi119.stdout:74 files removed 2024-03-19T03:04:41.742 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-19T03:04:41.913 INFO:teuthology.orchestra.run.smithi117.stdout:74 files removed 2024-03-19T03:04:41.952 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-19T03:04:42.374 INFO:teuthology.orchestra.run.smithi178.stdout:ceph packages for x86_64 180 kB/s | 83 kB 00:00 2024-03-19T03:04:42.437 INFO:teuthology.orchestra.run.smithi119.stdout:ceph packages for x86_64 206 kB/s | 83 kB 00:00 2024-03-19T03:04:42.752 INFO:teuthology.orchestra.run.smithi119.stdout:ceph noarch packages 36 kB/s | 10 kB 00:00 2024-03-19T03:04:42.758 INFO:teuthology.orchestra.run.smithi178.stdout:ceph noarch packages 29 kB/s | 10 kB 00:00 2024-03-19T03:04:42.865 INFO:teuthology.orchestra.run.smithi117.stdout:ceph packages for x86_64 203 kB/s | 83 kB 00:00 2024-03-19T03:04:43.076 INFO:teuthology.orchestra.run.smithi178.stdout:ceph source packages 6.6 kB/s | 1.9 kB 00:00 2024-03-19T03:04:43.094 INFO:teuthology.orchestra.run.smithi119.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-03-19T03:04:43.449 INFO:teuthology.orchestra.run.smithi117.stdout:ceph noarch packages 20 kB/s | 10 kB 00:00 2024-03-19T03:04:43.495 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-19T03:04:43.546 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-19T03:04:43.941 INFO:teuthology.orchestra.run.smithi117.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-03-19T03:04:44.644 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-19T03:04:44.962 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS-9 - BaseOS 5.8 MB/s | 2.0 MB 00:00 2024-03-19T03:04:44.980 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-03-19T03:04:45.838 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-19T03:04:46.278 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-03-19T03:04:48.490 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.0 MB 00:01 2024-03-19T03:05:01.015 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - AppStream 1.7 MB/s | 19 MB 00:11 2024-03-19T03:05:01.187 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 19 MB 00:14 2024-03-19T03:05:05.547 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - CRB 6.9 MB/s | 6.1 MB 00:00 2024-03-19T03:05:07.039 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 16 kB 00:00 2024-03-19T03:05:08.132 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-03-19T03:05:13.077 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-19T03:05:15.416 INFO:teuthology.orchestra.run.smithi119.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T03:05:15.417 INFO:teuthology.orchestra.run.smithi119.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T03:05:15.431 INFO:teuthology.orchestra.run.smithi119.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-19T03:05:15.432 INFO:teuthology.orchestra.run.smithi119.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-19T03:05:15.433 INFO:teuthology.orchestra.run.smithi119.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-19T03:05:15.434 INFO:teuthology.orchestra.run.smithi119.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-19T03:05:15.436 INFO:teuthology.orchestra.run.smithi119.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-19T03:05:15.439 INFO:teuthology.orchestra.run.smithi119.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-19T03:05:15.440 INFO:teuthology.orchestra.run.smithi119.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-19T03:05:15.521 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-19T03:05:15.548 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================================ 2024-03-19T03:05:15.548 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-03-19T03:05:15.548 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================================ 2024-03-19T03:05:15.548 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-19T03:05:15.548 INFO:teuthology.orchestra.run.smithi119.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-19T03:05:15.548 INFO:teuthology.orchestra.run.smithi119.stdout: ceph x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 6.6 k 2024-03-19T03:05:15.548 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.4 M 2024-03-19T03:05:15.548 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 891 k 2024-03-19T03:05:15.548 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 145 k 2024-03-19T03:05:15.548 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 1.5 M 2024-03-19T03:05:15.548 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 139 k 2024-03-19T03:05:15.549 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 1.8 M 2024-03-19T03:05:15.549 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 7.4 M 2024-03-19T03:05:15.549 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 49 k 2024-03-19T03:05:15.549 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 9.8 M 2024-03-19T03:05:15.549 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 48 M 2024-03-19T03:05:15.549 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 264 k 2024-03-19T03:05:15.549 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 749 k 2024-03-19T03:05:15.549 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 221 k 2024-03-19T03:05:15.549 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-shell x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 27 k 2024-03-19T03:05:15.549 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-top noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 18 k 2024-03-19T03:05:15.549 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-19T03:05:15.549 INFO:teuthology.orchestra.run.smithi119.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-19T03:05:15.550 INFO:teuthology.orchestra.run.smithi119.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-19T03:05:15.550 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 32 k 2024-03-19T03:05:15.550 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 746 k 2024-03-19T03:05:15.550 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 127 k 2024-03-19T03:05:15.550 INFO:teuthology.orchestra.run.smithi119.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-19T03:05:15.550 INFO:teuthology.orchestra.run.smithi119.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-19T03:05:15.550 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-19T03:05:15.550 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-19T03:05:15.550 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-19T03:05:15.550 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-19T03:05:15.550 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 162 k 2024-03-19T03:05:15.551 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 321 k 2024-03-19T03:05:15.551 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 298 k 2024-03-19T03:05:15.551 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 99 k 2024-03-19T03:05:15.551 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 85 k 2024-03-19T03:05:15.551 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T03:05:15.551 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 170 k 2024-03-19T03:05:15.551 INFO:teuthology.orchestra.run.smithi119.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-19T03:05:15.551 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrading: 2024-03-19T03:05:15.551 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-19T03:05:15.551 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-19T03:05:15.551 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-19T03:05:15.551 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-19T03:05:15.551 INFO:teuthology.orchestra.run.smithi119.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-19T03:05:15.551 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-19T03:05:15.552 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-19T03:05:15.552 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-19T03:05:15.552 INFO:teuthology.orchestra.run.smithi119.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-19T03:05:15.552 INFO:teuthology.orchestra.run.smithi119.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-19T03:05:15.552 INFO:teuthology.orchestra.run.smithi119.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-19T03:05:15.552 INFO:teuthology.orchestra.run.smithi119.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-19T03:05:15.552 INFO:teuthology.orchestra.run.smithi119.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-19T03:05:15.552 INFO:teuthology.orchestra.run.smithi119.stdout: librados2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.4 M 2024-03-19T03:05:15.552 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T03:05:15.552 INFO:teuthology.orchestra.run.smithi119.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-19T03:05:15.552 INFO:teuthology.orchestra.run.smithi119.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-19T03:05:15.552 INFO:teuthology.orchestra.run.smithi119.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-19T03:05:15.552 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-19T03:05:15.553 INFO:teuthology.orchestra.run.smithi119.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-19T03:05:15.553 INFO:teuthology.orchestra.run.smithi119.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-19T03:05:15.553 INFO:teuthology.orchestra.run.smithi119.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-19T03:05:15.553 INFO:teuthology.orchestra.run.smithi119.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-19T03:05:15.553 INFO:teuthology.orchestra.run.smithi119.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-19T03:05:15.553 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-03-19T03:05:15.553 INFO:teuthology.orchestra.run.smithi119.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-19T03:05:15.553 INFO:teuthology.orchestra.run.smithi119.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-19T03:05:15.553 INFO:teuthology.orchestra.run.smithi119.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-19T03:05:15.553 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 21 M 2024-03-19T03:05:15.553 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 34 k 2024-03-19T03:05:15.553 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 2.4 M 2024-03-19T03:05:15.554 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 248 k 2024-03-19T03:05:15.554 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 4.7 M 2024-03-19T03:05:15.554 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 17 M 2024-03-19T03:05:15.554 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 16 k 2024-03-19T03:05:15.554 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 25 k 2024-03-19T03:05:15.554 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-19T03:05:15.554 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-19T03:05:15.554 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-19T03:05:15.554 INFO:teuthology.orchestra.run.smithi119.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-19T03:05:15.554 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-19T03:05:15.554 INFO:teuthology.orchestra.run.smithi119.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-19T03:05:15.554 INFO:teuthology.orchestra.run.smithi119.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-19T03:05:15.554 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-19T03:05:15.555 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-19T03:05:15.555 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 163 k 2024-03-19T03:05:15.555 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-19T03:05:15.555 INFO:teuthology.orchestra.run.smithi119.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-19T03:05:15.555 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-19T03:05:15.555 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-19T03:05:15.555 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-19T03:05:15.555 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 501 k 2024-03-19T03:05:15.555 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-19T03:05:15.555 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.1 M 2024-03-19T03:05:15.556 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-19T03:05:15.556 INFO:teuthology.orchestra.run.smithi119.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-19T03:05:15.556 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-19T03:05:15.556 INFO:teuthology.orchestra.run.smithi119.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-19T03:05:15.556 INFO:teuthology.orchestra.run.smithi119.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-19T03:05:15.556 INFO:teuthology.orchestra.run.smithi119.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-19T03:05:15.556 INFO:teuthology.orchestra.run.smithi119.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-19T03:05:15.556 INFO:teuthology.orchestra.run.smithi119.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-19T03:05:15.556 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-19T03:05:15.557 INFO:teuthology.orchestra.run.smithi119.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-19T03:05:15.557 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-19T03:05:15.557 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-19T03:05:15.557 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-19T03:05:15.557 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-19T03:05:15.557 INFO:teuthology.orchestra.run.smithi119.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-19T03:05:15.557 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-19T03:05:15.557 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-19T03:05:15.557 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-19T03:05:15.557 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 45 k 2024-03-19T03:05:15.557 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 131 k 2024-03-19T03:05:15.557 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-19T03:05:15.557 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-19T03:05:15.557 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-19T03:05:15.557 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-19T03:05:15.558 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-19T03:05:15.558 INFO:teuthology.orchestra.run.smithi119.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-19T03:05:15.558 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-19T03:05:15.558 INFO:teuthology.orchestra.run.smithi119.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-19T03:05:15.558 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-19T03:05:15.558 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-19T03:05:15.558 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-19T03:05:15.558 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-19T03:05:15.558 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-19T03:05:15.558 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-19T03:05:15.558 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-19T03:05:15.558 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-19T03:05:15.559 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-19T03:05:15.559 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-19T03:05:15.559 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-19T03:05:15.559 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-19T03:05:15.559 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-19T03:05:15.559 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-19T03:05:15.559 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-19T03:05:15.559 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-19T03:05:15.559 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-19T03:05:15.559 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-19T03:05:15.559 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-19T03:05:15.559 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-19T03:05:15.559 INFO:teuthology.orchestra.run.smithi119.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-19T03:05:15.560 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-19T03:05:15.560 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-19T03:05:15.560 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-19T03:05:15.560 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-19T03:05:15.560 INFO:teuthology.orchestra.run.smithi119.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-19T03:05:15.560 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-19T03:05:15.560 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-19T03:05:15.560 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-19T03:05:15.560 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-19T03:05:15.560 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-19T03:05:15.560 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-19T03:05:15.560 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-19T03:05:15.560 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-19T03:05:15.560 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-19T03:05:15.560 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-19T03:05:15.561 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-19T03:05:15.561 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-19T03:05:15.561 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-19T03:05:15.561 INFO:teuthology.orchestra.run.smithi119.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-19T03:05:15.561 INFO:teuthology.orchestra.run.smithi119.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-19T03:05:15.561 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-19T03:05:15.561 INFO:teuthology.orchestra.run.smithi119.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-19T03:05:15.561 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-19T03:05:15.561 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-19T03:05:15.561 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-19T03:05:15.561 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-19T03:05:15.561 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-19T03:05:15.561 INFO:teuthology.orchestra.run.smithi119.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-19T03:05:15.562 INFO:teuthology.orchestra.run.smithi119.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-19T03:05:15.562 INFO:teuthology.orchestra.run.smithi119.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-19T03:05:15.562 INFO:teuthology.orchestra.run.smithi119.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-19T03:05:15.562 INFO:teuthology.orchestra.run.smithi119.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-19T03:05:15.562 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-19T03:05:15.562 INFO:teuthology.orchestra.run.smithi119.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-19T03:05:15.562 INFO:teuthology.orchestra.run.smithi119.stdout:Installing weak dependencies: 2024-03-19T03:05:15.563 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-19T03:05:15.563 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-19T03:05:15.563 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-19T03:05:15.563 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================================ 2024-03-19T03:05:15.563 INFO:teuthology.orchestra.run.smithi119.stdout:Install 150 Packages 2024-03-19T03:05:15.563 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrade 24 Packages 2024-03-19T03:05:15.563 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-19T03:05:15.563 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 224 M 2024-03-19T03:05:15.564 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-19T03:05:16.204 INFO:teuthology.orchestra.run.smithi119.stdout:(1/174): ceph-19.0.0-2206.gb6ae2491.el9.x86_64. 32 kB/s | 6.6 kB 00:00 2024-03-19T03:05:17.195 INFO:teuthology.orchestra.run.smithi119.stdout:(2/174): ceph-common-19.0.0-2206.gb6ae2491.el9. 17 MB/s | 21 MB 00:01 2024-03-19T03:05:17.220 INFO:teuthology.orchestra.run.smithi119.stdout:(3/174): ceph-fuse-19.0.0-2206.gb6ae2491.el9.x8 876 kB/s | 891 kB 00:01 2024-03-19T03:05:17.246 INFO:teuthology.orchestra.run.smithi119.stdout:(4/174): ceph-immutable-object-cache-19.0.0-220 2.8 MB/s | 145 kB 00:00 2024-03-19T03:05:17.430 INFO:teuthology.orchestra.run.smithi119.stdout:(5/174): ceph-mgr-19.0.0-2206.gb6ae2491.el9.x86 8.1 MB/s | 1.5 MB 00:00 2024-03-19T03:05:17.780 INFO:teuthology.orchestra.run.smithi119.stdout:(6/174): ceph-mon-19.0.0-2206.gb6ae2491.el9.x86 13 MB/s | 4.7 MB 00:00 2024-03-19T03:05:18.539 INFO:teuthology.orchestra.run.smithi119.stdout:(7/174): ceph-mds-19.0.0-2206.gb6ae2491.el9.x86 1.8 MB/s | 2.4 MB 00:01 2024-03-19T03:05:18.756 INFO:teuthology.orchestra.run.smithi119.stdout:(8/174): ceph-osd-19.0.0-2206.gb6ae2491.el9.x86 17 MB/s | 17 MB 00:00 2024-03-19T03:05:18.831 INFO:teuthology.orchestra.run.smithi119.stdout:(9/174): ceph-base-19.0.0-2206.gb6ae2491.el9.x8 1.9 MB/s | 5.4 MB 00:02 2024-03-19T03:05:18.856 INFO:teuthology.orchestra.run.smithi119.stdout:(10/174): ceph-selinux-19.0.0-2206.gb6ae2491.el 252 kB/s | 25 kB 00:00 2024-03-19T03:05:18.907 INFO:teuthology.orchestra.run.smithi119.stdout:(11/174): cephfs-mirror-19.0.0-2206.gb6ae2491.e 4.3 MB/s | 221 kB 00:00 2024-03-19T03:05:18.957 INFO:teuthology.orchestra.run.smithi119.stdout:(12/174): cephfs-shell-19.0.0-2206.gb6ae2491.el 542 kB/s | 27 kB 00:00 2024-03-19T03:05:19.007 INFO:teuthology.orchestra.run.smithi119.stdout:(13/174): libcephfs-devel-19.0.0-2206.gb6ae2491 638 kB/s | 32 kB 00:00 2024-03-19T03:05:19.082 INFO:teuthology.orchestra.run.smithi119.stdout:(14/174): libcephfs2-19.0.0-2206.gb6ae2491.el9. 9.7 MB/s | 746 kB 00:00 2024-03-19T03:05:19.133 INFO:teuthology.orchestra.run.smithi119.stdout:(15/174): libcephsqlite-19.0.0-2206.gb6ae2491.e 3.2 MB/s | 163 kB 00:00 2024-03-19T03:05:19.183 INFO:teuthology.orchestra.run.smithi119.stdout:(16/174): librados-devel-19.0.0-2206.gb6ae2491. 2.5 MB/s | 127 kB 00:00 2024-03-19T03:05:19.250 INFO:teuthology.orchestra.run.smithi119.stdout:(17/174): libradosstriper1-19.0.0-2206.gb6ae249 7.4 MB/s | 501 kB 00:00 2024-03-19T03:05:19.542 INFO:teuthology.orchestra.run.smithi119.stdout:(18/174): librgw2-19.0.0-2206.gb6ae2491.el9.x86 18 MB/s | 5.1 MB 00:00 2024-03-19T03:05:19.583 INFO:teuthology.orchestra.run.smithi119.stdout:(19/174): python3-ceph-argparse-19.0.0-2206.gb6 1.1 MB/s | 45 kB 00:00 2024-03-19T03:05:19.634 INFO:teuthology.orchestra.run.smithi119.stdout:(20/174): python3-ceph-common-19.0.0-2206.gb6ae 2.6 MB/s | 131 kB 00:00 2024-03-19T03:05:19.684 INFO:teuthology.orchestra.run.smithi119.stdout:(21/174): python3-cephfs-19.0.0-2206.gb6ae2491. 3.2 MB/s | 162 kB 00:00 2024-03-19T03:05:19.743 INFO:teuthology.orchestra.run.smithi119.stdout:(22/174): python3-rados-19.0.0-2206.gb6ae2491.e 5.4 MB/s | 321 kB 00:00 2024-03-19T03:05:19.801 INFO:teuthology.orchestra.run.smithi119.stdout:(23/174): python3-rbd-19.0.0-2206.gb6ae2491.el9 5.0 MB/s | 298 kB 00:00 2024-03-19T03:05:19.852 INFO:teuthology.orchestra.run.smithi119.stdout:(24/174): python3-rgw-19.0.0-2206.gb6ae2491.el9 1.9 MB/s | 99 kB 00:00 2024-03-19T03:05:19.923 INFO:teuthology.orchestra.run.smithi119.stdout:(25/174): rbd-fuse-19.0.0-2206.gb6ae2491.el9.x8 1.2 MB/s | 85 kB 00:00 2024-03-19T03:05:20.127 INFO:teuthology.orchestra.run.smithi119.stdout:(26/174): rbd-mirror-19.0.0-2206.gb6ae2491.el9. 15 MB/s | 3.1 MB 00:00 2024-03-19T03:05:20.178 INFO:teuthology.orchestra.run.smithi119.stdout:(27/174): rbd-nbd-19.0.0-2206.gb6ae2491.el9.x86 3.3 MB/s | 170 kB 00:00 2024-03-19T03:05:20.219 INFO:teuthology.orchestra.run.smithi119.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2206.g 818 kB/s | 34 kB 00:00 2024-03-19T03:05:20.270 INFO:teuthology.orchestra.run.smithi119.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2206.gb6ae249 2.7 MB/s | 139 kB 00:00 2024-03-19T03:05:20.403 INFO:teuthology.orchestra.run.smithi119.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2206.gb6ae2 13 MB/s | 1.8 MB 00:00 2024-03-19T03:05:20.837 INFO:teuthology.orchestra.run.smithi119.stdout:(31/174): ceph-mgr-diskprediction-local-19.0.0- 17 MB/s | 7.4 MB 00:00 2024-03-19T03:05:20.896 INFO:teuthology.orchestra.run.smithi119.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2206.gb6 4.2 MB/s | 248 kB 00:00 2024-03-19T03:05:20.937 INFO:teuthology.orchestra.run.smithi119.stdout:(33/174): ceph-mgr-rook-19.0.0-2206.gb6ae2491.e 1.2 MB/s | 49 kB 00:00 2024-03-19T03:05:20.971 INFO:teuthology.orchestra.run.smithi119.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2206.gb 493 kB/s | 16 kB 00:00 2024-03-19T03:05:21.021 INFO:teuthology.orchestra.run.smithi119.stdout:(35/174): ceph-volume-19.0.0-2206.gb6ae2491.el9 5.2 MB/s | 264 kB 00:00 2024-03-19T03:05:21.097 INFO:teuthology.orchestra.run.smithi119.stdout:(36/174): cephadm-19.0.0-2206.gb6ae2491.el9.noa 9.7 MB/s | 749 kB 00:00 2024-03-19T03:05:21.131 INFO:teuthology.orchestra.run.smithi119.stdout:(37/174): cephfs-top-19.0.0-2206.gb6ae2491.el9. 545 kB/s | 18 kB 00:00 2024-03-19T03:05:21.373 INFO:teuthology.orchestra.run.smithi119.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 2.7 MB/s | 662 kB 00:00 2024-03-19T03:05:21.473 INFO:teuthology.orchestra.run.smithi119.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 9.0 MB/s | 920 kB 00:00 2024-03-19T03:05:21.531 INFO:teuthology.orchestra.run.smithi119.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-19T03:05:21.598 INFO:teuthology.orchestra.run.smithi119.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 4.5 MB/s | 307 kB 00:00 2024-03-19T03:05:21.648 INFO:teuthology.orchestra.run.smithi119.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 634 kB/s | 32 kB 00:00 2024-03-19T03:05:21.774 INFO:teuthology.orchestra.run.smithi119.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 24 MB/s | 3.0 MB 00:00 2024-03-19T03:05:21.808 INFO:teuthology.orchestra.run.smithi119.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 514 kB/s | 17 kB 00:00 2024-03-19T03:05:21.841 INFO:teuthology.orchestra.run.smithi119.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 303 kB/s | 10 kB 00:00 2024-03-19T03:05:21.875 INFO:teuthology.orchestra.run.smithi119.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2024-03-19T03:05:21.908 INFO:teuthology.orchestra.run.smithi119.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-19T03:05:21.975 INFO:teuthology.orchestra.run.smithi119.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-19T03:05:22.026 INFO:teuthology.orchestra.run.smithi119.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 578 kB 00:00 2024-03-19T03:05:22.059 INFO:teuthology.orchestra.run.smithi119.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-19T03:05:22.101 INFO:teuthology.orchestra.run.smithi119.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-19T03:05:22.135 INFO:teuthology.orchestra.run.smithi119.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 5.5 MB/s | 187 kB 00:00 2024-03-19T03:05:22.243 INFO:teuthology.orchestra.run.smithi119.stdout:(53/174): ceph-radosgw-19.0.0-2206.gb6ae2491.el 2.7 MB/s | 9.8 MB 00:03 2024-03-19T03:05:22.269 INFO:teuthology.orchestra.run.smithi119.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.6 MB/s | 217 kB 00:00 2024-03-19T03:05:22.286 INFO:teuthology.orchestra.run.smithi119.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 829 kB/s | 34 kB 00:00 2024-03-19T03:05:22.411 INFO:teuthology.orchestra.run.smithi119.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 33 MB/s | 4.7 MB 00:00 2024-03-19T03:05:22.453 INFO:teuthology.orchestra.run.smithi119.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 9.2 MB/s | 1.5 MB 00:00 2024-03-19T03:05:22.495 INFO:teuthology.orchestra.run.smithi119.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 9.8 MB/s | 811 kB 00:00 2024-03-19T03:05:22.520 INFO:teuthology.orchestra.run.smithi119.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.0 MB/s | 134 kB 00:00 2024-03-19T03:05:22.646 INFO:teuthology.orchestra.run.smithi119.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 38 MB/s | 5.7 MB 00:00 2024-03-19T03:05:22.679 INFO:teuthology.orchestra.run.smithi119.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-19T03:05:22.704 INFO:teuthology.orchestra.run.smithi119.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-19T03:05:22.838 INFO:teuthology.orchestra.run.smithi119.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 36 MB/s | 5.8 MB 00:00 2024-03-19T03:05:22.863 INFO:teuthology.orchestra.run.smithi119.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 155 kB/s | 24 kB 00:00 2024-03-19T03:05:22.880 INFO:teuthology.orchestra.run.smithi119.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 432 kB/s | 18 kB 00:00 2024-03-19T03:05:22.905 INFO:teuthology.orchestra.run.smithi119.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-19T03:05:22.923 INFO:teuthology.orchestra.run.smithi119.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 760 kB/s | 32 kB 00:00 2024-03-19T03:05:23.048 INFO:teuthology.orchestra.run.smithi119.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-19T03:05:23.073 INFO:teuthology.orchestra.run.smithi119.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-19T03:05:23.098 INFO:teuthology.orchestra.run.smithi119.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-19T03:05:23.124 INFO:teuthology.orchestra.run.smithi119.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-19T03:05:23.149 INFO:teuthology.orchestra.run.smithi119.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 822 kB/s | 41 kB 00:00 2024-03-19T03:05:23.174 INFO:teuthology.orchestra.run.smithi119.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-19T03:05:23.200 INFO:teuthology.orchestra.run.smithi119.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-19T03:05:23.225 INFO:teuthology.orchestra.run.smithi119.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-19T03:05:23.250 INFO:teuthology.orchestra.run.smithi119.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 946 kB/s | 47 kB 00:00 2024-03-19T03:05:23.275 INFO:teuthology.orchestra.run.smithi119.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 854 kB/s | 43 kB 00:00 2024-03-19T03:05:23.492 INFO:teuthology.orchestra.run.smithi119.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 179 kB/s | 39 kB 00:00 2024-03-19T03:05:23.726 INFO:teuthology.orchestra.run.smithi119.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-03-19T03:05:23.751 INFO:teuthology.orchestra.run.smithi119.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 160 kB/s | 41 kB 00:00 2024-03-19T03:05:23.777 INFO:teuthology.orchestra.run.smithi119.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-19T03:05:23.802 INFO:teuthology.orchestra.run.smithi119.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-19T03:05:23.827 INFO:teuthology.orchestra.run.smithi119.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-19T03:05:23.852 INFO:teuthology.orchestra.run.smithi119.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 879 kB/s | 44 kB 00:00 2024-03-19T03:05:23.878 INFO:teuthology.orchestra.run.smithi119.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-19T03:05:23.903 INFO:teuthology.orchestra.run.smithi119.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-19T03:05:23.967 INFO:teuthology.orchestra.run.smithi119.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 13 MB/s | 1.1 MB 00:00 2024-03-19T03:05:23.995 INFO:teuthology.orchestra.run.smithi119.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-19T03:05:24.021 INFO:teuthology.orchestra.run.smithi119.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 124 kB 00:00 2024-03-19T03:05:24.046 INFO:teuthology.orchestra.run.smithi119.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-19T03:05:24.546 INFO:teuthology.orchestra.run.smithi119.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 357 kB/s | 187 kB 00:00 2024-03-19T03:05:24.722 INFO:teuthology.orchestra.run.smithi119.stdout:(92/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2024-03-19T03:05:24.764 INFO:teuthology.orchestra.run.smithi119.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 899 kB/s | 195 kB 00:00 2024-03-19T03:05:24.831 INFO:teuthology.orchestra.run.smithi119.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-03-19T03:05:24.847 INFO:teuthology.orchestra.run.smithi119.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 258 kB/s | 21 kB 00:00 2024-03-19T03:05:24.873 INFO:teuthology.orchestra.run.smithi119.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 609 kB/s | 25 kB 00:00 2024-03-19T03:05:24.965 INFO:teuthology.orchestra.run.smithi119.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.6 MB/s | 245 kB 00:00 2024-03-19T03:05:25.015 INFO:teuthology.orchestra.run.smithi119.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-03-19T03:05:25.057 INFO:teuthology.orchestra.run.smithi119.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 5.1 MB/s | 216 kB 00:00 2024-03-19T03:05:25.099 INFO:teuthology.orchestra.run.smithi119.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 839 kB/s | 35 kB 00:00 2024-03-19T03:05:25.158 INFO:teuthology.orchestra.run.smithi119.stdout:(101/174): openssl-devel-3.0.7-27.el9.x86_64.rp 13 MB/s | 4.1 MB 00:00 2024-03-19T03:05:25.183 INFO:teuthology.orchestra.run.smithi119.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 532 kB/s | 44 kB 00:00 2024-03-19T03:05:25.208 INFO:teuthology.orchestra.run.smithi119.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-19T03:05:25.292 INFO:teuthology.orchestra.run.smithi119.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 205 kB/s | 22 kB 00:00 2024-03-19T03:05:25.434 INFO:teuthology.orchestra.run.smithi119.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.7 MB/s | 863 kB 00:00 2024-03-19T03:05:25.717 INFO:teuthology.orchestra.run.smithi119.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.3 MB/s | 548 kB 00:00 2024-03-19T03:05:25.785 INFO:teuthology.orchestra.run.smithi119.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 292 kB/s | 19 kB 00:00 2024-03-19T03:05:25.809 INFO:teuthology.orchestra.run.smithi119.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 822 kB/s | 309 kB 00:00 2024-03-19T03:05:25.877 INFO:teuthology.orchestra.run.smithi119.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 389 kB/s | 26 kB 00:00 2024-03-19T03:05:25.943 INFO:teuthology.orchestra.run.smithi119.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 731 kB/s | 49 kB 00:00 2024-03-19T03:05:26.019 INFO:teuthology.orchestra.run.smithi119.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 897 kB/s | 67 kB 00:00 2024-03-19T03:05:26.086 INFO:teuthology.orchestra.run.smithi119.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-03-19T03:05:26.111 INFO:teuthology.orchestra.run.smithi119.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-03-19T03:05:26.170 INFO:teuthology.orchestra.run.smithi119.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-03-19T03:05:26.236 INFO:teuthology.orchestra.run.smithi119.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 653 kB/s | 43 kB 00:00 2024-03-19T03:05:26.262 INFO:teuthology.orchestra.run.smithi119.stdout:(116/174): python3-asyncssh-2.13.2-4.el9.noarch 3.6 MB/s | 548 kB 00:00 2024-03-19T03:05:26.304 INFO:teuthology.orchestra.run.smithi119.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 481 kB/s | 32 kB 00:00 2024-03-19T03:05:26.329 INFO:teuthology.orchestra.run.smithi119.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 224 kB/s | 15 kB 00:00 2024-03-19T03:05:26.371 INFO:teuthology.orchestra.run.smithi119.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-03-19T03:05:26.413 INFO:teuthology.orchestra.run.smithi119.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-03-19T03:05:26.447 INFO:teuthology.orchestra.run.smithi119.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-03-19T03:05:26.480 INFO:teuthology.orchestra.run.smithi119.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 524 kB/s | 35 kB 00:00 2024-03-19T03:05:26.522 INFO:teuthology.orchestra.run.smithi119.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-03-19T03:05:26.590 INFO:teuthology.orchestra.run.smithi119.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-03-19T03:05:26.640 INFO:teuthology.orchestra.run.smithi119.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 13 MB/s | 2.0 MB 00:00 2024-03-19T03:05:26.665 INFO:teuthology.orchestra.run.smithi119.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-03-19T03:05:26.707 INFO:teuthology.orchestra.run.smithi119.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 264 kB/s | 18 kB 00:00 2024-03-19T03:05:26.732 INFO:teuthology.orchestra.run.smithi119.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 346 kB/s | 23 kB 00:00 2024-03-19T03:05:26.774 INFO:teuthology.orchestra.run.smithi119.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 290 kB/s | 19 kB 00:00 2024-03-19T03:05:26.799 INFO:teuthology.orchestra.run.smithi119.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 294 kB/s | 20 kB 00:00 2024-03-19T03:05:26.858 INFO:teuthology.orchestra.run.smithi119.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-03-19T03:05:26.875 INFO:teuthology.orchestra.run.smithi119.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 615 kB/s | 46 kB 00:00 2024-03-19T03:05:26.925 INFO:teuthology.orchestra.run.smithi119.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-03-19T03:05:26.942 INFO:teuthology.orchestra.run.smithi119.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 865 kB/s | 58 kB 00:00 2024-03-19T03:05:26.993 INFO:teuthology.orchestra.run.smithi119.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-19T03:05:27.009 INFO:teuthology.orchestra.run.smithi119.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 245 kB/s | 16 kB 00:00 2024-03-19T03:05:27.060 INFO:teuthology.orchestra.run.smithi119.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-03-19T03:05:27.085 INFO:teuthology.orchestra.run.smithi119.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 375 kB/s | 28 kB 00:00 2024-03-19T03:05:27.127 INFO:teuthology.orchestra.run.smithi119.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 460 kB/s | 31 kB 00:00 2024-03-19T03:05:27.152 INFO:teuthology.orchestra.run.smithi119.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 2.7 MB/s | 188 kB 00:00 2024-03-19T03:05:27.194 INFO:teuthology.orchestra.run.smithi119.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 885 kB/s | 59 kB 00:00 2024-03-19T03:05:27.219 INFO:teuthology.orchestra.run.smithi119.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 535 kB/s | 36 kB 00:00 2024-03-19T03:05:27.261 INFO:teuthology.orchestra.run.smithi119.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 762 kB/s | 51 kB 00:00 2024-03-19T03:05:27.295 INFO:teuthology.orchestra.run.smithi119.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-19T03:05:27.329 INFO:teuthology.orchestra.run.smithi119.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-03-19T03:05:27.370 INFO:teuthology.orchestra.run.smithi119.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.6 MB/s | 427 kB 00:00 2024-03-19T03:05:27.396 INFO:teuthology.orchestra.run.smithi119.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 299 kB/s | 20 kB 00:00 2024-03-19T03:05:27.438 INFO:teuthology.orchestra.run.smithi119.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-19T03:05:27.505 INFO:teuthology.orchestra.run.smithi119.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-19T03:05:27.797 INFO:teuthology.orchestra.run.smithi119.stdout:(150/174): librbd1-19.0.0-2206.gb6ae2491.el9.x8 11 MB/s | 3.1 MB 00:00 2024-03-19T03:05:27.847 INFO:teuthology.orchestra.run.smithi119.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-19T03:05:28.489 INFO:teuthology.orchestra.run.smithi119.stdout:(152/174): librados2-19.0.0-2206.gb6ae2491.el9. 3.2 MB/s | 3.4 MB 00:01 2024-03-19T03:05:28.948 INFO:teuthology.orchestra.run.smithi119.stdout:(153/174): ceph-test-19.0.0-2206.gb6ae2491.el9. 4.7 MB/s | 48 MB 00:10 2024-03-19T03:05:28.998 INFO:teuthology.orchestra.run.smithi119.stdout:(154/174): NetworkManager-1.47.2-1.el9.x86_64.r 2.0 MB/s | 2.3 MB 00:01 2024-03-19T03:05:29.106 INFO:teuthology.orchestra.run.smithi119.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.2 MB/s | 248 kB 00:00 2024-03-19T03:05:29.215 INFO:teuthology.orchestra.run.smithi119.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 658 kB/s | 71 kB 00:00 2024-03-19T03:05:29.299 INFO:teuthology.orchestra.run.smithi119.stdout:(157/174): NetworkManager-team-1.47.2-1.el9.x86 120 kB/s | 42 kB 00:00 2024-03-19T03:05:29.324 INFO:teuthology.orchestra.run.smithi119.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 336 kB/s | 36 kB 00:00 2024-03-19T03:05:29.433 INFO:teuthology.orchestra.run.smithi119.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.3 MB/s | 256 kB 00:00 2024-03-19T03:05:29.474 INFO:teuthology.orchestra.run.smithi119.stdout:(160/174): NetworkManager-libnm-1.47.2-1.el9.x8 1.9 MB/s | 1.8 MB 00:00 2024-03-19T03:05:29.533 INFO:teuthology.orchestra.run.smithi119.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 230 kB/s | 23 kB 00:00 2024-03-19T03:05:29.567 INFO:teuthology.orchestra.run.smithi119.stdout:(162/174): elfutils-libelf-0.190-2.el9.x86_64.r 718 kB/s | 192 kB 00:00 2024-03-19T03:05:29.592 INFO:teuthology.orchestra.run.smithi119.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 925 kB/s | 108 kB 00:00 2024-03-19T03:05:29.642 INFO:teuthology.orchestra.run.smithi119.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 155 kB 00:00 2024-03-19T03:05:29.676 INFO:teuthology.orchestra.run.smithi119.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 136 kB 00:00 2024-03-19T03:05:29.701 INFO:teuthology.orchestra.run.smithi119.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 586 kB/s | 64 kB 00:00 2024-03-19T03:05:29.751 INFO:teuthology.orchestra.run.smithi119.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 263 kB/s | 29 kB 00:00 2024-03-19T03:05:30.052 INFO:teuthology.orchestra.run.smithi119.stdout:(168/174): util-linux-2.37.4-18.el9.x86_64.rpm 7.6 MB/s | 2.3 MB 00:00 2024-03-19T03:05:30.085 INFO:teuthology.orchestra.run.smithi119.stdout:(169/174): openssl-3.0.7-27.el9.x86_64.rpm 3.0 MB/s | 1.2 MB 00:00 2024-03-19T03:05:30.194 INFO:teuthology.orchestra.run.smithi119.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 287 kB/s | 31 kB 00:00 2024-03-19T03:05:30.403 INFO:teuthology.orchestra.run.smithi119.stdout:(171/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 5.2 MB/s | 1.1 MB 00:00 2024-03-19T03:05:30.486 INFO:teuthology.orchestra.run.smithi119.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 892 kB/s | 74 kB 00:00 2024-03-19T03:05:30.603 INFO:teuthology.orchestra.run.smithi119.stdout:(173/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 2.4 MB/s | 2.2 MB 00:00 2024-03-19T03:05:30.628 INFO:teuthology.orchestra.run.smithi119.stdout:(174/174): util-linux-core-2.37.4-18.el9.x86_64 807 kB/s | 465 kB 00:00 2024-03-19T03:05:30.633 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-19T03:05:30.633 INFO:teuthology.orchestra.run.smithi119.stdout:Total 15 MB/s | 224 MB 00:15 2024-03-19T03:05:31.123 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-19T03:05:31.396 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-19T03:05:31.397 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-19T03:05:33.123 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-19T03:05:33.124 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-19T03:05:40.006 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-19T03:05:40.322 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-19T03:05:40.416 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-19T03:05:40.455 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T03:05:40.465 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T03:05:40.505 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-19T03:05:40.558 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-19T03:05:40.593 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-19T03:05:40.618 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-19T03:05:40.669 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-19T03:05:40.991 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-19T03:05:41.052 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T03:05:41.174 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T03:05:41.203 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T03:05:41.278 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T03:05:41.307 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T03:05:41.347 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-19T03:05:41.525 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-19T03:05:41.580 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-19T03:05:41.608 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T03:05:41.703 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T03:05:41.726 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T03:05:41.773 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T03:05:41.813 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-19T03:05:41.836 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T03:05:41.852 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T03:05:42.011 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T03:05:42.042 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T03:05:42.082 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-19T03:05:42.187 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-19T03:05:42.309 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T03:05:42.353 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T03:05:42.433 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T03:05:42.495 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T03:05:42.713 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T03:05:42.818 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-19T03:05:42.913 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-19T03:05:42.950 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-19T03:05:42.985 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-19T03:05:43.029 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-19T03:05:43.104 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-19T03:05:43.157 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-19T03:05:43.196 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-19T03:05:43.260 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-19T03:05:43.303 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-19T03:05:43.354 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-19T03:05:43.387 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-19T03:05:43.415 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-19T03:05:43.486 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-19T03:05:43.520 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 39/198 2024-03-19T03:05:43.563 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 40/198 2024-03-19T03:05:43.592 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 41/198 2024-03-19T03:05:43.662 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-19T03:05:43.707 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-19T03:05:43.781 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-19T03:05:43.926 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-19T03:05:43.976 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-19T03:05:44.501 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-19T03:05:44.580 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-19T03:05:44.661 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-19T03:05:44.700 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T03:05:44.708 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T03:05:44.740 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-19T03:05:44.789 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-19T03:05:44.827 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 53/198 2024-03-19T03:05:44.856 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-19T03:05:45.079 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-19T03:05:45.233 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-19T03:05:45.292 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-19T03:05:45.352 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-19T03:05:45.621 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-19T03:05:45.743 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-19T03:05:45.815 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-19T03:05:45.888 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-19T03:05:45.923 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-19T03:05:45.960 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-19T03:05:46.008 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-19T03:05:46.068 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-19T03:05:46.120 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-19T03:05:46.162 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-19T03:05:46.299 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-19T03:05:46.350 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-19T03:05:46.382 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-19T03:05:46.430 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-19T03:05:46.533 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-19T03:05:46.563 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-19T03:05:46.603 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-19T03:05:46.629 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-19T03:05:46.654 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-19T03:05:46.726 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-19T03:05:46.848 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-19T03:05:46.945 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-19T03:05:46.995 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-19T03:05:47.053 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-19T03:05:47.108 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-19T03:05:47.296 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-19T03:05:47.349 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-19T03:05:47.495 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-19T03:05:47.539 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-19T03:05:47.570 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-19T03:05:47.602 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-19T03:05:47.630 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-19T03:05:47.664 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-19T03:05:47.703 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-19T03:05:47.735 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-19T03:05:47.765 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-19T03:05:47.851 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-19T03:05:47.892 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-19T03:05:47.925 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-19T03:05:47.967 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-19T03:05:48.001 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-19T03:05:48.092 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-19T03:05:48.182 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-19T03:05:48.249 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-19T03:05:48.289 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-19T03:05:48.337 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-19T03:05:48.436 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-19T03:05:48.491 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-19T03:05:49.238 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-19T03:05:49.296 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-19T03:05:49.329 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-19T03:05:49.378 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-19T03:05:49.466 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-19T03:05:49.497 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-19T03:05:49.674 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-19T03:05:49.703 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-19T03:05:49.781 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-19T03:05:50.437 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-19T03:05:50.552 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-19T03:05:51.800 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-19T03:05:51.895 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-19T03:05:51.926 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-19T03:05:51.957 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-19T03:05:51.984 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-19T03:05:52.307 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-19T03:05:52.339 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T03:05:52.432 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T03:05:52.463 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 125/198 2024-03-19T03:05:52.498 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-19T03:05:52.960 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-19T03:05:52.986 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T03:05:53.057 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T03:05:53.080 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 129/198 2024-03-19T03:05:55.309 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T03:05:55.342 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T03:05:55.885 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T03:05:55.920 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T03:05:55.985 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T03:05:55.985 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-19T03:05:55.985 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-19T03:05:55.985 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-19T03:05:56.013 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T03:06:07.831 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T03:06:07.831 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /sys 2024-03-19T03:06:07.831 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /proc 2024-03-19T03:06:07.831 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /mnt 2024-03-19T03:06:07.832 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /var/tmp 2024-03-19T03:06:07.832 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /home 2024-03-19T03:06:07.832 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /root 2024-03-19T03:06:07.832 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /tmp 2024-03-19T03:06:07.832 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-19T03:06:07.991 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T03:06:08.007 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T03:06:08.007 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:06:08.007 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-19T03:06:08.007 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-19T03:06:09.477 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T03:06:09.503 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T03:06:09.503 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:06:09.503 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-19T03:06:09.503 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T03:06:09.503 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T03:06:09.503 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-19T03:06:09.729 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T03:06:09.752 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T03:06:09.752 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:06:09.752 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-19T03:06:09.752 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T03:06:09.752 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T03:06:09.753 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-19T03:06:10.205 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T03:06:10.229 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T03:06:10.229 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:06:10.229 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-19T03:06:10.229 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T03:06:10.229 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T03:06:10.229 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-19T03:06:10.324 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-19T03:06:10.666 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T03:06:10.697 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T03:06:10.729 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 139/198 2024-03-19T03:06:10.789 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 140/198 2024-03-19T03:06:10.812 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T03:06:11.047 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T03:06:11.079 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T03:06:12.049 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T03:06:12.071 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T03:06:12.172 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T03:06:12.336 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 144/198 2024-03-19T03:06:12.365 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T03:06:12.388 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T03:06:12.389 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:06:12.389 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-19T03:06:12.389 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T03:06:12.389 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T03:06:12.389 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-19T03:06:12.432 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T03:06:12.444 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T03:06:12.489 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 147/198 2024-03-19T03:06:13.382 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-19T03:06:13.410 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T03:06:13.432 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T03:06:13.433 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:06:13.433 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-19T03:06:13.433 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T03:06:13.433 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T03:06:13.433 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-19T03:06:13.477 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T03:06:13.499 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T03:06:13.499 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:06:13.499 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-19T03:06:13.499 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-19T03:06:13.544 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T03:06:13.566 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T03:06:13.567 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:06:13.567 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-19T03:06:13.567 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T03:06:13.567 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T03:06:13.567 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-19T03:06:13.878 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T03:06:13.901 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T03:06:13.901 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:06:13.901 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-19T03:06:13.901 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T03:06:13.902 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T03:06:13.902 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-19T03:06:18.357 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T03:06:18.407 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 154/198 2024-03-19T03:06:18.446 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 155/198 2024-03-19T03:06:18.473 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-19T03:06:18.655 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 157/198 2024-03-19T03:06:18.764 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-19T03:06:18.830 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 159/198 2024-03-19T03:06:18.869 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-19T03:06:18.930 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-19T03:06:18.973 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-19T03:06:19.005 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 163/198 2024-03-19T03:06:19.046 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-19T03:06:19.076 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-19T03:06:19.122 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-19T03:06:19.183 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-19T03:06:19.215 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-19T03:06:19.789 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T03:06:19.830 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T03:06:19.850 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T03:06:20.685 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 170/198 2024-03-19T03:06:20.783 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-19T03:06:20.824 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-19T03:06:20.906 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-19T03:06:20.937 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-19T03:06:20.962 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-19T03:06:20.997 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-19T03:06:21.022 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-19T03:06:21.022 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T03:06:21.110 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T03:06:21.140 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-19T03:06:21.140 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T03:06:21.224 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T03:06:21.250 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-19T03:06:21.292 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-19T03:06:21.326 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-19T03:06:21.351 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-19T03:06:21.384 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-19T03:06:21.424 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-19T03:06:21.456 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-19T03:06:21.481 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-19T03:06:21.515 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-19T03:06:21.540 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-19T03:06:21.565 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-19T03:06:21.572 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T03:06:21.572 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T03:06:21.751 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T03:06:21.751 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T03:06:21.834 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T03:06:21.867 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-19T03:06:21.899 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-19T03:06:21.927 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-19T03:06:21.960 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-19T03:06:22.025 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T03:06:26.808 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T03:06:26.808 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 1/198 2024-03-19T03:06:26.808 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2/198 2024-03-19T03:06:26.809 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 3/198 2024-03-19T03:06:26.809 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 4/198 2024-03-19T03:06:26.809 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 5/198 2024-03-19T03:06:26.809 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 6/198 2024-03-19T03:06:26.809 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 7/198 2024-03-19T03:06:26.809 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 8/198 2024-03-19T03:06:26.809 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 9/198 2024-03-19T03:06:26.809 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T03:06:26.809 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T03:06:26.809 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T03:06:26.809 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 13/198 2024-03-19T03:06:26.809 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 14/198 2024-03-19T03:06:26.809 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 15/198 2024-03-19T03:06:26.810 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T03:06:26.810 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 17/198 2024-03-19T03:06:26.810 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 18/198 2024-03-19T03:06:26.810 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 19/198 2024-03-19T03:06:26.810 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 20/198 2024-03-19T03:06:26.810 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 21/198 2024-03-19T03:06:26.810 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 22/198 2024-03-19T03:06:26.810 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 23/198 2024-03-19T03:06:26.810 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 24/198 2024-03-19T03:06:26.810 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 25/198 2024-03-19T03:06:26.810 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 26/198 2024-03-19T03:06:26.811 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 27/198 2024-03-19T03:06:26.811 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 28/198 2024-03-19T03:06:26.811 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 29/198 2024-03-19T03:06:26.811 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 30/198 2024-03-19T03:06:26.826 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 31/198 2024-03-19T03:06:26.826 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 32/198 2024-03-19T03:06:26.827 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 33/198 2024-03-19T03:06:26.827 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 34/198 2024-03-19T03:06:26.827 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 35/198 2024-03-19T03:06:26.827 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 36/198 2024-03-19T03:06:26.827 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 37/198 2024-03-19T03:06:26.827 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 38/198 2024-03-19T03:06:26.827 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 39/198 2024-03-19T03:06:26.827 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-19T03:06:26.827 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-19T03:06:26.827 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-19T03:06:26.827 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-19T03:06:26.827 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-19T03:06:26.827 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-19T03:06:26.828 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-19T03:06:26.828 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-19T03:06:26.828 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-19T03:06:26.828 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-19T03:06:26.828 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-19T03:06:26.828 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-19T03:06:26.828 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-19T03:06:26.828 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-19T03:06:26.828 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-19T03:06:26.829 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-19T03:06:26.829 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-19T03:06:26.829 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-19T03:06:26.829 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-19T03:06:26.829 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-19T03:06:26.829 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-19T03:06:26.829 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-19T03:06:26.829 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-19T03:06:26.829 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-19T03:06:26.829 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-19T03:06:26.829 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-19T03:06:26.829 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-19T03:06:26.829 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-19T03:06:26.830 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-19T03:06:26.830 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-19T03:06:26.830 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-19T03:06:26.830 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-19T03:06:26.830 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-19T03:06:26.830 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-19T03:06:26.830 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-19T03:06:26.830 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-19T03:06:26.830 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-19T03:06:26.830 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-19T03:06:26.830 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-19T03:06:26.830 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-19T03:06:26.831 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-19T03:06:26.831 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-19T03:06:26.831 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-19T03:06:26.831 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-19T03:06:26.831 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-19T03:06:26.831 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-19T03:06:26.831 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-19T03:06:26.831 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-19T03:06:26.831 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-19T03:06:26.831 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-19T03:06:26.832 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-19T03:06:26.832 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-19T03:06:26.832 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-19T03:06:26.832 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-19T03:06:26.832 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-19T03:06:26.832 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-19T03:06:26.832 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-19T03:06:26.832 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-19T03:06:26.832 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-19T03:06:26.832 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-19T03:06:26.832 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-19T03:06:26.832 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-19T03:06:26.832 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-19T03:06:26.832 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-19T03:06:26.833 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-19T03:06:26.833 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-19T03:06:26.833 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-19T03:06:26.833 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-19T03:06:26.833 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-19T03:06:26.833 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-19T03:06:26.833 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-19T03:06:26.833 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-19T03:06:26.833 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-19T03:06:26.833 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-19T03:06:26.833 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-19T03:06:26.833 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-19T03:06:26.833 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-19T03:06:26.833 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-19T03:06:26.834 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-19T03:06:26.834 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-19T03:06:26.834 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-19T03:06:26.834 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-19T03:06:26.834 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-19T03:06:26.834 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-19T03:06:26.834 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-19T03:06:26.834 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-19T03:06:26.834 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-19T03:06:26.834 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-19T03:06:26.834 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-19T03:06:26.834 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-19T03:06:26.834 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-19T03:06:26.834 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-19T03:06:26.835 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-19T03:06:26.835 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-19T03:06:26.835 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-19T03:06:26.835 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-19T03:06:26.835 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-19T03:06:26.835 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-19T03:06:26.835 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-19T03:06:26.835 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-19T03:06:26.835 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-19T03:06:26.835 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-19T03:06:26.835 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-19T03:06:26.835 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-19T03:06:26.835 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-19T03:06:26.836 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-19T03:06:26.836 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-19T03:06:26.836 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-19T03:06:26.836 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-19T03:06:26.836 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-19T03:06:26.836 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-19T03:06:26.836 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T03:06:26.836 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-19T03:06:26.836 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T03:06:26.836 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-19T03:06:26.836 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-19T03:06:26.836 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-19T03:06:26.836 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-19T03:06:26.836 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-19T03:06:26.837 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-19T03:06:26.837 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-19T03:06:26.837 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-19T03:06:26.837 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-19T03:06:26.837 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-19T03:06:26.837 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-19T03:06:26.837 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-19T03:06:26.837 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-19T03:06:26.837 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-19T03:06:26.837 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-19T03:06:26.837 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-19T03:06:26.837 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-19T03:06:26.837 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-19T03:06:26.837 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-19T03:06:26.838 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-19T03:06:26.838 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-19T03:06:26.838 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-19T03:06:26.838 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-19T03:06:26.838 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-19T03:06:26.838 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-19T03:06:26.838 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-19T03:06:26.838 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-19T03:06:26.838 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-19T03:06:26.838 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-19T03:06:26.838 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-19T03:06:26.838 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-19T03:06:26.838 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-19T03:06:26.839 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-19T03:06:26.839 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-19T03:06:26.839 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-19T03:06:26.839 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-19T03:06:26.839 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-19T03:06:26.839 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-19T03:06:26.840 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-19T03:06:26.840 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-19T03:06:26.840 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-19T03:06:26.840 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-19T03:06:26.840 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-19T03:06:26.840 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-19T03:06:29.978 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-19T03:06:29.978 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-19T03:06:29.978 INFO:teuthology.orchestra.run.smithi119.stdout:Upgraded: 2024-03-19T03:06:29.978 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-19T03:06:29.978 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-19T03:06:29.979 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-19T03:06:29.979 INFO:teuthology.orchestra.run.smithi119.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-19T03:06:29.979 INFO:teuthology.orchestra.run.smithi119.stdout: acl-2.3.1-4.el9.x86_64 2024-03-19T03:06:29.979 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-19T03:06:29.979 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-19T03:06:29.979 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-19T03:06:29.980 INFO:teuthology.orchestra.run.smithi119.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-19T03:06:29.980 INFO:teuthology.orchestra.run.smithi119.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-19T03:06:29.980 INFO:teuthology.orchestra.run.smithi119.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-19T03:06:29.980 INFO:teuthology.orchestra.run.smithi119.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-19T03:06:29.980 INFO:teuthology.orchestra.run.smithi119.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-19T03:06:29.980 INFO:teuthology.orchestra.run.smithi119.stdout: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:29.981 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:29.981 INFO:teuthology.orchestra.run.smithi119.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-19T03:06:29.981 INFO:teuthology.orchestra.run.smithi119.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-19T03:06:29.981 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-19T03:06:29.981 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-19T03:06:29.981 INFO:teuthology.orchestra.run.smithi119.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-19T03:06:29.982 INFO:teuthology.orchestra.run.smithi119.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-19T03:06:29.982 INFO:teuthology.orchestra.run.smithi119.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-19T03:06:29.982 INFO:teuthology.orchestra.run.smithi119.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-19T03:06:29.982 INFO:teuthology.orchestra.run.smithi119.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-19T03:06:29.982 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-19T03:06:29.982 INFO:teuthology.orchestra.run.smithi119.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-19T03:06:29.983 INFO:teuthology.orchestra.run.smithi119.stdout: automake-1.16.2-8.el9.noarch 2024-03-19T03:06:29.984 INFO:teuthology.orchestra.run.smithi119.stdout: bison-3.7.4-5.el9.x86_64 2024-03-19T03:06:29.984 INFO:teuthology.orchestra.run.smithi119.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-19T03:06:29.984 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:29.984 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:29.984 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:29.984 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:29.985 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:06:29.985 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:29.985 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:29.985 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:29.985 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:06:29.985 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:06:29.986 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:06:29.986 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:06:29.986 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:06:29.986 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:29.986 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:29.986 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:06:29.987 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:29.987 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:29.987 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:29.987 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:06:29.987 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:06:29.988 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:29.988 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:29.988 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:06:29.988 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-19T03:06:29.988 INFO:teuthology.orchestra.run.smithi119.stdout: flex-2.6.4-9.el9.x86_64 2024-03-19T03:06:29.988 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-19T03:06:29.989 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-19T03:06:29.989 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-19T03:06:29.989 INFO:teuthology.orchestra.run.smithi119.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-19T03:06:29.989 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-19T03:06:29.989 INFO:teuthology.orchestra.run.smithi119.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-19T03:06:29.989 INFO:teuthology.orchestra.run.smithi119.stdout: jq-1.6-16.el9.x86_64 2024-03-19T03:06:29.990 INFO:teuthology.orchestra.run.smithi119.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-19T03:06:29.990 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-19T03:06:29.990 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-19T03:06:29.990 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:29.991 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:29.991 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:29.991 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-19T03:06:29.991 INFO:teuthology.orchestra.run.smithi119.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-19T03:06:29.991 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-19T03:06:29.991 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-19T03:06:29.992 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-19T03:06:29.992 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:29.992 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:29.992 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-19T03:06:29.992 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:29.992 INFO:teuthology.orchestra.run.smithi119.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-19T03:06:29.992 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-19T03:06:29.992 INFO:teuthology.orchestra.run.smithi119.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-19T03:06:29.992 INFO:teuthology.orchestra.run.smithi119.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-19T03:06:29.993 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-19T03:06:29.993 INFO:teuthology.orchestra.run.smithi119.stdout: lua-5.4.4-4.el9.x86_64 2024-03-19T03:06:29.993 INFO:teuthology.orchestra.run.smithi119.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-19T03:06:29.993 INFO:teuthology.orchestra.run.smithi119.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-19T03:06:29.993 INFO:teuthology.orchestra.run.smithi119.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-19T03:06:29.993 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-19T03:06:29.993 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-19T03:06:29.994 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-19T03:06:29.994 INFO:teuthology.orchestra.run.smithi119.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-19T03:06:29.994 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-13.11-1.el9.x86_64 2024-03-19T03:06:29.994 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-19T03:06:29.994 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-19T03:06:29.995 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-19T03:06:29.995 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-19T03:06:29.995 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-19T03:06:29.995 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-19T03:06:29.995 INFO:teuthology.orchestra.run.smithi119.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-19T03:06:29.995 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-19T03:06:29.995 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-19T03:06:29.995 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-19T03:06:29.996 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:29.996 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:29.996 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:29.996 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-19T03:06:29.996 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-19T03:06:29.996 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-19T03:06:29.997 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-19T03:06:29.997 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-19T03:06:29.997 INFO:teuthology.orchestra.run.smithi119.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-19T03:06:29.997 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-19T03:06:29.997 INFO:teuthology.orchestra.run.smithi119.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-19T03:06:29.997 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-19T03:06:29.997 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-19T03:06:29.997 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-19T03:06:29.997 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-19T03:06:29.997 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-19T03:06:29.997 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-19T03:06:29.997 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-19T03:06:29.997 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-19T03:06:29.998 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-19T03:06:29.998 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-19T03:06:29.998 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-19T03:06:29.998 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-19T03:06:29.998 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-19T03:06:29.998 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-19T03:06:29.998 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-19T03:06:29.998 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-19T03:06:29.998 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-19T03:06:29.998 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-19T03:06:29.998 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-19T03:06:29.998 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-19T03:06:29.999 INFO:teuthology.orchestra.run.smithi119.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-19T03:06:29.999 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-19T03:06:29.999 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ply-3.11-14.el9.noarch 2024-03-19T03:06:29.999 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-19T03:06:29.999 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-19T03:06:29.999 INFO:teuthology.orchestra.run.smithi119.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-19T03:06:29.999 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-19T03:06:29.999 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-19T03:06:29.999 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-19T03:06:30.000 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-19T03:06:30.000 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-19T03:06:30.000 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-19T03:06:30.000 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:30.000 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:30.000 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-19T03:06:30.000 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-19T03:06:30.001 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-19T03:06:30.001 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:30.001 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-19T03:06:30.001 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-19T03:06:30.001 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-19T03:06:30.001 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-19T03:06:30.001 INFO:teuthology.orchestra.run.smithi119.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-19T03:06:30.001 INFO:teuthology.orchestra.run.smithi119.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-19T03:06:30.002 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-19T03:06:30.002 INFO:teuthology.orchestra.run.smithi119.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-19T03:06:30.002 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-19T03:06:30.002 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-19T03:06:30.002 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-19T03:06:30.002 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-19T03:06:30.003 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-19T03:06:30.003 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-19T03:06:30.003 INFO:teuthology.orchestra.run.smithi119.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-19T03:06:30.003 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:30.003 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:30.004 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:06:30.004 INFO:teuthology.orchestra.run.smithi119.stdout: re2-1:20211101-3.el9.x86_64 2024-03-19T03:06:30.004 INFO:teuthology.orchestra.run.smithi119.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-19T03:06:30.004 INFO:teuthology.orchestra.run.smithi119.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-19T03:06:30.004 INFO:teuthology.orchestra.run.smithi119.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-19T03:06:30.004 INFO:teuthology.orchestra.run.smithi119.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-19T03:06:30.004 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-19T03:06:30.005 INFO:teuthology.orchestra.run.smithi119.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-19T03:06:30.006 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-19T03:06:30.006 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-19T03:06:30.510 DEBUG:teuthology.parallel:result is None 2024-03-19T03:09:10.650 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - BaseOS 31 kB/s | 8.0 MB 04:25 2024-03-19T03:09:38.525 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - CRB 23 kB/s | 6.1 MB 04:33 2024-03-19T03:09:40.425 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 16 kB 00:00 2024-03-19T03:09:56.389 INFO:teuthology.orchestra.run.smithi117.stdout:Extra Packages for Enterprise Linux 1.3 MB/s | 21 MB 00:15 2024-03-19T03:10:02.090 INFO:teuthology.orchestra.run.smithi117.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-03-19T03:10:04.691 INFO:teuthology.orchestra.run.smithi117.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T03:10:04.693 INFO:teuthology.orchestra.run.smithi117.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T03:10:04.717 INFO:teuthology.orchestra.run.smithi117.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-19T03:10:04.718 INFO:teuthology.orchestra.run.smithi117.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-19T03:10:04.719 INFO:teuthology.orchestra.run.smithi117.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-19T03:10:04.721 INFO:teuthology.orchestra.run.smithi117.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-19T03:10:04.724 INFO:teuthology.orchestra.run.smithi117.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-19T03:10:04.728 INFO:teuthology.orchestra.run.smithi117.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-19T03:10:04.729 INFO:teuthology.orchestra.run.smithi117.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-19T03:10:04.818 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-19T03:10:04.837 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================================ 2024-03-19T03:10:04.837 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-19T03:10:04.837 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================================ 2024-03-19T03:10:04.837 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-19T03:10:04.837 INFO:teuthology.orchestra.run.smithi117.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-19T03:10:04.837 INFO:teuthology.orchestra.run.smithi117.stdout: ceph x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 6.6 k 2024-03-19T03:10:04.837 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.4 M 2024-03-19T03:10:04.837 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 891 k 2024-03-19T03:10:04.837 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 145 k 2024-03-19T03:10:04.838 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 1.5 M 2024-03-19T03:10:04.838 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 139 k 2024-03-19T03:10:04.838 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 1.8 M 2024-03-19T03:10:04.838 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 7.4 M 2024-03-19T03:10:04.838 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-rook noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 49 k 2024-03-19T03:10:04.838 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 9.8 M 2024-03-19T03:10:04.838 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 48 M 2024-03-19T03:10:04.838 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-volume noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 264 k 2024-03-19T03:10:04.838 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 749 k 2024-03-19T03:10:04.838 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 221 k 2024-03-19T03:10:04.838 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-shell x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 27 k 2024-03-19T03:10:04.838 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-top noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 18 k 2024-03-19T03:10:04.839 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-19T03:10:04.839 INFO:teuthology.orchestra.run.smithi117.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-19T03:10:04.839 INFO:teuthology.orchestra.run.smithi117.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-19T03:10:04.839 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 32 k 2024-03-19T03:10:04.839 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 746 k 2024-03-19T03:10:04.839 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 127 k 2024-03-19T03:10:04.839 INFO:teuthology.orchestra.run.smithi117.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-19T03:10:04.839 INFO:teuthology.orchestra.run.smithi117.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-19T03:10:04.839 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-19T03:10:04.839 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-19T03:10:04.839 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-19T03:10:04.839 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-19T03:10:04.840 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 162 k 2024-03-19T03:10:04.840 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 321 k 2024-03-19T03:10:04.840 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 298 k 2024-03-19T03:10:04.840 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 99 k 2024-03-19T03:10:04.840 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 85 k 2024-03-19T03:10:04.840 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T03:10:04.840 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 170 k 2024-03-19T03:10:04.840 INFO:teuthology.orchestra.run.smithi117.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-19T03:10:04.840 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrading: 2024-03-19T03:10:04.840 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-19T03:10:04.840 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-19T03:10:04.840 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-19T03:10:04.841 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-19T03:10:04.841 INFO:teuthology.orchestra.run.smithi117.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-19T03:10:04.841 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-19T03:10:04.841 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-19T03:10:04.841 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-19T03:10:04.841 INFO:teuthology.orchestra.run.smithi117.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-19T03:10:04.841 INFO:teuthology.orchestra.run.smithi117.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-19T03:10:04.841 INFO:teuthology.orchestra.run.smithi117.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-19T03:10:04.841 INFO:teuthology.orchestra.run.smithi117.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-19T03:10:04.841 INFO:teuthology.orchestra.run.smithi117.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-19T03:10:04.841 INFO:teuthology.orchestra.run.smithi117.stdout: librados2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.4 M 2024-03-19T03:10:04.841 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T03:10:04.841 INFO:teuthology.orchestra.run.smithi117.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-19T03:10:04.841 INFO:teuthology.orchestra.run.smithi117.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-19T03:10:04.842 INFO:teuthology.orchestra.run.smithi117.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-19T03:10:04.842 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-19T03:10:04.842 INFO:teuthology.orchestra.run.smithi117.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-19T03:10:04.842 INFO:teuthology.orchestra.run.smithi117.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-19T03:10:04.842 INFO:teuthology.orchestra.run.smithi117.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-19T03:10:04.842 INFO:teuthology.orchestra.run.smithi117.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-19T03:10:04.842 INFO:teuthology.orchestra.run.smithi117.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-19T03:10:04.842 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-19T03:10:04.842 INFO:teuthology.orchestra.run.smithi117.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-19T03:10:04.842 INFO:teuthology.orchestra.run.smithi117.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-19T03:10:04.842 INFO:teuthology.orchestra.run.smithi117.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-19T03:10:04.843 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 21 M 2024-03-19T03:10:04.843 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 34 k 2024-03-19T03:10:04.843 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 2.4 M 2024-03-19T03:10:04.843 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 248 k 2024-03-19T03:10:04.843 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 4.7 M 2024-03-19T03:10:04.843 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 17 M 2024-03-19T03:10:04.843 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 16 k 2024-03-19T03:10:04.843 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 25 k 2024-03-19T03:10:04.843 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-19T03:10:04.843 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-19T03:10:04.843 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-19T03:10:04.843 INFO:teuthology.orchestra.run.smithi117.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-19T03:10:04.843 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-19T03:10:04.844 INFO:teuthology.orchestra.run.smithi117.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-19T03:10:04.844 INFO:teuthology.orchestra.run.smithi117.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-19T03:10:04.844 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-19T03:10:04.844 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-19T03:10:04.844 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 163 k 2024-03-19T03:10:04.844 INFO:teuthology.orchestra.run.smithi117.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-19T03:10:04.844 INFO:teuthology.orchestra.run.smithi117.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-19T03:10:04.844 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-19T03:10:04.844 INFO:teuthology.orchestra.run.smithi117.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-19T03:10:04.844 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-19T03:10:04.844 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 501 k 2024-03-19T03:10:04.844 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-19T03:10:04.844 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.1 M 2024-03-19T03:10:04.845 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-19T03:10:04.845 INFO:teuthology.orchestra.run.smithi117.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-19T03:10:04.845 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-19T03:10:04.845 INFO:teuthology.orchestra.run.smithi117.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-19T03:10:04.845 INFO:teuthology.orchestra.run.smithi117.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-19T03:10:04.845 INFO:teuthology.orchestra.run.smithi117.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-19T03:10:04.845 INFO:teuthology.orchestra.run.smithi117.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-19T03:10:04.845 INFO:teuthology.orchestra.run.smithi117.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-19T03:10:04.845 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-19T03:10:04.845 INFO:teuthology.orchestra.run.smithi117.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-19T03:10:04.845 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-19T03:10:04.845 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-19T03:10:04.845 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-19T03:10:04.846 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-19T03:10:04.846 INFO:teuthology.orchestra.run.smithi117.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-19T03:10:04.846 INFO:teuthology.orchestra.run.smithi117.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-19T03:10:04.846 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-19T03:10:04.846 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-19T03:10:04.846 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 45 k 2024-03-19T03:10:04.846 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 131 k 2024-03-19T03:10:04.846 INFO:teuthology.orchestra.run.smithi117.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-19T03:10:04.846 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-19T03:10:04.846 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-19T03:10:04.847 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-19T03:10:04.847 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-19T03:10:04.847 INFO:teuthology.orchestra.run.smithi117.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-19T03:10:04.847 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-19T03:10:04.847 INFO:teuthology.orchestra.run.smithi117.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-19T03:10:04.847 INFO:teuthology.orchestra.run.smithi117.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-19T03:10:04.847 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-19T03:10:04.847 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-19T03:10:04.847 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-19T03:10:04.847 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-19T03:10:04.847 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-19T03:10:04.847 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-19T03:10:04.847 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-19T03:10:04.848 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-19T03:10:04.848 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-19T03:10:04.848 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-19T03:10:04.848 INFO:teuthology.orchestra.run.smithi117.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-19T03:10:04.848 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-19T03:10:04.848 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-19T03:10:04.848 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-19T03:10:04.848 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-19T03:10:04.848 INFO:teuthology.orchestra.run.smithi117.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-19T03:10:04.848 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-19T03:10:04.848 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-19T03:10:04.848 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-19T03:10:04.848 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-19T03:10:04.848 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-19T03:10:04.849 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-19T03:10:04.849 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-19T03:10:04.849 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-19T03:10:04.849 INFO:teuthology.orchestra.run.smithi117.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-19T03:10:04.849 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-19T03:10:04.849 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-19T03:10:04.849 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-19T03:10:04.849 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-19T03:10:04.849 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-19T03:10:04.849 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-19T03:10:04.849 INFO:teuthology.orchestra.run.smithi117.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-19T03:10:04.849 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-19T03:10:04.849 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-19T03:10:04.849 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-19T03:10:04.850 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-19T03:10:04.850 INFO:teuthology.orchestra.run.smithi117.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-19T03:10:04.850 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-19T03:10:04.850 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-19T03:10:04.850 INFO:teuthology.orchestra.run.smithi117.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-19T03:10:04.850 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-19T03:10:04.850 INFO:teuthology.orchestra.run.smithi117.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-19T03:10:04.850 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-19T03:10:04.850 INFO:teuthology.orchestra.run.smithi117.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-19T03:10:04.850 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-19T03:10:04.850 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-19T03:10:04.850 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-19T03:10:04.850 INFO:teuthology.orchestra.run.smithi117.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-19T03:10:04.851 INFO:teuthology.orchestra.run.smithi117.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-19T03:10:04.851 INFO:teuthology.orchestra.run.smithi117.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-19T03:10:04.851 INFO:teuthology.orchestra.run.smithi117.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-19T03:10:04.851 INFO:teuthology.orchestra.run.smithi117.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-19T03:10:04.851 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-19T03:10:04.851 INFO:teuthology.orchestra.run.smithi117.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-19T03:10:04.851 INFO:teuthology.orchestra.run.smithi117.stdout:Installing weak dependencies: 2024-03-19T03:10:04.851 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-19T03:10:04.851 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:10:04.851 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-19T03:10:04.851 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================================ 2024-03-19T03:10:04.851 INFO:teuthology.orchestra.run.smithi117.stdout:Install 150 Packages 2024-03-19T03:10:04.851 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrade 24 Packages 2024-03-19T03:10:04.851 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:10:04.852 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 224 M 2024-03-19T03:10:04.852 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-19T03:10:40.118 INFO:teuthology.orchestra.run.smithi117.stdout:(1/174): ceph-19.0.0-2206.gb6ae2491.el9.x86_64. 27 kB/s | 6.6 kB 00:00 2024-03-19T03:10:40.326 INFO:teuthology.orchestra.run.smithi117.stdout:(2/174): ceph-fuse-19.0.0-2206.gb6ae2491.el9.x8 4.2 MB/s | 891 kB 00:00 2024-03-19T03:10:40.376 INFO:teuthology.orchestra.run.smithi117.stdout:(3/174): ceph-immutable-object-cache-19.0.0-220 2.9 MB/s | 145 kB 00:00 2024-03-19T03:10:40.685 INFO:teuthology.orchestra.run.smithi117.stdout:(4/174): ceph-mds-19.0.0-2206.gb6ae2491.el9.x86 7.7 MB/s | 2.4 MB 00:00 2024-03-19T03:10:40.843 INFO:teuthology.orchestra.run.smithi117.stdout:(5/174): ceph-base-19.0.0-2206.gb6ae2491.el9.x8 5.5 MB/s | 5.4 MB 00:00 2024-03-19T03:10:40.927 INFO:teuthology.orchestra.run.smithi117.stdout:(6/174): ceph-mgr-19.0.0-2206.gb6ae2491.el9.x86 6.2 MB/s | 1.5 MB 00:00 2024-03-19T03:10:41.336 INFO:teuthology.orchestra.run.smithi117.stdout:(7/174): ceph-mon-19.0.0-2206.gb6ae2491.el9.x86 9.6 MB/s | 4.7 MB 00:00 2024-03-19T03:10:42.303 INFO:teuthology.orchestra.run.smithi117.stdout:(8/174): ceph-radosgw-19.0.0-2206.gb6ae2491.el9 10 MB/s | 9.8 MB 00:00 2024-03-19T03:10:42.345 INFO:teuthology.orchestra.run.smithi117.stdout:(9/174): ceph-selinux-19.0.0-2206.gb6ae2491.el9 621 kB/s | 25 kB 00:00 2024-03-19T03:10:42.803 INFO:teuthology.orchestra.run.smithi117.stdout:(10/174): ceph-osd-19.0.0-2206.gb6ae2491.el9.x8 9.0 MB/s | 17 MB 00:01 2024-03-19T03:10:42.862 INFO:teuthology.orchestra.run.smithi117.stdout:(11/174): cephfs-mirror-19.0.0-2206.gb6ae2491.e 3.8 MB/s | 221 kB 00:00 2024-03-19T03:10:42.937 INFO:teuthology.orchestra.run.smithi117.stdout:(12/174): cephfs-shell-19.0.0-2206.gb6ae2491.el 365 kB/s | 27 kB 00:00 2024-03-19T03:10:43.004 INFO:teuthology.orchestra.run.smithi117.stdout:(13/174): libcephfs-devel-19.0.0-2206.gb6ae2491 483 kB/s | 32 kB 00:00 2024-03-19T03:10:43.254 INFO:teuthology.orchestra.run.smithi117.stdout:(14/174): ceph-common-19.0.0-2206.gb6ae2491.el9 6.1 MB/s | 21 MB 00:03 2024-03-19T03:10:43.305 INFO:teuthology.orchestra.run.smithi117.stdout:(15/174): libcephfs2-19.0.0-2206.gb6ae2491.el9. 2.4 MB/s | 746 kB 00:00 2024-03-19T03:10:43.338 INFO:teuthology.orchestra.run.smithi117.stdout:(16/174): libcephsqlite-19.0.0-2206.gb6ae2491.e 1.9 MB/s | 163 kB 00:00 2024-03-19T03:10:43.372 INFO:teuthology.orchestra.run.smithi117.stdout:(17/174): librados-devel-19.0.0-2206.gb6ae2491. 1.9 MB/s | 127 kB 00:00 2024-03-19T03:10:43.464 INFO:teuthology.orchestra.run.smithi117.stdout:(18/174): libradosstriper1-19.0.0-2206.gb6ae249 3.9 MB/s | 501 kB 00:00 2024-03-19T03:10:43.564 INFO:teuthology.orchestra.run.smithi117.stdout:(19/174): python3-ceph-argparse-19.0.0-2206.gb6 454 kB/s | 45 kB 00:00 2024-03-19T03:10:43.681 INFO:teuthology.orchestra.run.smithi117.stdout:(20/174): python3-ceph-common-19.0.0-2206.gb6ae 1.1 MB/s | 131 kB 00:00 2024-03-19T03:10:43.789 INFO:teuthology.orchestra.run.smithi117.stdout:(21/174): python3-cephfs-19.0.0-2206.gb6ae2491. 1.5 MB/s | 162 kB 00:00 2024-03-19T03:10:43.898 INFO:teuthology.orchestra.run.smithi117.stdout:(22/174): librgw2-19.0.0-2206.gb6ae2491.el9.x86 9.8 MB/s | 5.1 MB 00:00 2024-03-19T03:10:43.940 INFO:teuthology.orchestra.run.smithi117.stdout:(23/174): python3-rados-19.0.0-2206.gb6ae2491.e 2.1 MB/s | 321 kB 00:00 2024-03-19T03:10:43.982 INFO:teuthology.orchestra.run.smithi117.stdout:(24/174): python3-rbd-19.0.0-2206.gb6ae2491.el9 3.5 MB/s | 298 kB 00:00 2024-03-19T03:10:44.015 INFO:teuthology.orchestra.run.smithi117.stdout:(25/174): python3-rgw-19.0.0-2206.gb6ae2491.el9 1.3 MB/s | 99 kB 00:00 2024-03-19T03:10:44.041 INFO:teuthology.orchestra.run.smithi117.stdout:(26/174): rbd-fuse-19.0.0-2206.gb6ae2491.el9.x8 1.5 MB/s | 85 kB 00:00 2024-03-19T03:10:44.091 INFO:teuthology.orchestra.run.smithi117.stdout:(27/174): rbd-nbd-19.0.0-2206.gb6ae2491.el9.x86 3.4 MB/s | 170 kB 00:00 2024-03-19T03:10:44.158 INFO:teuthology.orchestra.run.smithi117.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2206.g 514 kB/s | 34 kB 00:00 2024-03-19T03:10:44.250 INFO:teuthology.orchestra.run.smithi117.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2206.gb6ae249 1.5 MB/s | 139 kB 00:00 2024-03-19T03:10:44.508 INFO:teuthology.orchestra.run.smithi117.stdout:(30/174): rbd-mirror-19.0.0-2206.gb6ae2491.el9. 6.3 MB/s | 3.1 MB 00:00 2024-03-19T03:10:44.567 INFO:teuthology.orchestra.run.smithi117.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2206.gb6ae2 5.7 MB/s | 1.8 MB 00:00 2024-03-19T03:10:44.659 INFO:teuthology.orchestra.run.smithi117.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2206.gb6 2.7 MB/s | 248 kB 00:00 2024-03-19T03:10:44.742 INFO:teuthology.orchestra.run.smithi117.stdout:(33/174): ceph-mgr-rook-19.0.0-2206.gb6ae2491.e 598 kB/s | 49 kB 00:00 2024-03-19T03:10:44.810 INFO:teuthology.orchestra.run.smithi117.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2206.gb 251 kB/s | 16 kB 00:00 2024-03-19T03:10:44.893 INFO:teuthology.orchestra.run.smithi117.stdout:(35/174): ceph-volume-19.0.0-2206.gb6ae2491.el9 3.1 MB/s | 264 kB 00:00 2024-03-19T03:10:45.060 INFO:teuthology.orchestra.run.smithi117.stdout:(36/174): cephadm-19.0.0-2206.gb6ae2491.el9.noa 4.4 MB/s | 749 kB 00:00 2024-03-19T03:10:45.135 INFO:teuthology.orchestra.run.smithi117.stdout:(37/174): cephfs-top-19.0.0-2206.gb6ae2491.el9. 245 kB/s | 18 kB 00:00 2024-03-19T03:10:45.319 INFO:teuthology.orchestra.run.smithi117.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 3.6 MB/s | 662 kB 00:00 2024-03-19T03:10:45.377 INFO:teuthology.orchestra.run.smithi117.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 16 MB/s | 920 kB 00:00 2024-03-19T03:10:45.494 INFO:teuthology.orchestra.run.smithi117.stdout:(40/174): ceph-mgr-diskprediction-local-19.0.0- 7.5 MB/s | 7.4 MB 00:00 2024-03-19T03:10:45.520 INFO:teuthology.orchestra.run.smithi117.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 749 kB/s | 106 kB 00:00 2024-03-19T03:10:45.553 INFO:teuthology.orchestra.run.smithi117.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.3 MB/s | 307 kB 00:00 2024-03-19T03:10:45.579 INFO:teuthology.orchestra.run.smithi117.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 570 kB/s | 32 kB 00:00 2024-03-19T03:10:45.679 INFO:teuthology.orchestra.run.smithi117.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 24 MB/s | 3.0 MB 00:00 2024-03-19T03:10:45.704 INFO:teuthology.orchestra.run.smithi117.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 157 kB/s | 17 kB 00:00 2024-03-19T03:10:46.188 INFO:teuthology.orchestra.run.smithi117.stdout:(46/174): ceph-test-19.0.0-2206.gb6ae2491.el9.x 12 MB/s | 48 MB 00:03 2024-03-19T03:10:46.213 INFO:teuthology.orchestra.run.smithi117.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 20 kB/s | 10 kB 00:00 2024-03-19T03:10:46.238 INFO:teuthology.orchestra.run.smithi117.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 299 kB/s | 159 kB 00:00 2024-03-19T03:10:46.351 INFO:teuthology.orchestra.run.smithi117.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 274 kB/s | 44 kB 00:00 2024-03-19T03:10:46.421 INFO:teuthology.orchestra.run.smithi117.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 3.6 MB/s | 661 kB 00:00 2024-03-19T03:10:46.455 INFO:teuthology.orchestra.run.smithi117.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 2.6 MB/s | 578 kB 00:00 2024-03-19T03:10:46.481 INFO:teuthology.orchestra.run.smithi117.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 394 kB/s | 46 kB 00:00 2024-03-19T03:10:46.506 INFO:teuthology.orchestra.run.smithi117.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 282 kB 00:00 2024-03-19T03:10:46.531 INFO:teuthology.orchestra.run.smithi117.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.5 MB/s | 187 kB 00:00 2024-03-19T03:10:46.565 INFO:teuthology.orchestra.run.smithi117.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.7 MB/s | 217 kB 00:00 2024-03-19T03:10:46.607 INFO:teuthology.orchestra.run.smithi117.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 471 kB/s | 34 kB 00:00 2024-03-19T03:10:46.732 INFO:teuthology.orchestra.run.smithi117.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 24 MB/s | 4.7 MB 00:00 2024-03-19T03:10:46.790 INFO:teuthology.orchestra.run.smithi117.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 7.3 MB/s | 1.5 MB 00:00 2024-03-19T03:10:46.832 INFO:teuthology.orchestra.run.smithi117.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 3.7 MB/s | 811 kB 00:00 2024-03-19T03:10:46.857 INFO:teuthology.orchestra.run.smithi117.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.1 MB/s | 134 kB 00:00 2024-03-19T03:10:46.916 INFO:teuthology.orchestra.run.smithi117.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 12 MB/s | 1.0 MB 00:00 2024-03-19T03:10:47.058 INFO:teuthology.orchestra.run.smithi117.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 25 MB/s | 5.7 MB 00:00 2024-03-19T03:10:47.083 INFO:teuthology.orchestra.run.smithi117.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 346 kB/s | 78 kB 00:00 2024-03-19T03:10:47.208 INFO:teuthology.orchestra.run.smithi117.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-19T03:10:47.225 INFO:teuthology.orchestra.run.smithi117.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 192 kB/s | 24 kB 00:00 2024-03-19T03:10:47.242 INFO:teuthology.orchestra.run.smithi117.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 126 kB/s | 18 kB 00:00 2024-03-19T03:10:47.275 INFO:teuthology.orchestra.run.smithi117.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 996 kB/s | 32 kB 00:00 2024-03-19T03:10:47.326 INFO:teuthology.orchestra.run.smithi117.stdout:(68/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 144 kB 00:00 2024-03-19T03:10:47.435 INFO:teuthology.orchestra.run.smithi117.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2024-03-19T03:10:47.459 INFO:teuthology.orchestra.run.smithi117.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 217 kB 00:00 2024-03-19T03:10:47.493 INFO:teuthology.orchestra.run.smithi117.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.1 MB/s | 174 kB 00:00 2024-03-19T03:10:47.527 INFO:teuthology.orchestra.run.smithi117.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 1.3 MB/s | 41 kB 00:00 2024-03-19T03:10:47.552 INFO:teuthology.orchestra.run.smithi117.stdout:(73/174): python3-packaging-20.9-5.el9.noarch.r 1.2 MB/s | 69 kB 00:00 2024-03-19T03:10:47.577 INFO:teuthology.orchestra.run.smithi117.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 2.8 MB/s | 238 kB 00:00 2024-03-19T03:10:47.603 INFO:teuthology.orchestra.run.smithi117.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.8 MB/s | 132 kB 00:00 2024-03-19T03:10:47.619 INFO:teuthology.orchestra.run.smithi117.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 47 kB 00:00 2024-03-19T03:10:47.653 INFO:teuthology.orchestra.run.smithi117.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.8 MB/s | 211 kB 00:00 2024-03-19T03:10:47.670 INFO:teuthology.orchestra.run.smithi117.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 644 kB/s | 43 kB 00:00 2024-03-19T03:10:47.820 INFO:teuthology.orchestra.run.smithi117.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 287 kB/s | 39 kB 00:00 2024-03-19T03:10:47.837 INFO:teuthology.orchestra.run.smithi117.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 271 kB/s | 41 kB 00:00 2024-03-19T03:10:48.029 INFO:teuthology.orchestra.run.smithi117.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-03-19T03:10:48.046 INFO:teuthology.orchestra.run.smithi117.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 275 kB/s | 56 kB 00:00 2024-03-19T03:10:48.079 INFO:teuthology.orchestra.run.smithi117.stdout:(83/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.2 MB/s | 300 kB 00:00 2024-03-19T03:10:48.096 INFO:teuthology.orchestra.run.smithi117.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 958 kB/s | 63 kB 00:00 2024-03-19T03:10:48.121 INFO:teuthology.orchestra.run.smithi117.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 614 kB/s | 44 kB 00:00 2024-03-19T03:10:48.155 INFO:teuthology.orchestra.run.smithi117.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 79 kB 00:00 2024-03-19T03:10:48.189 INFO:teuthology.orchestra.run.smithi117.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.6 MB/s | 241 kB 00:00 2024-03-19T03:10:48.238 INFO:teuthology.orchestra.run.smithi117.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 9.9 MB/s | 1.1 MB 00:00 2024-03-19T03:10:48.264 INFO:teuthology.orchestra.run.smithi117.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 103 kB 00:00 2024-03-19T03:10:48.289 INFO:teuthology.orchestra.run.smithi117.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.4 MB/s | 124 kB 00:00 2024-03-19T03:10:48.323 INFO:teuthology.orchestra.run.smithi117.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-19T03:13:02.441 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] jq-1.6-16.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/jq-1.6-16.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T03:13:02.441 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T03:13:02.442 INFO:teuthology.orchestra.run.smithi117.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T03:13:03.078 INFO:teuthology.orchestra.run.smithi117.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 kB/s | 195 kB 02:14 2024-03-19T03:13:03.095 INFO:teuthology.orchestra.run.smithi117.stdout:(93/174): jq-1.6-16.el9.x86_64.rpm 1.4 kB/s | 187 kB 02:14 2024-03-19T03:13:03.278 INFO:teuthology.orchestra.run.smithi117.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 625 kB/s | 125 kB 00:00 2024-03-19T03:13:03.362 INFO:teuthology.orchestra.run.smithi117.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 81 kB/s | 21 kB 00:00 2024-03-19T03:13:03.554 INFO:teuthology.orchestra.run.smithi117.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 93 kB/s | 25 kB 00:00 2024-03-19T03:13:03.588 INFO:teuthology.orchestra.run.smithi117.stdout:(97/174): libgfortran-11.4.1-3.el9.x86_64.rpm 5.9 kB/s | 803 kB 02:15 2024-03-19T03:13:04.005 INFO:teuthology.orchestra.run.smithi117.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 545 kB/s | 245 kB 00:00 2024-03-19T03:13:04.121 INFO:teuthology.orchestra.run.smithi117.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 1.8 MB/s | 216 kB 00:00 2024-03-19T03:13:04.197 INFO:teuthology.orchestra.run.smithi117.stdout:(100/174): python3-jinja2-2.11.3-5.el9.noarch.r 407 kB/s | 248 kB 00:00 2024-03-19T03:13:04.230 INFO:teuthology.orchestra.run.smithi117.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 323 kB/s | 35 kB 00:00 2024-03-19T03:13:04.306 INFO:teuthology.orchestra.run.smithi117.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 411 kB/s | 44 kB 00:00 2024-03-19T03:13:04.347 INFO:teuthology.orchestra.run.smithi117.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 806 kB/s | 94 kB 00:00 2024-03-19T03:13:04.439 INFO:teuthology.orchestra.run.smithi117.stdout:(104/174): openssl-devel-3.0.7-27.el9.x86_64.rp 3.8 MB/s | 4.1 MB 00:01 2024-03-19T03:13:04.465 INFO:teuthology.orchestra.run.smithi117.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 141 kB/s | 22 kB 00:00 2024-03-19T03:13:04.706 INFO:teuthology.orchestra.run.smithi117.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.3 MB/s | 309 kB 00:00 2024-03-19T03:13:04.740 INFO:teuthology.orchestra.run.smithi117.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.8 MB/s | 548 kB 00:00 2024-03-19T03:13:04.765 INFO:teuthology.orchestra.run.smithi117.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 336 kB/s | 19 kB 00:00 2024-03-19T03:13:04.807 INFO:teuthology.orchestra.run.smithi117.stdout:(109/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.8 MB/s | 863 kB 00:00 2024-03-19T03:13:04.832 INFO:teuthology.orchestra.run.smithi117.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 392 kB/s | 26 kB 00:00 2024-03-19T03:13:04.891 INFO:teuthology.orchestra.run.smithi117.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-19T03:13:04.960 INFO:teuthology.orchestra.run.smithi117.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-03-19T03:13:04.986 INFO:teuthology.orchestra.run.smithi117.stdout:(113/174): liboath-2.6.7-2.el9.x86_64.rpm 274 kB/s | 49 kB 00:00 2024-03-19T03:13:05.103 INFO:teuthology.orchestra.run.smithi117.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-03-19T03:13:05.144 INFO:teuthology.orchestra.run.smithi117.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.5 MB/s | 838 kB 00:00 2024-03-19T03:13:05.170 INFO:teuthology.orchestra.run.smithi117.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 655 kB/s | 43 kB 00:00 2024-03-19T03:13:05.204 INFO:teuthology.orchestra.run.smithi117.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 551 kB/s | 32 kB 00:00 2024-03-19T03:13:05.229 INFO:teuthology.orchestra.run.smithi117.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 259 kB/s | 15 kB 00:00 2024-03-19T03:13:05.262 INFO:teuthology.orchestra.run.smithi117.stdout:(119/174): python3-asyncssh-2.13.2-4.el9.noarch 1.9 MB/s | 548 kB 00:00 2024-03-19T03:13:05.297 INFO:teuthology.orchestra.run.smithi117.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-03-19T03:13:05.330 INFO:teuthology.orchestra.run.smithi117.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-03-19T03:13:05.355 INFO:teuthology.orchestra.run.smithi117.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 612 kB/s | 35 kB 00:00 2024-03-19T03:13:05.389 INFO:teuthology.orchestra.run.smithi117.stdout:(123/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-03-19T03:13:05.422 INFO:teuthology.orchestra.run.smithi117.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 2.2 MB/s | 203 kB 00:00 2024-03-19T03:13:05.464 INFO:teuthology.orchestra.run.smithi117.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-03-19T03:13:05.490 INFO:teuthology.orchestra.run.smithi117.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 162 kB/s | 11 kB 00:00 2024-03-19T03:13:05.540 INFO:teuthology.orchestra.run.smithi117.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 237 kB/s | 18 kB 00:00 2024-03-19T03:13:05.598 INFO:teuthology.orchestra.run.smithi117.stdout:(128/174): python3-grpcio-1.46.7-10.el9.x86_64. 8.4 MB/s | 2.0 MB 00:00 2024-03-19T03:13:05.624 INFO:teuthology.orchestra.run.smithi117.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 174 kB/s | 23 kB 00:00 2024-03-19T03:13:05.668 INFO:teuthology.orchestra.run.smithi117.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 153 kB/s | 19 kB 00:00 2024-03-19T03:13:05.708 INFO:teuthology.orchestra.run.smithi117.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 181 kB/s | 20 kB 00:00 2024-03-19T03:13:05.758 INFO:teuthology.orchestra.run.smithi117.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 519 kB/s | 46 kB 00:00 2024-03-19T03:13:05.800 INFO:teuthology.orchestra.run.smithi117.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 5.8 MB/s | 1.0 MB 00:00 2024-03-19T03:13:05.825 INFO:teuthology.orchestra.run.smithi117.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 873 kB/s | 58 kB 00:00 2024-03-19T03:13:05.850 INFO:teuthology.orchestra.run.smithi117.stdout:(135/174): python3-more-itertools-8.12.0-2.el9. 556 kB/s | 79 kB 00:00 2024-03-19T03:13:05.884 INFO:teuthology.orchestra.run.smithi117.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-19T03:13:05.913 INFO:teuthology.orchestra.run.smithi117.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 196 kB/s | 16 kB 00:00 2024-03-19T03:13:05.935 INFO:teuthology.orchestra.run.smithi117.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.1 MB/s | 90 kB 00:00 2024-03-19T03:13:05.969 INFO:teuthology.orchestra.run.smithi117.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 338 kB/s | 28 kB 00:00 2024-03-19T03:13:05.999 INFO:teuthology.orchestra.run.smithi117.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 370 kB/s | 31 kB 00:00 2024-03-19T03:13:06.027 INFO:teuthology.orchestra.run.smithi117.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-03-19T03:13:06.052 INFO:teuthology.orchestra.run.smithi117.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 710 kB/s | 59 kB 00:00 2024-03-19T03:13:06.078 INFO:teuthology.orchestra.run.smithi117.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 429 kB/s | 36 kB 00:00 2024-03-19T03:13:06.103 INFO:teuthology.orchestra.run.smithi117.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 679 kB/s | 51 kB 00:00 2024-03-19T03:13:06.153 INFO:teuthology.orchestra.run.smithi117.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.3 MB/s | 230 kB 00:00 2024-03-19T03:13:06.170 INFO:teuthology.orchestra.run.smithi117.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 981 kB/s | 90 kB 00:00 2024-03-19T03:13:06.212 INFO:teuthology.orchestra.run.smithi117.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.9 MB/s | 427 kB 00:00 2024-03-19T03:13:06.229 INFO:teuthology.orchestra.run.smithi117.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 267 kB/s | 20 kB 00:00 2024-03-19T03:13:06.271 INFO:teuthology.orchestra.run.smithi117.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-19T03:13:06.446 INFO:teuthology.orchestra.run.smithi117.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 6.8 MB/s | 1.6 MB 00:00 2024-03-19T03:13:06.779 INFO:teuthology.orchestra.run.smithi117.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 970 kB/s | 321 kB 00:00 2024-03-19T03:13:07.696 INFO:teuthology.orchestra.run.smithi117.stdout:(152/174): librados2-19.0.0-2206.gb6ae2491.el9. 2.3 MB/s | 3.4 MB 00:01 2024-03-19T03:13:07.922 INFO:teuthology.orchestra.run.smithi117.stdout:(153/174): librbd1-19.0.0-2206.gb6ae2491.el9.x8 1.9 MB/s | 3.1 MB 00:01 2024-03-19T03:13:08.338 INFO:teuthology.orchestra.run.smithi117.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 101 kB/s | 42 kB 00:00 2024-03-19T03:13:08.797 INFO:teuthology.orchestra.run.smithi117.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 543 kB/s | 248 kB 00:00 2024-03-19T03:13:08.989 INFO:teuthology.orchestra.run.smithi117.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 373 kB/s | 71 kB 00:00 2024-03-19T03:13:09.064 INFO:teuthology.orchestra.run.smithi117.stdout:(157/174): NetworkManager-1.47.2-1.el9.x86_64.r 1.0 MB/s | 2.3 MB 00:02 2024-03-19T03:13:09.114 INFO:teuthology.orchestra.run.smithi117.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 292 kB/s | 36 kB 00:00 2024-03-19T03:13:09.256 INFO:teuthology.orchestra.run.smithi117.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.0 MB/s | 192 kB 00:00 2024-03-19T03:13:09.390 INFO:teuthology.orchestra.run.smithi117.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 173 kB/s | 23 kB 00:00 2024-03-19T03:13:09.490 INFO:teuthology.orchestra.run.smithi117.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 682 kB/s | 256 kB 00:00 2024-03-19T03:13:09.524 INFO:teuthology.orchestra.run.smithi117.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 815 kB/s | 108 kB 00:00 2024-03-19T03:13:09.699 INFO:teuthology.orchestra.run.smithi117.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 744 kB/s | 155 kB 00:00 2024-03-19T03:13:09.724 INFO:teuthology.orchestra.run.smithi117.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 679 kB/s | 136 kB 00:00 2024-03-19T03:13:09.844 INFO:teuthology.orchestra.run.smithi117.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 444 kB/s | 64 kB 00:00 2024-03-19T03:13:09.894 INFO:teuthology.orchestra.run.smithi117.stdout:(166/174): NetworkManager-libnm-1.47.2-1.el9.x8 854 kB/s | 1.8 MB 00:02 2024-03-19T03:13:09.936 INFO:teuthology.orchestra.run.smithi117.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 136 kB/s | 29 kB 00:00 2024-03-19T03:13:10.819 INFO:teuthology.orchestra.run.smithi117.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 1.2 MB/s | 1.2 MB 00:00 2024-03-19T03:13:11.278 INFO:teuthology.orchestra.run.smithi117.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 1.0 MB/s | 465 kB 00:00 2024-03-19T03:13:11.411 INFO:teuthology.orchestra.run.smithi117.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 234 kB/s | 31 kB 00:00 2024-03-19T03:13:11.728 INFO:teuthology.orchestra.run.smithi117.stdout:(171/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 1.2 MB/s | 2.2 MB 00:01 2024-03-19T03:13:11.945 INFO:teuthology.orchestra.run.smithi117.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 2.3 MB 00:02 2024-03-19T03:13:12.162 INFO:teuthology.orchestra.run.smithi117.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 172 kB/s | 74 kB 00:00 2024-03-19T03:13:12.470 INFO:teuthology.orchestra.run.smithi117.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 1.0 MB/s | 1.1 MB 00:01 2024-03-19T03:13:12.478 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-19T03:13:12.478 INFO:teuthology.orchestra.run.smithi117.stdout:Total 1.2 MB/s | 224 MB 03:07 2024-03-19T03:13:13.888 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-19T03:13:14.260 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-19T03:13:14.260 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-19T03:13:16.678 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-19T03:13:16.678 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-19T03:13:23.844 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-19T03:13:23.943 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-19T03:13:23.990 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-19T03:13:24.038 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T03:13:24.081 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T03:13:24.177 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-19T03:13:24.276 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-19T03:13:24.338 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-19T03:13:24.370 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-19T03:13:24.465 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-19T03:13:24.874 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-19T03:13:24.918 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T03:13:25.100 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T03:13:25.127 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T03:13:25.323 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T03:13:25.366 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T03:13:25.435 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-19T03:13:25.735 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-19T03:13:25.790 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-19T03:13:25.815 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T03:13:25.965 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T03:13:26.000 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T03:13:26.089 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T03:13:26.136 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-19T03:13:26.169 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T03:13:26.234 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T03:13:26.536 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T03:13:26.585 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T03:13:26.635 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-19T03:13:27.086 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-19T03:13:27.184 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T03:13:27.274 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T03:13:27.492 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T03:13:27.545 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T03:13:27.896 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T03:13:28.081 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-19T03:13:28.212 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-19T03:13:28.251 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-19T03:13:28.287 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-19T03:13:28.344 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-19T03:13:28.452 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-19T03:13:28.526 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-19T03:13:28.586 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-19T03:13:28.696 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-19T03:13:28.741 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-19T03:13:28.802 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-19T03:13:28.844 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-19T03:13:28.877 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-19T03:13:28.986 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-19T03:13:29.030 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 39/198 2024-03-19T03:13:29.081 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 40/198 2024-03-19T03:13:29.112 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 41/198 2024-03-19T03:13:29.221 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-19T03:13:29.283 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-19T03:13:29.390 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-19T03:13:29.567 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-19T03:13:29.646 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-19T03:13:30.330 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-19T03:13:30.431 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-19T03:13:30.544 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-19T03:13:30.600 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T03:13:30.625 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T03:13:30.681 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-19T03:13:30.744 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-19T03:13:30.801 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 53/198 2024-03-19T03:13:30.849 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-19T03:13:31.259 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-19T03:13:31.473 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-19T03:13:31.570 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-19T03:13:31.626 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-19T03:13:31.843 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-19T03:13:31.900 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-19T03:13:31.985 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-19T03:13:32.089 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-19T03:13:32.134 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-19T03:13:32.183 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-19T03:13:32.254 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-19T03:13:32.299 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-19T03:13:32.364 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-19T03:13:32.429 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-19T03:13:32.618 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-19T03:13:32.679 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-19T03:13:32.717 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-19T03:13:32.777 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-19T03:13:32.915 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-19T03:13:32.957 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-19T03:13:33.004 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-19T03:13:33.040 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-19T03:13:33.073 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-19T03:13:33.225 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-19T03:13:33.396 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-19T03:13:33.549 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-19T03:13:33.610 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-19T03:13:33.673 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-19T03:13:33.759 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-19T03:13:33.999 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-19T03:13:34.064 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-19T03:13:34.242 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-19T03:13:34.286 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-19T03:13:34.325 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-19T03:13:34.381 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-19T03:13:34.423 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-19T03:13:34.477 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-19T03:13:34.555 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-19T03:13:34.614 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-19T03:13:34.661 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-19T03:13:34.787 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-19T03:13:34.830 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-19T03:13:34.865 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-19T03:13:34.918 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-19T03:13:34.960 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-19T03:13:35.103 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-19T03:13:35.225 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-19T03:13:35.282 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-19T03:13:35.318 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-19T03:13:35.374 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-19T03:13:35.513 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-19T03:13:35.583 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-19T03:13:36.564 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-19T03:13:36.616 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-19T03:13:36.649 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-19T03:13:36.706 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-19T03:13:36.826 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-19T03:13:36.867 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-19T03:13:37.097 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-19T03:13:37.130 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-19T03:13:37.230 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-19T03:13:38.017 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-19T03:13:38.151 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-19T03:13:39.670 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-19T03:13:39.759 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-19T03:13:39.797 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-19T03:13:39.828 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-19T03:13:39.868 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-19T03:13:40.241 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-19T03:13:40.275 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T03:13:40.377 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T03:13:40.406 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 125/198 2024-03-19T03:13:40.461 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-19T03:13:40.998 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-19T03:13:41.028 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T03:13:41.119 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T03:13:41.147 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 129/198 2024-03-19T03:13:43.993 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T03:13:44.027 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T03:13:44.646 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T03:13:44.688 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T03:13:44.840 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T03:13:44.840 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-19T03:13:44.840 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-19T03:13:44.840 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:13:44.889 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T03:13:57.591 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T03:13:57.591 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /sys 2024-03-19T03:13:57.591 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /proc 2024-03-19T03:13:57.591 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /mnt 2024-03-19T03:13:57.591 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /var/tmp 2024-03-19T03:13:57.591 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /home 2024-03-19T03:13:57.591 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /root 2024-03-19T03:13:57.591 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /tmp 2024-03-19T03:13:57.591 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:13:57.819 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T03:13:57.869 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T03:13:57.869 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:13:57.869 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-19T03:13:57.869 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:13:59.504 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T03:13:59.588 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T03:13:59.588 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:13:59.588 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-19T03:13:59.588 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T03:13:59.588 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T03:13:59.588 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:13:59.838 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T03:13:59.904 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T03:13:59.904 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:13:59.904 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-19T03:13:59.904 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T03:13:59.904 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T03:13:59.904 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:14:00.380 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T03:14:00.449 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T03:14:00.449 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:14:00.449 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-19T03:14:00.449 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T03:14:00.450 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T03:14:00.450 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:14:00.557 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-19T03:14:01.187 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T03:14:01.225 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T03:14:01.276 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 139/198 2024-03-19T03:14:01.375 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 140/198 2024-03-19T03:14:01.403 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T03:14:01.733 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T03:14:01.763 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T03:14:02.876 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T03:14:02.921 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T03:14:03.062 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T03:14:03.245 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 144/198 2024-03-19T03:14:03.282 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T03:14:03.352 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T03:14:03.352 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:14:03.352 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-19T03:14:03.352 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T03:14:03.352 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T03:14:03.352 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:14:03.407 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T03:14:03.444 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T03:14:03.505 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 147/198 2024-03-19T03:14:04.447 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-19T03:14:04.501 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T03:14:04.569 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T03:14:04.570 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:14:04.570 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-19T03:14:04.570 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T03:14:04.570 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T03:14:04.570 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:14:04.619 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T03:14:04.689 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T03:14:04.689 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:14:04.690 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-19T03:14:04.690 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:14:04.761 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T03:14:04.837 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T03:14:04.837 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:14:04.837 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-19T03:14:04.837 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T03:14:04.837 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T03:14:04.837 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:14:05.203 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T03:14:05.283 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T03:14:05.283 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:14:05.283 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-19T03:14:05.283 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T03:14:05.283 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T03:14:05.284 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:14:10.069 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T03:14:10.114 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 154/198 2024-03-19T03:14:10.150 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 155/198 2024-03-19T03:14:10.187 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-19T03:14:10.440 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 157/198 2024-03-19T03:14:10.562 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-19T03:14:10.609 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 159/198 2024-03-19T03:14:10.665 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-19T03:14:10.735 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-19T03:14:10.771 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-19T03:14:10.815 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 163/198 2024-03-19T03:14:10.870 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-19T03:14:10.920 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-19T03:14:11.010 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-19T03:14:11.132 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-19T03:14:11.169 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-19T03:14:12.065 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T03:14:12.107 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T03:14:12.162 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T03:14:13.482 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 170/198 2024-03-19T03:14:13.571 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-19T03:14:13.626 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-19T03:14:13.764 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-19T03:14:13.806 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-19T03:14:13.840 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-19T03:14:13.874 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-19T03:14:13.916 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-19T03:14:13.916 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T03:14:14.111 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T03:14:14.162 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-19T03:14:14.162 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T03:14:14.333 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T03:14:14.386 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-19T03:14:14.440 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-19T03:14:14.473 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-19T03:14:14.506 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-19T03:14:14.572 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-19T03:14:14.644 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-19T03:14:14.694 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-19T03:14:14.727 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-19T03:14:14.762 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-19T03:14:14.795 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-19T03:14:14.828 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-19T03:14:14.864 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T03:14:14.864 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T03:14:15.195 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T03:14:15.195 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T03:14:15.392 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T03:14:15.422 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-19T03:14:15.483 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-19T03:14:15.517 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-19T03:14:15.557 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-19T03:14:15.676 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T03:14:15.944 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T03:14:15.944 INFO:teuthology.orchestra.run.smithi117.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-19T03:14:15.944 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:14:19.406 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - AppStream 63 kB/s | 19 MB 05:07 2024-03-19T03:14:20.638 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 1/198 2024-03-19T03:14:20.638 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2/198 2024-03-19T03:14:20.638 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 3/198 2024-03-19T03:14:20.638 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 4/198 2024-03-19T03:14:20.638 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 5/198 2024-03-19T03:14:20.638 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 6/198 2024-03-19T03:14:20.639 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 7/198 2024-03-19T03:14:20.639 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 8/198 2024-03-19T03:14:20.639 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 9/198 2024-03-19T03:14:20.639 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T03:14:20.639 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T03:14:20.639 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T03:14:20.639 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 13/198 2024-03-19T03:14:20.639 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 14/198 2024-03-19T03:14:20.639 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 15/198 2024-03-19T03:14:20.639 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T03:14:20.639 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 17/198 2024-03-19T03:14:20.639 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 18/198 2024-03-19T03:14:20.639 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 19/198 2024-03-19T03:14:20.639 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 20/198 2024-03-19T03:14:20.640 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 21/198 2024-03-19T03:14:20.640 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 22/198 2024-03-19T03:14:20.640 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 23/198 2024-03-19T03:14:20.640 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 24/198 2024-03-19T03:14:20.640 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 25/198 2024-03-19T03:14:20.640 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 26/198 2024-03-19T03:14:20.640 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 27/198 2024-03-19T03:14:20.640 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 28/198 2024-03-19T03:14:20.640 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 29/198 2024-03-19T03:14:20.640 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 30/198 2024-03-19T03:14:20.640 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 31/198 2024-03-19T03:14:20.640 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 32/198 2024-03-19T03:14:20.640 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 33/198 2024-03-19T03:14:20.640 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 34/198 2024-03-19T03:14:20.641 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 35/198 2024-03-19T03:14:20.641 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 36/198 2024-03-19T03:14:20.641 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 37/198 2024-03-19T03:14:20.641 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 38/198 2024-03-19T03:14:20.641 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 39/198 2024-03-19T03:14:20.641 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-19T03:14:20.641 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-19T03:14:20.641 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-19T03:14:20.641 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-19T03:14:20.641 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-19T03:14:20.641 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-19T03:14:20.641 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-19T03:14:20.641 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-19T03:14:20.641 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-19T03:14:20.642 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-19T03:14:20.642 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-19T03:14:20.642 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-19T03:14:20.642 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-19T03:14:20.642 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-19T03:14:20.642 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-19T03:14:20.642 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-19T03:14:20.642 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-19T03:14:20.642 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-19T03:14:20.642 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-19T03:14:20.642 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-19T03:14:20.642 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-19T03:14:20.642 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-19T03:14:20.643 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-19T03:14:20.643 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-19T03:14:20.643 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-19T03:14:20.643 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-19T03:14:20.643 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-19T03:14:20.643 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-19T03:14:20.643 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-19T03:14:20.643 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-19T03:14:20.643 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-19T03:14:20.643 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-19T03:14:20.643 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-19T03:14:20.643 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-19T03:14:20.643 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-19T03:14:20.643 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-19T03:14:20.643 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-19T03:14:20.644 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-19T03:14:20.644 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-19T03:14:20.644 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-19T03:14:20.644 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-19T03:14:20.644 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-19T03:14:20.644 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-19T03:14:20.644 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-19T03:14:20.644 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-19T03:14:20.644 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-19T03:14:20.644 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-19T03:14:20.644 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-19T03:14:20.644 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-19T03:14:20.644 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-19T03:14:20.644 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-19T03:14:20.645 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-19T03:14:20.645 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-19T03:14:20.645 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-19T03:14:20.645 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-19T03:14:20.645 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-19T03:14:20.645 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-19T03:14:20.645 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-19T03:14:20.645 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-19T03:14:20.645 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-19T03:14:20.645 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-19T03:14:20.645 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-19T03:14:20.645 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-19T03:14:20.645 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-19T03:14:20.645 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-19T03:14:20.645 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-19T03:14:20.646 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-19T03:14:20.646 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-19T03:14:20.646 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-19T03:14:20.646 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-19T03:14:20.646 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-19T03:14:20.646 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-19T03:14:20.646 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-19T03:14:20.646 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-19T03:14:20.646 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-19T03:14:20.646 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-19T03:14:20.646 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-19T03:14:20.646 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-19T03:14:20.646 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-19T03:14:20.647 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-19T03:14:20.647 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-19T03:14:20.647 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-19T03:14:20.647 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-19T03:14:20.647 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-19T03:14:20.647 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-19T03:14:20.647 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-19T03:14:20.647 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-19T03:14:20.647 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-19T03:14:20.647 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-19T03:14:20.647 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-19T03:14:20.647 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-19T03:14:20.647 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-19T03:14:20.647 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-19T03:14:20.647 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-19T03:14:20.648 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-19T03:14:20.648 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-19T03:14:20.648 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-19T03:14:20.648 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-19T03:14:20.648 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-19T03:14:20.648 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-19T03:14:20.648 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-19T03:14:20.648 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-19T03:14:20.648 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-19T03:14:20.648 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-19T03:14:20.648 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-19T03:14:20.648 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-19T03:14:20.648 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-19T03:14:20.648 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-19T03:14:20.648 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-19T03:14:20.649 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-19T03:14:20.649 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-19T03:14:20.649 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T03:14:20.649 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-19T03:14:20.649 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T03:14:20.649 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-19T03:14:20.649 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-19T03:14:20.649 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-19T03:14:20.649 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-19T03:14:20.649 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-19T03:14:20.649 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-19T03:14:20.649 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-19T03:14:20.649 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-19T03:14:20.649 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-19T03:14:20.649 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-19T03:14:20.650 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-19T03:14:20.650 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-19T03:14:20.650 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-19T03:14:20.650 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-19T03:14:20.650 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-19T03:14:20.650 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-19T03:14:20.650 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-19T03:14:20.650 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-19T03:14:20.650 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-19T03:14:20.650 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-19T03:14:20.650 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-19T03:14:20.650 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-19T03:14:20.650 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-19T03:14:20.650 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-19T03:14:20.650 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-19T03:14:20.651 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-19T03:14:20.651 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-19T03:14:20.651 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-19T03:14:20.651 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-19T03:14:20.651 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-19T03:14:20.651 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-19T03:14:20.651 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-19T03:14:20.651 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-19T03:14:20.651 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-19T03:14:20.651 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-19T03:14:20.651 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-19T03:14:20.651 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-19T03:14:20.651 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-19T03:14:20.651 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-19T03:14:20.651 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-19T03:14:20.651 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-19T03:14:20.652 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-19T03:14:20.652 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-19T03:14:20.652 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-19T03:14:23.019 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-19T03:14:23.019 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:14:23.019 INFO:teuthology.orchestra.run.smithi117.stdout:Upgraded: 2024-03-19T03:14:23.019 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-19T03:14:23.019 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-19T03:14:23.019 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-19T03:14:23.019 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-19T03:14:23.019 INFO:teuthology.orchestra.run.smithi117.stdout: acl-2.3.1-4.el9.x86_64 2024-03-19T03:14:23.019 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-19T03:14:23.019 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-19T03:14:23.020 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-19T03:14:23.020 INFO:teuthology.orchestra.run.smithi117.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-19T03:14:23.020 INFO:teuthology.orchestra.run.smithi117.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-19T03:14:23.020 INFO:teuthology.orchestra.run.smithi117.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-19T03:14:23.020 INFO:teuthology.orchestra.run.smithi117.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-19T03:14:23.020 INFO:teuthology.orchestra.run.smithi117.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-19T03:14:23.020 INFO:teuthology.orchestra.run.smithi117.stdout: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.020 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.020 INFO:teuthology.orchestra.run.smithi117.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-19T03:14:23.020 INFO:teuthology.orchestra.run.smithi117.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-19T03:14:23.020 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-19T03:14:23.020 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-19T03:14:23.020 INFO:teuthology.orchestra.run.smithi117.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-19T03:14:23.021 INFO:teuthology.orchestra.run.smithi117.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-19T03:14:23.021 INFO:teuthology.orchestra.run.smithi117.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-19T03:14:23.021 INFO:teuthology.orchestra.run.smithi117.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-19T03:14:23.021 INFO:teuthology.orchestra.run.smithi117.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-19T03:14:23.021 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-19T03:14:23.021 INFO:teuthology.orchestra.run.smithi117.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-19T03:14:23.021 INFO:teuthology.orchestra.run.smithi117.stdout: automake-1.16.2-8.el9.noarch 2024-03-19T03:14:23.021 INFO:teuthology.orchestra.run.smithi117.stdout: bison-3.7.4-5.el9.x86_64 2024-03-19T03:14:23.021 INFO:teuthology.orchestra.run.smithi117.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-19T03:14:23.021 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.021 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.021 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.022 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.022 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:14:23.022 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.022 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.022 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.022 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:14:23.022 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:14:23.022 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:14:23.022 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:14:23.022 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:14:23.022 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.022 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.022 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:14:23.023 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.023 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.023 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.023 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:14:23.023 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:14:23.023 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.023 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.023 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:14:23.023 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-19T03:14:23.023 INFO:teuthology.orchestra.run.smithi117.stdout: flex-2.6.4-9.el9.x86_64 2024-03-19T03:14:23.023 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-19T03:14:23.023 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-19T03:14:23.023 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-19T03:14:23.024 INFO:teuthology.orchestra.run.smithi117.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-19T03:14:23.024 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-19T03:14:23.024 INFO:teuthology.orchestra.run.smithi117.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-19T03:14:23.024 INFO:teuthology.orchestra.run.smithi117.stdout: jq-1.6-16.el9.x86_64 2024-03-19T03:14:23.024 INFO:teuthology.orchestra.run.smithi117.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-19T03:14:23.024 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-19T03:14:23.024 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-19T03:14:23.024 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.024 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.024 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.024 INFO:teuthology.orchestra.run.smithi117.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-19T03:14:23.024 INFO:teuthology.orchestra.run.smithi117.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-19T03:14:23.024 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-19T03:14:23.025 INFO:teuthology.orchestra.run.smithi117.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-19T03:14:23.025 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-19T03:14:23.025 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.025 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.025 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-19T03:14:23.025 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.025 INFO:teuthology.orchestra.run.smithi117.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-19T03:14:23.025 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-19T03:14:23.025 INFO:teuthology.orchestra.run.smithi117.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-19T03:14:23.025 INFO:teuthology.orchestra.run.smithi117.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-19T03:14:23.025 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-19T03:14:23.025 INFO:teuthology.orchestra.run.smithi117.stdout: lua-5.4.4-4.el9.x86_64 2024-03-19T03:14:23.025 INFO:teuthology.orchestra.run.smithi117.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-19T03:14:23.025 INFO:teuthology.orchestra.run.smithi117.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-19T03:14:23.026 INFO:teuthology.orchestra.run.smithi117.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-19T03:14:23.026 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-19T03:14:23.026 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-19T03:14:23.026 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-19T03:14:23.026 INFO:teuthology.orchestra.run.smithi117.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-19T03:14:23.026 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-13.11-1.el9.x86_64 2024-03-19T03:14:23.026 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-19T03:14:23.026 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-19T03:14:23.026 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-19T03:14:23.026 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-19T03:14:23.026 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-19T03:14:23.026 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-19T03:14:23.026 INFO:teuthology.orchestra.run.smithi117.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-19T03:14:23.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-19T03:14:23.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-19T03:14:23.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-19T03:14:23.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-19T03:14:23.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-19T03:14:23.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-19T03:14:23.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-19T03:14:23.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-19T03:14:23.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-19T03:14:23.027 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-19T03:14:23.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-19T03:14:23.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-19T03:14:23.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-19T03:14:23.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-19T03:14:23.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-19T03:14:23.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-19T03:14:23.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-19T03:14:23.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-19T03:14:23.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-19T03:14:23.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-19T03:14:23.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-19T03:14:23.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-19T03:14:23.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-19T03:14:23.028 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-19T03:14:23.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-19T03:14:23.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-19T03:14:23.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-19T03:14:23.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-19T03:14:23.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-19T03:14:23.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-19T03:14:23.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-19T03:14:23.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-19T03:14:23.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-19T03:14:23.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply-3.11-14.el9.noarch 2024-03-19T03:14:23.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-19T03:14:23.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-19T03:14:23.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-19T03:14:23.029 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-19T03:14:23.030 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-19T03:14:23.030 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-19T03:14:23.030 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-19T03:14:23.030 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-19T03:14:23.030 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-19T03:14:23.030 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.030 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.030 INFO:teuthology.orchestra.run.smithi117.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-19T03:14:23.030 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-19T03:14:23.030 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-19T03:14:23.030 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.030 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-19T03:14:23.031 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-19T03:14:23.031 INFO:teuthology.orchestra.run.smithi117.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-19T03:14:23.031 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-19T03:14:23.031 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-19T03:14:23.031 INFO:teuthology.orchestra.run.smithi117.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-19T03:14:23.031 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-19T03:14:23.031 INFO:teuthology.orchestra.run.smithi117.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-19T03:14:23.031 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-19T03:14:23.031 INFO:teuthology.orchestra.run.smithi117.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-19T03:14:23.031 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-19T03:14:23.031 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-19T03:14:23.031 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-19T03:14:23.031 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-19T03:14:23.031 INFO:teuthology.orchestra.run.smithi117.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-19T03:14:23.032 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.032 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.032 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:14:23.032 INFO:teuthology.orchestra.run.smithi117.stdout: re2-1:20211101-3.el9.x86_64 2024-03-19T03:14:23.032 INFO:teuthology.orchestra.run.smithi117.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-19T03:14:23.032 INFO:teuthology.orchestra.run.smithi117.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-19T03:14:23.032 INFO:teuthology.orchestra.run.smithi117.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-19T03:14:23.032 INFO:teuthology.orchestra.run.smithi117.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-19T03:14:23.032 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-19T03:14:23.032 INFO:teuthology.orchestra.run.smithi117.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-19T03:14:23.032 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:14:23.033 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-19T03:14:23.784 DEBUG:teuthology.parallel:result is None 2024-03-19T03:14:25.622 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - CRB 2.2 MB/s | 6.1 MB 00:02 2024-03-19T03:14:27.548 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - Extras packages 20 kB/s | 16 kB 00:00 2024-03-19T03:14:47.574 INFO:teuthology.orchestra.run.smithi178.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:19 2024-03-19T03:14:52.526 INFO:teuthology.orchestra.run.smithi178.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-19T03:14:54.857 INFO:teuthology.orchestra.run.smithi178.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T03:14:54.858 INFO:teuthology.orchestra.run.smithi178.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T03:14:54.872 INFO:teuthology.orchestra.run.smithi178.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-19T03:14:54.873 INFO:teuthology.orchestra.run.smithi178.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-19T03:14:54.874 INFO:teuthology.orchestra.run.smithi178.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-19T03:14:54.875 INFO:teuthology.orchestra.run.smithi178.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-19T03:14:54.877 INFO:teuthology.orchestra.run.smithi178.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-19T03:14:54.880 INFO:teuthology.orchestra.run.smithi178.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-19T03:14:54.881 INFO:teuthology.orchestra.run.smithi178.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-19T03:14:54.960 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-19T03:14:54.978 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================================ 2024-03-19T03:14:54.978 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-03-19T03:14:54.978 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================================ 2024-03-19T03:14:54.978 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-19T03:14:54.978 INFO:teuthology.orchestra.run.smithi178.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-19T03:14:54.978 INFO:teuthology.orchestra.run.smithi178.stdout: ceph x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 6.6 k 2024-03-19T03:14:54.978 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.4 M 2024-03-19T03:14:54.978 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 891 k 2024-03-19T03:14:54.979 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 145 k 2024-03-19T03:14:54.979 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 1.5 M 2024-03-19T03:14:54.979 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 139 k 2024-03-19T03:14:54.979 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 1.8 M 2024-03-19T03:14:54.979 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 7.4 M 2024-03-19T03:14:54.979 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 49 k 2024-03-19T03:14:54.979 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 9.8 M 2024-03-19T03:14:54.979 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 48 M 2024-03-19T03:14:54.979 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-volume noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 264 k 2024-03-19T03:14:54.979 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 749 k 2024-03-19T03:14:54.979 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 221 k 2024-03-19T03:14:54.979 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-shell x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 27 k 2024-03-19T03:14:54.979 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-top noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 18 k 2024-03-19T03:14:54.980 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-19T03:14:54.980 INFO:teuthology.orchestra.run.smithi178.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-19T03:14:54.980 INFO:teuthology.orchestra.run.smithi178.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-19T03:14:54.980 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 32 k 2024-03-19T03:14:54.980 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 746 k 2024-03-19T03:14:54.980 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 127 k 2024-03-19T03:14:54.980 INFO:teuthology.orchestra.run.smithi178.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-19T03:14:54.980 INFO:teuthology.orchestra.run.smithi178.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-19T03:14:54.980 INFO:teuthology.orchestra.run.smithi178.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-19T03:14:54.980 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-19T03:14:54.980 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-19T03:14:54.980 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-19T03:14:54.981 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 162 k 2024-03-19T03:14:54.981 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 321 k 2024-03-19T03:14:54.981 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 298 k 2024-03-19T03:14:54.981 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 99 k 2024-03-19T03:14:54.981 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 85 k 2024-03-19T03:14:54.981 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T03:14:54.981 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 170 k 2024-03-19T03:14:54.981 INFO:teuthology.orchestra.run.smithi178.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-19T03:14:54.981 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrading: 2024-03-19T03:14:54.981 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-19T03:14:54.981 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-19T03:14:54.981 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-19T03:14:54.981 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-19T03:14:54.982 INFO:teuthology.orchestra.run.smithi178.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-19T03:14:54.982 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-19T03:14:54.982 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-19T03:14:54.982 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-19T03:14:54.982 INFO:teuthology.orchestra.run.smithi178.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-19T03:14:54.982 INFO:teuthology.orchestra.run.smithi178.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-19T03:14:54.982 INFO:teuthology.orchestra.run.smithi178.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-19T03:14:54.982 INFO:teuthology.orchestra.run.smithi178.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-19T03:14:54.982 INFO:teuthology.orchestra.run.smithi178.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-19T03:14:54.982 INFO:teuthology.orchestra.run.smithi178.stdout: librados2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.4 M 2024-03-19T03:14:54.982 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T03:14:54.982 INFO:teuthology.orchestra.run.smithi178.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-19T03:14:54.983 INFO:teuthology.orchestra.run.smithi178.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-19T03:14:54.983 INFO:teuthology.orchestra.run.smithi178.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-19T03:14:54.983 INFO:teuthology.orchestra.run.smithi178.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-19T03:14:54.983 INFO:teuthology.orchestra.run.smithi178.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-19T03:14:54.983 INFO:teuthology.orchestra.run.smithi178.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-19T03:14:54.983 INFO:teuthology.orchestra.run.smithi178.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-19T03:14:54.983 INFO:teuthology.orchestra.run.smithi178.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-19T03:14:54.983 INFO:teuthology.orchestra.run.smithi178.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-19T03:14:54.983 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-03-19T03:14:54.983 INFO:teuthology.orchestra.run.smithi178.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-19T03:14:54.983 INFO:teuthology.orchestra.run.smithi178.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-19T03:14:54.983 INFO:teuthology.orchestra.run.smithi178.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-19T03:14:54.984 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 21 M 2024-03-19T03:14:54.984 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 34 k 2024-03-19T03:14:54.984 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 2.4 M 2024-03-19T03:14:54.984 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 248 k 2024-03-19T03:14:54.984 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 4.7 M 2024-03-19T03:14:54.984 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 17 M 2024-03-19T03:14:54.984 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 16 k 2024-03-19T03:14:54.984 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 25 k 2024-03-19T03:14:54.984 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-19T03:14:54.984 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-19T03:14:54.984 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-19T03:14:54.984 INFO:teuthology.orchestra.run.smithi178.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-19T03:14:54.984 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-19T03:14:54.984 INFO:teuthology.orchestra.run.smithi178.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-19T03:14:54.985 INFO:teuthology.orchestra.run.smithi178.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-19T03:14:54.985 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-19T03:14:54.985 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-19T03:14:54.985 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 163 k 2024-03-19T03:14:54.985 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-19T03:14:54.985 INFO:teuthology.orchestra.run.smithi178.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-19T03:14:54.985 INFO:teuthology.orchestra.run.smithi178.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-19T03:14:54.985 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-19T03:14:54.985 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-19T03:14:54.985 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 501 k 2024-03-19T03:14:54.985 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-19T03:14:54.985 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.1 M 2024-03-19T03:14:54.985 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-19T03:14:54.986 INFO:teuthology.orchestra.run.smithi178.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-19T03:14:54.986 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-19T03:14:54.986 INFO:teuthology.orchestra.run.smithi178.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-19T03:14:54.986 INFO:teuthology.orchestra.run.smithi178.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-19T03:14:54.986 INFO:teuthology.orchestra.run.smithi178.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-19T03:14:54.986 INFO:teuthology.orchestra.run.smithi178.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-19T03:14:54.986 INFO:teuthology.orchestra.run.smithi178.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-19T03:14:54.986 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-19T03:14:54.986 INFO:teuthology.orchestra.run.smithi178.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-19T03:14:54.986 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-19T03:14:54.986 INFO:teuthology.orchestra.run.smithi178.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-19T03:14:54.986 INFO:teuthology.orchestra.run.smithi178.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-19T03:14:54.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-19T03:14:54.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-19T03:14:54.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-19T03:14:54.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-19T03:14:54.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-19T03:14:54.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 45 k 2024-03-19T03:14:54.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 131 k 2024-03-19T03:14:54.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-19T03:14:54.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-19T03:14:54.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-19T03:14:54.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-19T03:14:54.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-19T03:14:54.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-19T03:14:54.987 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-19T03:14:54.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-19T03:14:54.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-19T03:14:54.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-19T03:14:54.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-19T03:14:54.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-19T03:14:54.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-19T03:14:54.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-19T03:14:54.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-19T03:14:54.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-19T03:14:54.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-19T03:14:54.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-19T03:14:54.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-19T03:14:54.988 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-19T03:14:54.989 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-19T03:14:54.989 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-19T03:14:54.989 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-19T03:14:54.989 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-19T03:14:54.989 INFO:teuthology.orchestra.run.smithi178.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-19T03:14:54.989 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-19T03:14:54.989 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-19T03:14:54.989 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-19T03:14:54.989 INFO:teuthology.orchestra.run.smithi178.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-19T03:14:54.989 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-19T03:14:54.989 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-19T03:14:54.989 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-19T03:14:54.989 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-19T03:14:54.990 INFO:teuthology.orchestra.run.smithi178.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-19T03:14:54.990 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-19T03:14:54.990 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-19T03:14:54.990 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-19T03:14:54.990 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-19T03:14:54.990 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-19T03:14:54.990 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-19T03:14:54.990 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-19T03:14:54.990 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-19T03:14:54.990 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-19T03:14:54.990 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-19T03:14:54.990 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-19T03:14:54.990 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-19T03:14:54.991 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-19T03:14:54.991 INFO:teuthology.orchestra.run.smithi178.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-19T03:14:54.991 INFO:teuthology.orchestra.run.smithi178.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-19T03:14:54.991 INFO:teuthology.orchestra.run.smithi178.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-19T03:14:54.991 INFO:teuthology.orchestra.run.smithi178.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-19T03:14:54.991 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-19T03:14:54.991 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-19T03:14:54.991 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-19T03:14:54.991 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-19T03:14:54.991 INFO:teuthology.orchestra.run.smithi178.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-19T03:14:54.991 INFO:teuthology.orchestra.run.smithi178.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-19T03:14:54.991 INFO:teuthology.orchestra.run.smithi178.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-19T03:14:54.991 INFO:teuthology.orchestra.run.smithi178.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-19T03:14:54.991 INFO:teuthology.orchestra.run.smithi178.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-19T03:14:54.991 INFO:teuthology.orchestra.run.smithi178.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-19T03:14:54.992 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-19T03:14:54.992 INFO:teuthology.orchestra.run.smithi178.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-19T03:14:54.992 INFO:teuthology.orchestra.run.smithi178.stdout:Installing weak dependencies: 2024-03-19T03:14:54.992 INFO:teuthology.orchestra.run.smithi178.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-19T03:14:54.992 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-19T03:14:54.992 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-19T03:14:54.992 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================================ 2024-03-19T03:14:54.992 INFO:teuthology.orchestra.run.smithi178.stdout:Install 150 Packages 2024-03-19T03:14:54.992 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrade 24 Packages 2024-03-19T03:14:54.992 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-19T03:14:54.993 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 224 M 2024-03-19T03:14:54.993 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-19T03:14:56.011 INFO:teuthology.orchestra.run.smithi178.stdout:(1/174): ceph-19.0.0-2206.gb6ae2491.el9.x86_64. 37 kB/s | 6.6 kB 00:00 2024-03-19T03:14:56.203 INFO:teuthology.orchestra.run.smithi178.stdout:(2/174): ceph-fuse-19.0.0-2206.gb6ae2491.el9.x8 4.6 MB/s | 891 kB 00:00 2024-03-19T03:14:56.253 INFO:teuthology.orchestra.run.smithi178.stdout:(3/174): ceph-immutable-object-cache-19.0.0-220 2.8 MB/s | 145 kB 00:00 2024-03-19T03:14:56.470 INFO:teuthology.orchestra.run.smithi178.stdout:(4/174): ceph-mds-19.0.0-2206.gb6ae2491.el9.x86 11 MB/s | 2.4 MB 00:00 2024-03-19T03:14:56.637 INFO:teuthology.orchestra.run.smithi178.stdout:(5/174): ceph-mgr-19.0.0-2206.gb6ae2491.el9.x86 9.0 MB/s | 1.5 MB 00:00 2024-03-19T03:14:56.945 INFO:teuthology.orchestra.run.smithi178.stdout:(6/174): ceph-base-19.0.0-2206.gb6ae2491.el9.x8 4.8 MB/s | 5.4 MB 00:01 2024-03-19T03:14:57.029 INFO:teuthology.orchestra.run.smithi178.stdout:(7/174): ceph-mon-19.0.0-2206.gb6ae2491.el9.x86 12 MB/s | 4.7 MB 00:00 2024-03-19T03:14:57.938 INFO:teuthology.orchestra.run.smithi178.stdout:(8/174): ceph-radosgw-19.0.0-2206.gb6ae2491.el9 11 MB/s | 9.8 MB 00:00 2024-03-19T03:14:57.988 INFO:teuthology.orchestra.run.smithi178.stdout:(9/174): ceph-selinux-19.0.0-2206.gb6ae2491.el9 505 kB/s | 25 kB 00:00 2024-03-19T03:14:58.946 INFO:teuthology.orchestra.run.smithi178.stdout:(10/174): ceph-common-19.0.0-2206.gb6ae2491.el9 6.6 MB/s | 21 MB 00:03 2024-03-19T03:14:59.130 INFO:teuthology.orchestra.run.smithi178.stdout:(11/174): ceph-osd-19.0.0-2206.gb6ae2491.el9.x8 7.7 MB/s | 17 MB 00:02 2024-03-19T03:14:59.155 INFO:teuthology.orchestra.run.smithi178.stdout:(12/174): cephfs-mirror-19.0.0-2206.gb6ae2491.e 1.0 MB/s | 221 kB 00:00 2024-03-19T03:14:59.189 INFO:teuthology.orchestra.run.smithi178.stdout:(13/174): cephfs-shell-19.0.0-2206.gb6ae2491.el 463 kB/s | 27 kB 00:00 2024-03-19T03:14:59.214 INFO:teuthology.orchestra.run.smithi178.stdout:(14/174): libcephfs-devel-19.0.0-2206.gb6ae2491 542 kB/s | 32 kB 00:00 2024-03-19T03:14:59.314 INFO:teuthology.orchestra.run.smithi178.stdout:(15/174): libcephsqlite-19.0.0-2206.gb6ae2491.e 1.6 MB/s | 163 kB 00:00 2024-03-19T03:14:59.340 INFO:teuthology.orchestra.run.smithi178.stdout:(16/174): libcephfs2-19.0.0-2206.gb6ae2491.el9. 4.8 MB/s | 746 kB 00:00 2024-03-19T03:14:59.400 INFO:teuthology.orchestra.run.smithi178.stdout:(17/174): librados-devel-19.0.0-2206.gb6ae2491. 1.4 MB/s | 127 kB 00:00 2024-03-19T03:14:59.434 INFO:teuthology.orchestra.run.smithi178.stdout:(18/174): libradosstriper1-19.0.0-2206.gb6ae249 5.2 MB/s | 501 kB 00:00 2024-03-19T03:14:59.518 INFO:teuthology.orchestra.run.smithi178.stdout:(19/174): python3-ceph-argparse-19.0.0-2206.gb6 541 kB/s | 45 kB 00:00 2024-03-19T03:14:59.609 INFO:teuthology.orchestra.run.smithi178.stdout:(20/174): python3-ceph-common-19.0.0-2206.gb6ae 1.4 MB/s | 131 kB 00:00 2024-03-19T03:14:59.751 INFO:teuthology.orchestra.run.smithi178.stdout:(21/174): python3-cephfs-19.0.0-2206.gb6ae2491. 1.1 MB/s | 162 kB 00:00 2024-03-19T03:14:59.852 INFO:teuthology.orchestra.run.smithi178.stdout:(22/174): python3-rados-19.0.0-2206.gb6ae2491.e 3.1 MB/s | 321 kB 00:00 2024-03-19T03:14:59.952 INFO:teuthology.orchestra.run.smithi178.stdout:(23/174): python3-rbd-19.0.0-2206.gb6ae2491.el9 2.9 MB/s | 298 kB 00:00 2024-03-19T03:15:00.022 INFO:teuthology.orchestra.run.smithi178.stdout:(24/174): librgw2-19.0.0-2206.gb6ae2491.el9.x86 8.3 MB/s | 5.1 MB 00:00 2024-03-19T03:15:00.044 INFO:teuthology.orchestra.run.smithi178.stdout:(25/174): python3-rgw-19.0.0-2206.gb6ae2491.el9 1.1 MB/s | 99 kB 00:00 2024-03-19T03:15:00.086 INFO:teuthology.orchestra.run.smithi178.stdout:(26/174): rbd-fuse-19.0.0-2206.gb6ae2491.el9.x8 1.3 MB/s | 85 kB 00:00 2024-03-19T03:15:00.178 INFO:teuthology.orchestra.run.smithi178.stdout:(27/174): rbd-nbd-19.0.0-2206.gb6ae2491.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-19T03:15:00.261 INFO:teuthology.orchestra.run.smithi178.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2206.g 407 kB/s | 34 kB 00:00 2024-03-19T03:15:00.362 INFO:teuthology.orchestra.run.smithi178.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2206.gb6ae249 1.4 MB/s | 139 kB 00:00 2024-03-19T03:15:00.430 INFO:teuthology.orchestra.run.smithi178.stdout:(30/174): rbd-mirror-19.0.0-2206.gb6ae2491.el9. 8.0 MB/s | 3.1 MB 00:00 2024-03-19T03:15:00.612 INFO:teuthology.orchestra.run.smithi178.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2206.gb6ae2 7.2 MB/s | 1.8 MB 00:00 2024-03-19T03:15:00.771 INFO:teuthology.orchestra.run.smithi178.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2206.gb6 1.5 MB/s | 248 kB 00:00 2024-03-19T03:15:00.863 INFO:teuthology.orchestra.run.smithi178.stdout:(33/174): ceph-mgr-rook-19.0.0-2206.gb6ae2491.e 540 kB/s | 49 kB 00:00 2024-03-19T03:15:01.338 INFO:teuthology.orchestra.run.smithi178.stdout:(34/174): ceph-test-19.0.0-2206.gb6ae2491.el9.x 14 MB/s | 48 MB 00:03 2024-03-19T03:15:01.363 INFO:teuthology.orchestra.run.smithi178.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2206.gb 33 kB/s | 16 kB 00:00 2024-03-19T03:15:01.493 INFO:teuthology.orchestra.run.smithi178.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 6.9 MB/s | 7.4 MB 00:01 2024-03-19T03:15:01.522 INFO:teuthology.orchestra.run.smithi178.stdout:(37/174): ceph-volume-19.0.0-2206.gb6ae2491.el9 1.4 MB/s | 264 kB 00:00 2024-03-19T03:15:01.564 INFO:teuthology.orchestra.run.smithi178.stdout:(38/174): cephadm-19.0.0-2206.gb6ae2491.el9.noa 3.6 MB/s | 749 kB 00:00 2024-03-19T03:15:01.615 INFO:teuthology.orchestra.run.smithi178.stdout:(39/174): cephfs-top-19.0.0-2206.gb6ae2491.el9. 151 kB/s | 18 kB 00:00 2024-03-19T03:15:01.732 INFO:teuthology.orchestra.run.smithi178.stdout:(40/174): automake-1.16.2-8.el9.noarch.rpm 3.1 MB/s | 662 kB 00:00 2024-03-19T03:15:01.757 INFO:teuthology.orchestra.run.smithi178.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 918 kB/s | 106 kB 00:00 2024-03-19T03:15:01.790 INFO:teuthology.orchestra.run.smithi178.stdout:(42/174): bison-3.7.4-5.el9.x86_64.rpm 4.5 MB/s | 920 kB 00:00 2024-03-19T03:15:01.815 INFO:teuthology.orchestra.run.smithi178.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 551 kB/s | 32 kB 00:00 2024-03-19T03:15:01.841 INFO:teuthology.orchestra.run.smithi178.stdout:(44/174): flex-2.6.4-9.el9.x86_64.rpm 3.6 MB/s | 307 kB 00:00 2024-03-19T03:15:01.924 INFO:teuthology.orchestra.run.smithi178.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 22 MB/s | 3.0 MB 00:00 2024-03-19T03:15:01.941 INFO:teuthology.orchestra.run.smithi178.stdout:(46/174): flexiblas-openblas-openmp-3.0.4-8.el9 138 kB/s | 17 kB 00:00 2024-03-19T03:15:01.958 INFO:teuthology.orchestra.run.smithi178.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 87 kB/s | 10 kB 00:00 2024-03-19T03:15:01.984 INFO:teuthology.orchestra.run.smithi178.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.6 MB/s | 159 kB 00:00 2024-03-19T03:15:02.000 INFO:teuthology.orchestra.run.smithi178.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-19T03:15:02.026 INFO:teuthology.orchestra.run.smithi178.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-19T03:15:02.059 INFO:teuthology.orchestra.run.smithi178.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 7.6 MB/s | 578 kB 00:00 2024-03-19T03:15:02.076 INFO:teuthology.orchestra.run.smithi178.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 945 kB/s | 46 kB 00:00 2024-03-19T03:15:02.101 INFO:teuthology.orchestra.run.smithi178.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-19T03:15:02.127 INFO:teuthology.orchestra.run.smithi178.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 187 kB 00:00 2024-03-19T03:15:02.143 INFO:teuthology.orchestra.run.smithi178.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 829 kB/s | 34 kB 00:00 2024-03-19T03:15:02.169 INFO:teuthology.orchestra.run.smithi178.stdout:(56/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-03-19T03:15:02.277 INFO:teuthology.orchestra.run.smithi178.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 32 MB/s | 4.7 MB 00:00 2024-03-19T03:15:02.311 INFO:teuthology.orchestra.run.smithi178.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 5.7 MB/s | 811 kB 00:00 2024-03-19T03:15:02.345 INFO:teuthology.orchestra.run.smithi178.stdout:(59/174): postgresql-13.11-1.el9.x86_64.rpm 7.6 MB/s | 1.5 MB 00:00 2024-03-19T03:15:02.370 INFO:teuthology.orchestra.run.smithi178.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.5 MB/s | 134 kB 00:00 2024-03-19T03:15:02.570 INFO:teuthology.orchestra.run.smithi178.stdout:(61/174): postgresql-server-13.11-1.el9.x86_64. 22 MB/s | 5.7 MB 00:00 2024-03-19T03:15:02.595 INFO:teuthology.orchestra.run.smithi178.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 349 kB/s | 78 kB 00:00 2024-03-19T03:15:02.629 INFO:teuthology.orchestra.run.smithi178.stdout:(63/174): protobuf-3.14.0-13.el9.x86_64.rpm 3.9 MB/s | 1.0 MB 00:00 2024-03-19T03:15:02.829 INFO:teuthology.orchestra.run.smithi178.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-19T03:15:02.846 INFO:teuthology.orchestra.run.smithi178.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 99 kB/s | 24 kB 00:00 2024-03-19T03:15:02.879 INFO:teuthology.orchestra.run.smithi178.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 73 kB/s | 18 kB 00:00 2024-03-19T03:15:02.913 INFO:teuthology.orchestra.run.smithi178.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 960 kB/s | 32 kB 00:00 2024-03-19T03:15:02.938 INFO:teuthology.orchestra.run.smithi178.stdout:(68/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-19T03:15:03.072 INFO:teuthology.orchestra.run.smithi178.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2024-03-19T03:15:03.097 INFO:teuthology.orchestra.run.smithi178.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 217 kB 00:00 2024-03-19T03:15:03.122 INFO:teuthology.orchestra.run.smithi178.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 960 kB/s | 174 kB 00:00 2024-03-19T03:15:03.148 INFO:teuthology.orchestra.run.smithi178.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 920 kB/s | 69 kB 00:00 2024-03-19T03:15:03.173 INFO:teuthology.orchestra.run.smithi178.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 825 kB/s | 41 kB 00:00 2024-03-19T03:15:03.198 INFO:teuthology.orchestra.run.smithi178.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-19T03:15:03.223 INFO:teuthology.orchestra.run.smithi178.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-19T03:15:03.248 INFO:teuthology.orchestra.run.smithi178.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 953 kB/s | 47 kB 00:00 2024-03-19T03:15:03.273 INFO:teuthology.orchestra.run.smithi178.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-03-19T03:15:03.299 INFO:teuthology.orchestra.run.smithi178.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 569 kB/s | 43 kB 00:00 2024-03-19T03:15:03.624 INFO:teuthology.orchestra.run.smithi178.stdout:(79/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 127 kB/s | 41 kB 00:00 2024-03-19T03:15:03.649 INFO:teuthology.orchestra.run.smithi178.stdout:(80/174): python3-toml-0.10.2-6.el9.noarch.rpm 111 kB/s | 39 kB 00:00 2024-03-19T03:15:03.800 INFO:teuthology.orchestra.run.smithi178.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-03-19T03:15:03.825 INFO:teuthology.orchestra.run.smithi178.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 300 kB 00:00 2024-03-19T03:15:03.858 INFO:teuthology.orchestra.run.smithi178.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 271 kB/s | 56 kB 00:00 2024-03-19T03:15:03.884 INFO:teuthology.orchestra.run.smithi178.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 754 kB/s | 63 kB 00:00 2024-03-19T03:15:03.911 INFO:teuthology.orchestra.run.smithi178.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 879 kB/s | 44 kB 00:00 2024-03-19T03:15:03.934 INFO:teuthology.orchestra.run.smithi178.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-19T03:15:03.959 INFO:teuthology.orchestra.run.smithi178.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-19T03:15:03.984 INFO:teuthology.orchestra.run.smithi178.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-19T03:15:04.018 INFO:teuthology.orchestra.run.smithi178.stdout:(89/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-19T03:15:04.044 INFO:teuthology.orchestra.run.smithi178.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-03-19T03:15:04.077 INFO:teuthology.orchestra.run.smithi178.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.0 MB/s | 187 kB 00:00 2024-03-19T03:15:04.285 INFO:teuthology.orchestra.run.smithi178.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 938 kB/s | 195 kB 00:00 2024-03-19T03:15:04.311 INFO:teuthology.orchestra.run.smithi178.stdout:(93/174): jq-1.6-16.el9.x86_64.rpm 702 kB/s | 187 kB 00:00 2024-03-19T03:15:04.336 INFO:teuthology.orchestra.run.smithi178.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-19T03:15:04.436 INFO:teuthology.orchestra.run.smithi178.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.0 MB/s | 803 kB 00:00 2024-03-19T03:17:13.234 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] elfutils-libelf-devel-0.190-2.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/elfutils-libelf-devel-0.190-2.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T03:17:13.235 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] libuuid-devel-2.37.4-18.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libuuid-devel-2.37.4-18.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T03:17:13.235 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] openssl-devel-3.0.7-27.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/openssl-devel-3.0.7-27.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T03:17:13.539 INFO:teuthology.orchestra.run.smithi178.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 170 B/s | 21 kB 02:09 2024-03-19T03:17:13.555 INFO:teuthology.orchestra.run.smithi178.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 201 B/s | 25 kB 02:09 2024-03-19T03:17:13.923 INFO:teuthology.orchestra.run.smithi178.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 677 kB/s | 248 kB 00:00 2024-03-19T03:17:13.948 INFO:teuthology.orchestra.run.smithi178.stdout:(99/174): python3-devel-3.9.17-2.el9.x86_64.rpm 627 kB/s | 245 kB 00:00 2024-03-19T03:17:14.056 INFO:teuthology.orchestra.run.smithi178.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 322 kB/s | 35 kB 00:00 2024-03-19T03:17:14.157 INFO:teuthology.orchestra.run.smithi178.stdout:(101/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 444 kB/s | 44 kB 00:00 2024-03-19T03:17:14.190 INFO:teuthology.orchestra.run.smithi178.stdout:(102/174): qatlib-23.11.0-1.el9.x86_64.rpm 810 kB/s | 216 kB 00:00 2024-03-19T03:17:14.265 INFO:teuthology.orchestra.run.smithi178.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 297 kB/s | 22 kB 00:00 2024-03-19T03:17:14.291 INFO:teuthology.orchestra.run.smithi178.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 701 kB/s | 94 kB 00:00 2024-03-19T03:17:14.399 INFO:teuthology.orchestra.run.smithi178.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 6.3 MB/s | 863 kB 00:00 2024-03-19T03:17:15.091 INFO:teuthology.orchestra.run.smithi178.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 447 kB/s | 309 kB 00:00 2024-03-19T03:17:15.150 INFO:teuthology.orchestra.run.smithi178.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 333 kB/s | 19 kB 00:00 2024-03-19T03:17:16.916 INFO:teuthology.orchestra.run.smithi178.stdout:(108/174): openssl-devel-3.0.7-27.el9.x86_64.rp 32 kB/s | 4.1 MB 02:12 2024-03-19T03:17:17.050 INFO:teuthology.orchestra.run.smithi178.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 195 kB/s | 26 kB 00:00 2024-03-19T03:17:17.209 INFO:teuthology.orchestra.run.smithi178.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 307 kB/s | 49 kB 00:00 2024-03-19T03:17:17.359 INFO:teuthology.orchestra.run.smithi178.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 449 kB/s | 67 kB 00:00 2024-03-19T03:17:17.592 INFO:teuthology.orchestra.run.smithi178.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 647 kB/s | 151 kB 00:00 2024-03-19T03:17:18.234 INFO:teuthology.orchestra.run.smithi178.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.3 MB/s | 838 kB 00:00 2024-03-19T03:17:18.826 INFO:teuthology.orchestra.run.smithi178.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 926 kB/s | 548 kB 00:00 2024-03-19T03:17:18.943 INFO:teuthology.orchestra.run.smithi178.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 373 kB/s | 43 kB 00:00 2024-03-19T03:17:19.035 INFO:teuthology.orchestra.run.smithi178.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 351 kB/s | 32 kB 00:00 2024-03-19T03:17:19.094 INFO:teuthology.orchestra.run.smithi178.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 257 kB/s | 15 kB 00:00 2024-03-19T03:17:19.369 INFO:teuthology.orchestra.run.smithi178.stdout:(118/174): python3-cheroot-8.6.0-4.el9.noarch.r 625 kB/s | 172 kB 00:00 2024-03-19T03:17:19.761 INFO:teuthology.orchestra.run.smithi178.stdout:(119/174): python3-cherrypy-18.6.1-2.el9.noarch 916 kB/s | 358 kB 00:00 2024-03-19T03:17:20.061 INFO:teuthology.orchestra.run.smithi178.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.0 MB/s | 303 kB 00:00 2024-03-19T03:17:20.111 INFO:teuthology.orchestra.run.smithi178.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 705 kB/s | 35 kB 00:00 2024-03-19T03:17:20.353 INFO:teuthology.orchestra.run.smithi178.stdout:(122/174): python3-google-auth-2.28.0-1.el9.noa 839 kB/s | 203 kB 00:00 2024-03-19T03:17:21.070 INFO:teuthology.orchestra.run.smithi178.stdout:(123/174): libarrow-9.0.0-10.el9.x86_64.rpm 764 kB/s | 4.4 MB 00:05 2024-03-19T03:17:21.412 INFO:teuthology.orchestra.run.smithi178.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 423 kB/s | 144 kB 00:00 2024-03-19T03:17:21.470 INFO:teuthology.orchestra.run.smithi178.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 183 kB/s | 11 kB 00:00 2024-03-19T03:17:21.554 INFO:teuthology.orchestra.run.smithi178.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 212 kB/s | 18 kB 00:00 2024-03-19T03:17:21.637 INFO:teuthology.orchestra.run.smithi178.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 278 kB/s | 23 kB 00:00 2024-03-19T03:17:21.721 INFO:teuthology.orchestra.run.smithi178.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 233 kB/s | 19 kB 00:00 2024-03-19T03:17:21.813 INFO:teuthology.orchestra.run.smithi178.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 215 kB/s | 20 kB 00:00 2024-03-19T03:17:23.146 INFO:teuthology.orchestra.run.smithi178.stdout:(130/174): python3-grpcio-1.46.7-10.el9.x86_64. 747 kB/s | 2.0 MB 00:02 2024-03-19T03:17:23.321 INFO:teuthology.orchestra.run.smithi178.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 265 kB/s | 46 kB 00:00 2024-03-19T03:17:23.405 INFO:teuthology.orchestra.run.smithi178.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 657 kB/s | 1.0 MB 00:01 2024-03-19T03:17:23.513 INFO:teuthology.orchestra.run.smithi178.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 533 kB/s | 58 kB 00:00 2024-03-19T03:17:23.564 INFO:teuthology.orchestra.run.smithi178.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 326 kB/s | 79 kB 00:00 2024-03-19T03:17:23.647 INFO:teuthology.orchestra.run.smithi178.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 197 kB/s | 16 kB 00:00 2024-03-19T03:17:23.847 INFO:teuthology.orchestra.run.smithi178.stdout:(136/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 450 kB/s | 90 kB 00:00 2024-03-19T03:17:23.873 INFO:teuthology.orchestra.run.smithi178.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 758 kB/s | 272 kB 00:00 2024-03-19T03:17:23.906 INFO:teuthology.orchestra.run.smithi178.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 482 kB/s | 28 kB 00:00 2024-03-19T03:17:23.957 INFO:teuthology.orchestra.run.smithi178.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 368 kB/s | 31 kB 00:00 2024-03-19T03:17:24.082 INFO:teuthology.orchestra.run.smithi178.stdout:(140/174): python3-rsa-4.9-2.el9.noarch.rpm 473 kB/s | 59 kB 00:00 2024-03-19T03:17:24.166 INFO:teuthology.orchestra.run.smithi178.stdout:(141/174): python3-tempora-5.0.0-2.el9.noarch.r 428 kB/s | 36 kB 00:00 2024-03-19T03:17:24.283 INFO:teuthology.orchestra.run.smithi178.stdout:(142/174): python3-typing-extensions-4.4.0-2.el 437 kB/s | 51 kB 00:00 2024-03-19T03:17:24.541 INFO:teuthology.orchestra.run.smithi178.stdout:(143/174): python3-routes-2.5.1-5.el9.noarch.rp 297 kB/s | 188 kB 00:00 2024-03-19T03:17:24.591 INFO:teuthology.orchestra.run.smithi178.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 745 kB/s | 230 kB 00:00 2024-03-19T03:17:24.925 INFO:teuthology.orchestra.run.smithi178.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 234 kB/s | 90 kB 00:00 2024-03-19T03:17:25.017 INFO:teuthology.orchestra.run.smithi178.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 1.0 MB/s | 427 kB 00:00 2024-03-19T03:17:25.042 INFO:teuthology.orchestra.run.smithi178.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 171 kB/s | 20 kB 00:00 2024-03-19T03:17:25.409 INFO:teuthology.orchestra.run.smithi178.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 489 kB/s | 192 kB 00:00 2024-03-19T03:17:25.959 INFO:teuthology.orchestra.run.smithi178.stdout:(149/174): librados2-19.0.0-2206.gb6ae2491.el9. 6.2 MB/s | 3.4 MB 00:00 2024-03-19T03:17:26.151 INFO:teuthology.orchestra.run.smithi178.stdout:(150/174): librbd1-19.0.0-2206.gb6ae2491.el9.x8 16 MB/s | 3.1 MB 00:00 2024-03-19T03:17:26.285 INFO:teuthology.orchestra.run.smithi178.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.4 MB/s | 321 kB 00:00 2024-03-19T03:17:26.910 INFO:teuthology.orchestra.run.smithi178.stdout:(152/174): NetworkManager-1.47.2-1.el9.x86_64.r 3.7 MB/s | 2.3 MB 00:00 2024-03-19T03:17:26.977 INFO:teuthology.orchestra.run.smithi178.stdout:(153/174): thrift-0.15.0-2.el9.x86_64.rpm 837 kB/s | 1.6 MB 00:01 2024-03-19T03:17:27.085 INFO:teuthology.orchestra.run.smithi178.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 388 kB/s | 42 kB 00:00 2024-03-19T03:17:27.194 INFO:teuthology.orchestra.run.smithi178.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.2 MB/s | 248 kB 00:00 2024-03-19T03:17:27.252 INFO:teuthology.orchestra.run.smithi178.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-03-19T03:17:27.311 INFO:teuthology.orchestra.run.smithi178.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 624 kB/s | 36 kB 00:00 2024-03-19T03:17:27.353 INFO:teuthology.orchestra.run.smithi178.stdout:(158/174): NetworkManager-libnm-1.47.2-1.el9.x8 4.1 MB/s | 1.8 MB 00:00 2024-03-19T03:17:27.395 INFO:teuthology.orchestra.run.smithi178.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.3 MB/s | 192 kB 00:00 2024-03-19T03:17:27.445 INFO:teuthology.orchestra.run.smithi178.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.7 MB/s | 256 kB 00:00 2024-03-19T03:17:27.463 INFO:teuthology.orchestra.run.smithi178.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 344 kB/s | 23 kB 00:00 2024-03-19T03:17:27.496 INFO:teuthology.orchestra.run.smithi178.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 108 kB 00:00 2024-03-19T03:17:27.546 INFO:teuthology.orchestra.run.smithi178.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 155 kB 00:00 2024-03-19T03:17:27.571 INFO:teuthology.orchestra.run.smithi178.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 136 kB 00:00 2024-03-19T03:17:27.596 INFO:teuthology.orchestra.run.smithi178.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-03-19T03:17:27.622 INFO:teuthology.orchestra.run.smithi178.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 571 kB/s | 29 kB 00:00 2024-03-19T03:17:27.889 INFO:teuthology.orchestra.run.smithi178.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 4.1 MB/s | 1.2 MB 00:00 2024-03-19T03:17:28.055 INFO:teuthology.orchestra.run.smithi178.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 5.0 MB/s | 2.2 MB 00:00 2024-03-19T03:17:28.164 INFO:teuthology.orchestra.run.smithi178.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 4.2 MB/s | 465 kB 00:00 2024-03-19T03:17:28.214 INFO:teuthology.orchestra.run.smithi178.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 624 kB/s | 31 kB 00:00 2024-03-19T03:17:28.415 INFO:teuthology.orchestra.run.smithi178.stdout:(171/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 5.4 MB/s | 1.1 MB 00:00 2024-03-19T03:17:28.457 INFO:teuthology.orchestra.run.smithi178.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.0 MB/s | 2.3 MB 00:00 2024-03-19T03:17:28.690 INFO:teuthology.orchestra.run.smithi178.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 270 kB/s | 74 kB 00:00 2024-03-19T03:19:24.269 INFO:teuthology.orchestra.run.smithi178.stdout:[MIRROR] abseil-cpp-20211102.0-3.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://epel.stl.us.ssimn.org/9/Everything/x86_64/Packages/a/abseil-cpp-20211102.0-3.el9.x86_64.rpm [Failed to connect to epel.stl.us.ssimn.org port 80: Connection timed out] 2024-03-19T03:19:29.344 INFO:teuthology.orchestra.run.smithi178.stdout:(174/174): abseil-cpp-20211102.0-3.el9.x86_64.r 4.1 kB/s | 548 kB 02:15 2024-03-19T03:19:29.347 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-19T03:19:29.347 INFO:teuthology.orchestra.run.smithi178.stdout:Total 838 kB/s | 224 MB 04:34 2024-03-19T03:19:29.797 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-19T03:19:30.064 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-19T03:19:30.064 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-19T03:19:31.846 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-19T03:19:31.846 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-19T03:19:38.695 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-19T03:19:38.822 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-19T03:19:38.848 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-19T03:19:38.879 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T03:19:38.887 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T03:19:38.928 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-19T03:19:38.981 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-19T03:19:39.015 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-19T03:19:39.049 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-19T03:19:39.090 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-19T03:19:39.410 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-19T03:19:39.483 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T03:19:39.604 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T03:19:39.634 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T03:19:39.709 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T03:19:39.738 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T03:19:39.773 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-19T03:19:39.951 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-19T03:19:40.011 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-19T03:19:40.039 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T03:19:40.143 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T03:19:40.173 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T03:19:40.220 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T03:19:40.261 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-19T03:19:40.284 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T03:19:40.299 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T03:19:40.483 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T03:19:40.514 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T03:19:40.546 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-19T03:19:40.641 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-19T03:19:40.740 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T03:19:40.783 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T03:19:40.861 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T03:19:40.935 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T03:19:41.147 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T03:19:41.264 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-19T03:19:41.360 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-19T03:19:41.389 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-19T03:19:41.416 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-19T03:19:41.451 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-19T03:19:41.534 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-19T03:19:41.590 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-19T03:19:41.626 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-19T03:19:41.690 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-19T03:19:41.734 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-19T03:19:41.777 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-19T03:19:41.810 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-19T03:19:41.838 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-19T03:19:41.908 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-19T03:19:41.943 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 39/198 2024-03-19T03:19:41.986 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 40/198 2024-03-19T03:19:42.015 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 41/198 2024-03-19T03:19:42.084 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-19T03:19:42.130 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-19T03:19:42.203 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-19T03:19:42.338 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-19T03:19:42.399 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-19T03:19:42.924 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-19T03:19:43.003 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-19T03:19:43.072 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-19T03:19:43.114 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T03:19:43.121 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T03:19:43.154 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-19T03:19:43.211 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-19T03:19:43.249 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 53/198 2024-03-19T03:19:43.278 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-19T03:19:43.495 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-19T03:19:43.648 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-19T03:19:43.714 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-19T03:19:43.758 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-19T03:19:43.883 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-19T03:19:43.936 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-19T03:19:44.015 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-19T03:19:44.080 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-19T03:19:44.120 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-19T03:19:44.157 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-19T03:19:44.203 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-19T03:19:44.239 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-19T03:19:44.282 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-19T03:19:44.324 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-19T03:19:44.458 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-19T03:19:44.505 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-19T03:19:44.535 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-19T03:19:44.577 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-19T03:19:44.671 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-19T03:19:44.710 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-19T03:19:44.750 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-19T03:19:44.776 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-19T03:19:44.800 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-19T03:19:44.874 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-19T03:19:45.001 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-19T03:19:45.116 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-19T03:19:45.158 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-19T03:19:45.206 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-19T03:19:45.262 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-19T03:19:45.444 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-19T03:19:45.496 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-19T03:19:45.622 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-19T03:19:45.661 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-19T03:19:45.693 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-19T03:19:45.724 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-19T03:19:45.752 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-19T03:19:45.786 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-19T03:19:45.825 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-19T03:19:45.857 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-19T03:19:45.887 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-19T03:19:45.980 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-19T03:19:46.014 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-19T03:19:46.047 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-19T03:19:46.088 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-19T03:19:46.123 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-19T03:19:46.212 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-19T03:19:46.293 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-19T03:19:46.337 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-19T03:19:46.369 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-19T03:19:46.416 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-19T03:19:46.512 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-19T03:19:46.562 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-19T03:19:47.277 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-19T03:19:47.326 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-19T03:19:47.359 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-19T03:19:47.407 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-19T03:19:47.487 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-19T03:19:47.519 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-19T03:19:47.694 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-19T03:19:47.725 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-19T03:19:47.801 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-19T03:19:48.460 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-19T03:19:48.581 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-19T03:19:49.790 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-19T03:19:49.875 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-19T03:19:49.914 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-19T03:19:49.945 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-19T03:19:49.972 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-19T03:19:50.284 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-19T03:19:50.311 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T03:19:50.394 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T03:19:50.424 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 125/198 2024-03-19T03:19:50.460 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-19T03:19:50.922 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-19T03:19:50.948 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T03:19:51.019 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T03:19:51.041 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 129/198 2024-03-19T03:19:53.213 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T03:19:53.254 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T03:19:53.790 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T03:19:53.966 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T03:19:54.047 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T03:19:54.047 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-19T03:19:54.047 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-19T03:19:54.047 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-19T03:19:54.075 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T03:20:05.792 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T03:20:05.793 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /sys 2024-03-19T03:20:05.793 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /proc 2024-03-19T03:20:05.793 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /mnt 2024-03-19T03:20:05.793 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /var/tmp 2024-03-19T03:20:05.793 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /home 2024-03-19T03:20:05.793 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /root 2024-03-19T03:20:05.793 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /tmp 2024-03-19T03:20:05.793 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-19T03:20:05.963 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T03:20:05.978 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T03:20:05.978 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:20:05.978 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-19T03:20:05.978 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-19T03:20:07.476 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T03:20:07.500 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T03:20:07.501 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:20:07.501 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-19T03:20:07.501 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T03:20:07.501 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T03:20:07.501 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-19T03:20:07.726 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T03:20:07.748 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T03:20:07.748 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:20:07.748 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-19T03:20:07.748 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T03:20:07.748 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T03:20:07.748 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-19T03:20:08.185 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T03:20:08.207 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T03:20:08.207 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:20:08.207 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-19T03:20:08.207 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T03:20:08.207 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T03:20:08.207 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-19T03:20:08.295 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-19T03:20:08.688 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T03:20:08.718 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T03:20:08.751 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 139/198 2024-03-19T03:20:08.811 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 140/198 2024-03-19T03:20:08.842 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T03:20:09.075 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T03:20:09.109 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T03:20:10.070 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T03:20:10.101 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T03:20:10.199 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T03:20:10.354 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 144/198 2024-03-19T03:20:10.395 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T03:20:10.417 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T03:20:10.417 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:20:10.418 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-19T03:20:10.418 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T03:20:10.418 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T03:20:10.418 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-19T03:20:10.489 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T03:20:10.499 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T03:20:10.551 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 147/198 2024-03-19T03:20:11.425 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-19T03:20:11.456 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T03:20:11.478 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T03:20:11.478 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:20:11.478 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-19T03:20:11.478 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T03:20:11.478 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T03:20:11.478 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-19T03:20:11.523 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T03:20:11.544 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T03:20:11.545 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:20:11.545 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-19T03:20:11.545 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-19T03:20:11.599 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T03:20:11.621 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T03:20:11.621 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:20:11.621 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-19T03:20:11.621 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T03:20:11.621 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T03:20:11.621 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-19T03:20:11.941 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T03:20:11.963 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T03:20:11.963 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T03:20:11.963 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-19T03:20:11.963 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T03:20:11.963 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T03:20:11.964 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-19T03:20:16.386 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T03:20:16.427 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 154/198 2024-03-19T03:20:16.458 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 155/198 2024-03-19T03:20:16.485 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-19T03:20:16.663 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 157/198 2024-03-19T03:20:16.776 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-19T03:20:16.824 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 159/198 2024-03-19T03:20:16.864 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-19T03:20:16.925 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-19T03:20:16.968 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-19T03:20:16.992 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 163/198 2024-03-19T03:20:17.024 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-19T03:20:17.054 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-19T03:20:17.100 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-19T03:20:17.161 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-19T03:20:17.186 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-19T03:20:17.757 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T03:20:17.800 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T03:20:17.820 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T03:20:18.637 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 170/198 2024-03-19T03:20:18.720 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-19T03:20:18.761 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-19T03:20:18.850 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-19T03:20:18.899 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-19T03:20:18.924 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-19T03:20:18.949 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-19T03:20:18.976 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-19T03:20:18.976 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T03:20:19.047 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T03:20:19.076 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-19T03:20:19.076 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T03:20:19.152 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T03:20:19.179 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-19T03:20:19.212 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-19T03:20:19.237 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-19T03:20:19.262 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-19T03:20:19.295 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-19T03:20:19.335 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-19T03:20:19.367 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-19T03:20:19.392 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-19T03:20:19.418 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-19T03:20:19.443 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-19T03:20:19.468 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-19T03:20:19.474 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T03:20:19.474 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T03:20:19.663 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T03:20:19.663 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T03:20:19.745 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T03:20:19.770 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-19T03:20:19.810 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-19T03:20:19.838 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-19T03:20:19.863 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-19T03:20:19.927 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T03:20:24.739 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T03:20:24.739 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 1/198 2024-03-19T03:20:24.739 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2/198 2024-03-19T03:20:24.739 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 3/198 2024-03-19T03:20:24.739 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 4/198 2024-03-19T03:20:24.739 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 5/198 2024-03-19T03:20:24.740 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 6/198 2024-03-19T03:20:24.740 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 7/198 2024-03-19T03:20:24.740 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 8/198 2024-03-19T03:20:24.740 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 9/198 2024-03-19T03:20:24.740 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T03:20:24.740 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T03:20:24.740 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T03:20:24.740 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 13/198 2024-03-19T03:20:24.740 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 14/198 2024-03-19T03:20:24.740 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 15/198 2024-03-19T03:20:24.740 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T03:20:24.740 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 17/198 2024-03-19T03:20:24.741 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 18/198 2024-03-19T03:20:24.741 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 19/198 2024-03-19T03:20:24.741 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 20/198 2024-03-19T03:20:24.741 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 21/198 2024-03-19T03:20:24.741 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 22/198 2024-03-19T03:20:24.741 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 23/198 2024-03-19T03:20:24.741 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 24/198 2024-03-19T03:20:24.741 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 25/198 2024-03-19T03:20:24.741 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 26/198 2024-03-19T03:20:24.741 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 27/198 2024-03-19T03:20:24.741 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 28/198 2024-03-19T03:20:24.742 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 29/198 2024-03-19T03:20:24.742 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 30/198 2024-03-19T03:20:24.742 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 31/198 2024-03-19T03:20:24.742 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 32/198 2024-03-19T03:20:24.742 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 33/198 2024-03-19T03:20:24.742 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 34/198 2024-03-19T03:20:24.742 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 35/198 2024-03-19T03:20:24.742 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 36/198 2024-03-19T03:20:24.742 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 37/198 2024-03-19T03:20:24.742 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 38/198 2024-03-19T03:20:24.742 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 39/198 2024-03-19T03:20:24.742 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-19T03:20:24.742 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-19T03:20:24.743 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-19T03:20:24.743 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-19T03:20:24.743 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-19T03:20:24.743 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-19T03:20:24.743 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-19T03:20:24.743 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-19T03:20:24.743 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-19T03:20:24.743 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-19T03:20:24.743 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-19T03:20:24.743 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-19T03:20:24.743 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-19T03:20:24.743 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-19T03:20:24.743 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-19T03:20:24.743 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-19T03:20:24.744 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-19T03:20:24.744 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-19T03:20:24.744 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-19T03:20:24.744 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-19T03:20:24.744 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-19T03:20:24.744 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-19T03:20:24.744 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-19T03:20:24.744 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-19T03:20:24.744 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-19T03:20:24.744 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-19T03:20:24.744 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-19T03:20:24.744 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-19T03:20:24.745 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-19T03:20:24.745 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-19T03:20:24.745 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-19T03:20:24.745 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-19T03:20:24.745 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-19T03:20:24.745 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-19T03:20:24.745 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-19T03:20:24.745 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-19T03:20:24.745 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-19T03:20:24.745 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-19T03:20:24.745 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-19T03:20:24.745 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-19T03:20:24.745 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-19T03:20:24.746 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-19T03:20:24.746 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-19T03:20:24.746 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-19T03:20:24.746 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-19T03:20:24.746 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-19T03:20:24.746 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-19T03:20:24.746 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-19T03:20:24.746 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-19T03:20:24.746 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-19T03:20:24.746 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-19T03:20:24.746 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-19T03:20:24.746 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-19T03:20:24.746 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-19T03:20:24.747 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-19T03:20:24.747 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-19T03:20:24.747 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-19T03:20:24.747 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-19T03:20:24.747 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-19T03:20:24.747 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-19T03:20:24.747 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-19T03:20:24.747 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-19T03:20:24.747 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-19T03:20:24.747 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-19T03:20:24.747 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-19T03:20:24.747 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-19T03:20:24.747 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-19T03:20:24.748 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-19T03:20:24.748 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-19T03:20:24.748 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-19T03:20:24.748 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-19T03:20:24.748 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-19T03:20:24.748 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-19T03:20:24.748 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-19T03:20:24.748 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-19T03:20:24.748 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-19T03:20:24.748 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-19T03:20:24.748 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-19T03:20:24.748 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-19T03:20:24.748 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-19T03:20:24.748 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-19T03:20:24.749 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-19T03:20:24.749 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-19T03:20:24.749 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-19T03:20:24.749 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-19T03:20:24.749 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-19T03:20:24.749 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-19T03:20:24.749 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-19T03:20:24.749 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-19T03:20:24.749 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-19T03:20:24.749 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-19T03:20:24.749 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-19T03:20:24.749 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-19T03:20:24.749 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-19T03:20:24.750 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-19T03:20:24.750 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-19T03:20:24.750 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-19T03:20:24.750 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-19T03:20:24.750 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-19T03:20:24.750 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-19T03:20:24.750 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-19T03:20:24.750 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-19T03:20:24.750 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-19T03:20:24.750 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-19T03:20:24.750 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-19T03:20:24.750 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-19T03:20:24.750 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-19T03:20:24.750 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-19T03:20:24.751 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-19T03:20:24.751 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-19T03:20:24.751 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-19T03:20:24.751 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T03:20:24.751 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-19T03:20:24.751 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T03:20:24.751 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-19T03:20:24.751 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-19T03:20:24.751 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-19T03:20:24.751 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-19T03:20:24.751 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-19T03:20:24.751 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-19T03:20:24.752 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-19T03:20:24.752 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-19T03:20:24.752 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-19T03:20:24.752 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-19T03:20:24.752 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-19T03:20:24.752 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-19T03:20:24.752 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-19T03:20:24.752 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-19T03:20:24.752 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-19T03:20:24.752 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-19T03:20:24.752 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-19T03:20:24.752 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-19T03:20:24.752 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-19T03:20:24.753 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-19T03:20:24.753 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-19T03:20:24.753 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-19T03:20:24.753 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-19T03:20:24.753 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-19T03:20:24.753 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-19T03:20:24.753 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-19T03:20:24.753 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-19T03:20:24.753 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-19T03:20:24.753 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-19T03:20:24.753 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-19T03:20:24.753 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-19T03:20:24.753 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-19T03:20:24.754 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-19T03:20:24.754 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-19T03:20:24.754 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-19T03:20:24.754 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-19T03:20:24.754 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-19T03:20:24.754 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-19T03:20:24.754 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-19T03:20:24.754 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-19T03:20:24.754 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-19T03:20:24.754 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-19T03:20:24.754 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-19T03:20:24.754 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-19T03:20:27.540 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-19T03:20:27.540 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-19T03:20:27.540 INFO:teuthology.orchestra.run.smithi178.stdout:Upgraded: 2024-03-19T03:20:27.540 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-19T03:20:27.540 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-19T03:20:27.540 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-19T03:20:27.540 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-19T03:20:27.540 INFO:teuthology.orchestra.run.smithi178.stdout: acl-2.3.1-4.el9.x86_64 2024-03-19T03:20:27.540 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-19T03:20:27.541 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-19T03:20:27.541 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-19T03:20:27.541 INFO:teuthology.orchestra.run.smithi178.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-19T03:20:27.541 INFO:teuthology.orchestra.run.smithi178.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-19T03:20:27.541 INFO:teuthology.orchestra.run.smithi178.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-19T03:20:27.541 INFO:teuthology.orchestra.run.smithi178.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-19T03:20:27.541 INFO:teuthology.orchestra.run.smithi178.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-19T03:20:27.541 INFO:teuthology.orchestra.run.smithi178.stdout: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.541 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.541 INFO:teuthology.orchestra.run.smithi178.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-19T03:20:27.541 INFO:teuthology.orchestra.run.smithi178.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-19T03:20:27.541 INFO:teuthology.orchestra.run.smithi178.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-19T03:20:27.541 INFO:teuthology.orchestra.run.smithi178.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-19T03:20:27.542 INFO:teuthology.orchestra.run.smithi178.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-19T03:20:27.542 INFO:teuthology.orchestra.run.smithi178.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-19T03:20:27.542 INFO:teuthology.orchestra.run.smithi178.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-19T03:20:27.542 INFO:teuthology.orchestra.run.smithi178.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-19T03:20:27.542 INFO:teuthology.orchestra.run.smithi178.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-19T03:20:27.542 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-19T03:20:27.542 INFO:teuthology.orchestra.run.smithi178.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-19T03:20:27.542 INFO:teuthology.orchestra.run.smithi178.stdout: automake-1.16.2-8.el9.noarch 2024-03-19T03:20:27.542 INFO:teuthology.orchestra.run.smithi178.stdout: bison-3.7.4-5.el9.x86_64 2024-03-19T03:20:27.542 INFO:teuthology.orchestra.run.smithi178.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-19T03:20:27.542 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.542 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.542 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.543 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.543 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:20:27.543 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.543 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.543 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.543 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:20:27.543 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:20:27.543 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:20:27.543 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:20:27.543 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:20:27.543 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.543 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.543 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:20:27.543 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.544 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.544 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.544 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:20:27.544 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:20:27.544 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.544 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.544 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T03:20:27.544 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-19T03:20:27.544 INFO:teuthology.orchestra.run.smithi178.stdout: flex-2.6.4-9.el9.x86_64 2024-03-19T03:20:27.544 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-19T03:20:27.544 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-19T03:20:27.544 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-19T03:20:27.544 INFO:teuthology.orchestra.run.smithi178.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-19T03:20:27.545 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-19T03:20:27.545 INFO:teuthology.orchestra.run.smithi178.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-19T03:20:27.545 INFO:teuthology.orchestra.run.smithi178.stdout: jq-1.6-16.el9.x86_64 2024-03-19T03:20:27.545 INFO:teuthology.orchestra.run.smithi178.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-19T03:20:27.545 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-19T03:20:27.545 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-19T03:20:27.545 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.545 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.545 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.545 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-19T03:20:27.545 INFO:teuthology.orchestra.run.smithi178.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-19T03:20:27.545 INFO:teuthology.orchestra.run.smithi178.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-19T03:20:27.545 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-19T03:20:27.545 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-19T03:20:27.546 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.546 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.546 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-19T03:20:27.546 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.546 INFO:teuthology.orchestra.run.smithi178.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-19T03:20:27.546 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-19T03:20:27.546 INFO:teuthology.orchestra.run.smithi178.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-19T03:20:27.546 INFO:teuthology.orchestra.run.smithi178.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-19T03:20:27.546 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-19T03:20:27.546 INFO:teuthology.orchestra.run.smithi178.stdout: lua-5.4.4-4.el9.x86_64 2024-03-19T03:20:27.546 INFO:teuthology.orchestra.run.smithi178.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-19T03:20:27.546 INFO:teuthology.orchestra.run.smithi178.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-19T03:20:27.546 INFO:teuthology.orchestra.run.smithi178.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-19T03:20:27.547 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-19T03:20:27.547 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-19T03:20:27.547 INFO:teuthology.orchestra.run.smithi178.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-19T03:20:27.547 INFO:teuthology.orchestra.run.smithi178.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-19T03:20:27.547 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-13.11-1.el9.x86_64 2024-03-19T03:20:27.547 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-19T03:20:27.547 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-19T03:20:27.547 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-19T03:20:27.547 INFO:teuthology.orchestra.run.smithi178.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-19T03:20:27.547 INFO:teuthology.orchestra.run.smithi178.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-19T03:20:27.547 INFO:teuthology.orchestra.run.smithi178.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-19T03:20:27.547 INFO:teuthology.orchestra.run.smithi178.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-19T03:20:27.547 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-19T03:20:27.547 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-19T03:20:27.548 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-19T03:20:27.548 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.548 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.548 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.548 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-19T03:20:27.548 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-19T03:20:27.548 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-19T03:20:27.548 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-19T03:20:27.548 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-19T03:20:27.548 INFO:teuthology.orchestra.run.smithi178.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-19T03:20:27.548 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-19T03:20:27.548 INFO:teuthology.orchestra.run.smithi178.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-19T03:20:27.548 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-19T03:20:27.548 INFO:teuthology.orchestra.run.smithi178.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-19T03:20:27.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-19T03:20:27.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-19T03:20:27.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-19T03:20:27.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-19T03:20:27.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-19T03:20:27.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-19T03:20:27.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-19T03:20:27.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-19T03:20:27.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-19T03:20:27.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-19T03:20:27.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-19T03:20:27.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-19T03:20:27.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-19T03:20:27.549 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-19T03:20:27.550 INFO:teuthology.orchestra.run.smithi178.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-19T03:20:27.550 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-19T03:20:27.550 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-19T03:20:27.550 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-19T03:20:27.550 INFO:teuthology.orchestra.run.smithi178.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-19T03:20:27.550 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-19T03:20:27.550 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ply-3.11-14.el9.noarch 2024-03-19T03:20:27.550 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-19T03:20:27.550 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-19T03:20:27.550 INFO:teuthology.orchestra.run.smithi178.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-19T03:20:27.550 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-19T03:20:27.550 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-19T03:20:27.550 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-19T03:20:27.550 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-19T03:20:27.551 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-19T03:20:27.551 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-19T03:20:27.551 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.551 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.551 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-19T03:20:27.551 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-19T03:20:27.551 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-19T03:20:27.551 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.551 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-19T03:20:27.551 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-19T03:20:27.551 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-19T03:20:27.551 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-19T03:20:27.551 INFO:teuthology.orchestra.run.smithi178.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-19T03:20:27.552 INFO:teuthology.orchestra.run.smithi178.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-19T03:20:27.552 INFO:teuthology.orchestra.run.smithi178.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-19T03:20:27.552 INFO:teuthology.orchestra.run.smithi178.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-19T03:20:27.552 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-19T03:20:27.552 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-19T03:20:27.552 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-19T03:20:27.552 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-19T03:20:27.552 INFO:teuthology.orchestra.run.smithi178.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-19T03:20:27.552 INFO:teuthology.orchestra.run.smithi178.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-19T03:20:27.552 INFO:teuthology.orchestra.run.smithi178.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-19T03:20:27.552 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.552 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.552 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T03:20:27.552 INFO:teuthology.orchestra.run.smithi178.stdout: re2-1:20211101-3.el9.x86_64 2024-03-19T03:20:27.553 INFO:teuthology.orchestra.run.smithi178.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-19T03:20:27.553 INFO:teuthology.orchestra.run.smithi178.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-19T03:20:27.553 INFO:teuthology.orchestra.run.smithi178.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-19T03:20:27.553 INFO:teuthology.orchestra.run.smithi178.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-19T03:20:27.553 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-19T03:20:27.553 INFO:teuthology.orchestra.run.smithi178.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-19T03:20:27.553 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-19T03:20:27.553 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-19T03:20:28.060 DEBUG:teuthology.parallel:result is None 2024-03-19T03:20:28.060 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T03:20:28.278 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-19T03:20:28.351 INFO:teuthology.orchestra.run.smithi117.stdout:19.0.0-2206.gb6ae2491.el9 2024-03-19T03:20:28.352 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2206.gb6ae2491.el9 2024-03-19T03:20:28.352 INFO:teuthology.task.install:The correct ceph version 19.0.0-2206.gb6ae2491 is installed. 2024-03-19T03:20:28.354 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T03:20:28.493 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-19T03:20:28.519 INFO:teuthology.orchestra.run.smithi119.stdout:19.0.0-2206.gb6ae2491.el9 2024-03-19T03:20:28.519 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2206.gb6ae2491.el9 2024-03-19T03:20:28.519 INFO:teuthology.task.install:The correct ceph version 19.0.0-2206.gb6ae2491 is installed. 2024-03-19T03:20:28.521 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T03:20:28.742 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-19T03:20:28.767 INFO:teuthology.orchestra.run.smithi178.stdout:19.0.0-2206.gb6ae2491.el9 2024-03-19T03:20:28.767 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2206.gb6ae2491.el9 2024-03-19T03:20:28.767 INFO:teuthology.task.install:The correct ceph version 19.0.0-2206.gb6ae2491 is installed. 2024-03-19T03:20:28.769 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-19T03:20:28.769 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-19T03:20:28.769 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-19T03:20:28.894 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-19T03:20:28.894 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-19T03:20:28.930 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-19T03:20:28.930 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-19T03:20:28.965 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-19T03:20:28.994 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-19T03:20:28.994 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/daemon-helper 2024-03-19T03:20:29.090 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-19T03:20:29.229 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-19T03:20:29.230 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/daemon-helper 2024-03-19T03:20:29.263 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-19T03:20:29.332 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-19T03:20:29.332 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/daemon-helper 2024-03-19T03:20:29.364 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-19T03:20:29.435 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-19T03:20:29.435 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-19T03:20:29.435 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-19T03:20:29.522 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-19T03:20:29.650 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-19T03:20:29.651 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-19T03:20:29.677 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-19T03:20:29.744 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-19T03:20:29.745 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-19T03:20:29.775 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-19T03:20:29.846 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-19T03:20:29.847 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-19T03:20:29.847 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/stdin-killer 2024-03-19T03:20:29.923 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-19T03:20:30.043 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-19T03:20:30.044 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/stdin-killer 2024-03-19T03:20:30.070 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-19T03:20:30.136 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-19T03:20:30.136 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/stdin-killer 2024-03-19T03:20:30.167 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-19T03:20:30.237 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-19T03:20:30.332 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'PG_AVAILABILITY', 'PG_DEGRADED', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-19T03:20:30.332 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T03:20:30.333 INFO:tasks.cephadm:Cluster fsid is a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:20:30.333 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-19T03:20:30.333 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.117', 'mon.b': '172.21.15.119', 'mon.c': '172.21.15.178'} 2024-03-19T03:20:30.333 INFO:tasks.cephadm:First mon is mon.a on smithi117 2024-03-19T03:20:30.333 INFO:tasks.cephadm:First mgr is x 2024-03-19T03:20:30.333 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-19T03:20:30.333 DEBUG:teuthology.orchestra.run.smithi117:> sudo hostname $(hostname -s) 2024-03-19T03:20:30.422 DEBUG:teuthology.orchestra.run.smithi119:> sudo hostname $(hostname -s) 2024-03-19T03:20:30.448 DEBUG:teuthology.orchestra.run.smithi178:> sudo hostname $(hostname -s) 2024-03-19T03:20:30.479 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-19T03:20:30.479 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T03:20:30.618 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77736/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2206-gb6ae2491', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2206.gb6ae2491'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-18 20:42:27.872142', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240318.181317', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-19T03:20:30.832 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-batrick-testing-20240318.181317, sha1 b6ae24918b03b1b8a19c9263a50655bba80bb810 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T03:20:30.833 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/x86_64/flavors/default/cephadm 2024-03-19T03:20:30.833 DEBUG:teuthology.orchestra.run.smithi117:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-19T03:20:31.574 INFO:teuthology.orchestra.run.smithi117.stdout:-rw-r--r--. 1 ubuntu ubuntu 767404 Mar 19 03:20 /home/ubuntu/cephtest/cephadm 2024-03-19T03:20:31.575 DEBUG:teuthology.orchestra.run.smithi119:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-19T03:20:32.069 INFO:teuthology.orchestra.run.smithi119.stdout:-rw-r--r--. 1 ubuntu ubuntu 767404 Mar 19 03:20 /home/ubuntu/cephtest/cephadm 2024-03-19T03:20:32.070 DEBUG:teuthology.orchestra.run.smithi178:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-19T03:20:32.675 INFO:teuthology.orchestra.run.smithi178.stdout:-rw-r--r--. 1 ubuntu ubuntu 767404 Mar 19 03:20 /home/ubuntu/cephtest/cephadm 2024-03-19T03:20:32.676 DEBUG:teuthology.orchestra.run.smithi117:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-19T03:20:32.736 DEBUG:teuthology.orchestra.run.smithi119:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-19T03:20:32.755 DEBUG:teuthology.orchestra.run.smithi178:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-19T03:20:32.789 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 on all hosts... 2024-03-19T03:20:32.789 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 pull 2024-03-19T03:20:32.797 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 pull 2024-03-19T03:20:32.800 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 pull 2024-03-19T03:20:33.666 INFO:teuthology.orchestra.run.smithi119.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810... 2024-03-19T03:20:33.667 INFO:teuthology.orchestra.run.smithi178.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810... 2024-03-19T03:20:33.811 INFO:teuthology.orchestra.run.smithi117.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810... 2024-03-19T03:21:04.783 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-19T03:21:04.784 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev)", 2024-03-19T03:21:04.784 INFO:teuthology.orchestra.run.smithi119.stdout: "image_id": "2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c", 2024-03-19T03:21:04.784 INFO:teuthology.orchestra.run.smithi119.stdout: "repo_digests": [ 2024-03-19T03:21:04.784 INFO:teuthology.orchestra.run.smithi119.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639" 2024-03-19T03:21:04.784 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-03-19T03:21:04.784 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-19T03:21:06.994 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-19T03:21:06.995 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_version": "ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev)", 2024-03-19T03:21:06.995 INFO:teuthology.orchestra.run.smithi117.stdout: "image_id": "2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c", 2024-03-19T03:21:06.995 INFO:teuthology.orchestra.run.smithi117.stdout: "repo_digests": [ 2024-03-19T03:21:06.995 INFO:teuthology.orchestra.run.smithi117.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639" 2024-03-19T03:21:06.995 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-03-19T03:21:06.995 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-19T03:21:09.573 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-03-19T03:21:09.573 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_version": "ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev)", 2024-03-19T03:21:09.573 INFO:teuthology.orchestra.run.smithi178.stdout: "image_id": "2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c", 2024-03-19T03:21:09.573 INFO:teuthology.orchestra.run.smithi178.stdout: "repo_digests": [ 2024-03-19T03:21:09.573 INFO:teuthology.orchestra.run.smithi178.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639" 2024-03-19T03:21:09.573 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2024-03-19T03:21:09.573 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-03-19T03:21:09.598 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /etc/ceph 2024-03-19T03:21:09.701 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /etc/ceph 2024-03-19T03:21:09.737 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /etc/ceph 2024-03-19T03:21:09.767 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 777 /etc/ceph 2024-03-19T03:21:09.865 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 777 /etc/ceph 2024-03-19T03:21:09.895 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 777 /etc/ceph 2024-03-19T03:21:09.924 INFO:tasks.cephadm:Writing seed config... 2024-03-19T03:21:09.925 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-19T03:21:09.925 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-19T03:21:09.925 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-19T03:21:09.925 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-19T03:21:09.925 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-19T03:21:09.925 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-19T03:21:09.925 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-19T03:21:09.925 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-19T03:21:09.925 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-19T03:21:09.925 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-19T03:21:09.925 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-19T03:21:09.925 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-19T03:21:09.926 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-19T03:21:09.926 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-19T03:21:09.926 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-19T03:21:09.926 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-19T03:21:09.926 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-19T03:21:09.926 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-03-19T03:21:09.926 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-03-19T03:21:09.926 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-19T03:21:09.926 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-19T03:21:09.926 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-19T03:21:09.926 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-19T03:21:09.926 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-19T03:21:09.926 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-19T03:21:09.927 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-19T03:21:09.927 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-19T03:21:09.927 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-19T03:21:09.927 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-19T03:21:09.927 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-19T03:21:09.927 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-19T03:21:09.927 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-19T03:21:09.927 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-19T03:21:09.927 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-19T03:21:09.927 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-19T03:21:09.927 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-19T03:21:09.928 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-19T03:21:09.928 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-19T03:21:09.928 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-19T03:21:09.928 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-19T03:21:09.928 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-19T03:21:09.928 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-19T03:21:09.928 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-19T03:21:09.928 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-19T03:21:09.929 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-19T03:21:09.929 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-19T03:21:09.930 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-19T03:21:09.983 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = a3a82456-e59f-11ee-95c9-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-19T03:21:09.984 DEBUG:teuthology.orchestra.run.smithi117:mon.a> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mon.a.service 2024-03-19T03:21:10.029 DEBUG:teuthology.orchestra.run.smithi117:mgr.x> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mgr.x.service 2024-03-19T03:21:10.075 INFO:tasks.cephadm:Bootstrapping... 2024-03-19T03:21:10.075 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 -v bootstrap --fsid a3a82456-e59f-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.117 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-19T03:21:10.354 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-19T03:21:10.354 INFO:teuthology.orchestra.run.smithi117.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810', '-v', 'bootstrap', '--fsid', 'a3a82456-e59f-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.117', '--skip-admin-label'] 2024-03-19T03:21:10.406 INFO:teuthology.orchestra.run.smithi117.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-19T03:21:10.407 INFO:teuthology.orchestra.run.smithi117.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-19T03:21:10.407 INFO:teuthology.orchestra.run.smithi117.stdout:Verifying podman|docker is present... 2024-03-19T03:21:10.466 INFO:teuthology.orchestra.run.smithi117.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-19T03:21:10.466 INFO:teuthology.orchestra.run.smithi117.stdout:Verifying lvm2 is present... 2024-03-19T03:21:10.466 INFO:teuthology.orchestra.run.smithi117.stdout:Verifying time synchronization is in place... 2024-03-19T03:21:10.485 INFO:teuthology.orchestra.run.smithi117.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-19T03:21:10.485 INFO:teuthology.orchestra.run.smithi117.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-19T03:21:10.505 INFO:teuthology.orchestra.run.smithi117.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-19T03:21:10.505 INFO:teuthology.orchestra.run.smithi117.stdout:systemctl: stdout inactive 2024-03-19T03:21:10.529 INFO:teuthology.orchestra.run.smithi117.stdout:systemctl: stdout enabled 2024-03-19T03:21:10.551 INFO:teuthology.orchestra.run.smithi117.stdout:systemctl: stdout active 2024-03-19T03:21:10.551 INFO:teuthology.orchestra.run.smithi117.stdout:Unit chronyd.service is enabled and running 2024-03-19T03:21:10.552 INFO:teuthology.orchestra.run.smithi117.stdout:Repeating the final host check... 2024-03-19T03:21:10.609 INFO:teuthology.orchestra.run.smithi117.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-19T03:21:10.609 INFO:teuthology.orchestra.run.smithi117.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-19T03:21:10.609 INFO:teuthology.orchestra.run.smithi117.stdout:systemctl is present 2024-03-19T03:21:10.609 INFO:teuthology.orchestra.run.smithi117.stdout:lvcreate is present 2024-03-19T03:21:10.623 INFO:teuthology.orchestra.run.smithi117.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-19T03:21:10.623 INFO:teuthology.orchestra.run.smithi117.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-19T03:21:10.641 INFO:teuthology.orchestra.run.smithi117.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-19T03:21:10.641 INFO:teuthology.orchestra.run.smithi117.stdout:systemctl: stdout inactive 2024-03-19T03:21:10.663 INFO:teuthology.orchestra.run.smithi117.stdout:systemctl: stdout enabled 2024-03-19T03:21:10.683 INFO:teuthology.orchestra.run.smithi117.stdout:systemctl: stdout active 2024-03-19T03:21:10.683 INFO:teuthology.orchestra.run.smithi117.stdout:Unit chronyd.service is enabled and running 2024-03-19T03:21:10.684 INFO:teuthology.orchestra.run.smithi117.stdout:Host looks OK 2024-03-19T03:21:10.684 INFO:teuthology.orchestra.run.smithi117.stdout:Cluster fsid: a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:21:10.684 INFO:teuthology.orchestra.run.smithi117.stdout:Acquiring lock 139956033043904 on /run/cephadm/a3a82456-e59f-11ee-95c9-87774f69a715.lock 2024-03-19T03:21:10.685 INFO:teuthology.orchestra.run.smithi117.stdout:Lock 139956033043904 acquired on /run/cephadm/a3a82456-e59f-11ee-95c9-87774f69a715.lock 2024-03-19T03:21:10.686 INFO:teuthology.orchestra.run.smithi117.stdout:Verifying IP 172.21.15.117 port 3300 ... 2024-03-19T03:21:10.687 INFO:teuthology.orchestra.run.smithi117.stdout:Verifying IP 172.21.15.117 port 6789 ... 2024-03-19T03:21:10.687 INFO:teuthology.orchestra.run.smithi117.stdout:Base mon IP(s) is [172.21.15.117:3300, 172.21.15.117:6789], mon addrv is [v2:172.21.15.117:3300,v1:172.21.15.117:6789] 2024-03-19T03:21:10.702 INFO:teuthology.orchestra.run.smithi117.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.117 metric 100 2024-03-19T03:21:10.702 INFO:teuthology.orchestra.run.smithi117.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.117 metric 100 2024-03-19T03:21:10.714 INFO:teuthology.orchestra.run.smithi117.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-19T03:21:10.714 INFO:teuthology.orchestra.run.smithi117.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-19T03:21:10.715 INFO:teuthology.orchestra.run.smithi117.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1791sec hoplimit 64 pref medium 2024-03-19T03:21:10.725 INFO:teuthology.orchestra.run.smithi117.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-19T03:21:10.726 INFO:teuthology.orchestra.run.smithi117.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-19T03:21:10.726 INFO:teuthology.orchestra.run.smithi117.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-19T03:21:10.726 INFO:teuthology.orchestra.run.smithi117.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-19T03:21:10.726 INFO:teuthology.orchestra.run.smithi117.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:4463/64 scope link 2024-03-19T03:21:10.726 INFO:teuthology.orchestra.run.smithi117.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-19T03:21:10.728 INFO:teuthology.orchestra.run.smithi117.stdout:Mon IP `172.21.15.117` is in CIDR network `172.21.0.0/20` 2024-03-19T03:21:10.728 INFO:teuthology.orchestra.run.smithi117.stdout:Mon IP `172.21.15.117` is in CIDR network `172.21.0.0/20` 2024-03-19T03:21:10.728 INFO:teuthology.orchestra.run.smithi117.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-19T03:21:10.731 INFO:teuthology.orchestra.run.smithi117.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-19T03:21:10.733 INFO:teuthology.orchestra.run.smithi117.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810... 2024-03-19T03:21:11.460 INFO:teuthology.orchestra.run.smithi117.stdout:/bin/podman: stdout 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c 2024-03-19T03:21:11.461 INFO:teuthology.orchestra.run.smithi117.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810... 2024-03-19T03:21:11.461 INFO:teuthology.orchestra.run.smithi117.stdout:/bin/podman: stderr Getting image source signatures 2024-03-19T03:21:11.461 INFO:teuthology.orchestra.run.smithi117.stdout:/bin/podman: stderr Copying blob sha256:93b5a8d362aeee7e8b1f0bbf18238e1bb256c50573ad5b72aff9e382b1bb3ccb 2024-03-19T03:21:11.461 INFO:teuthology.orchestra.run.smithi117.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-19T03:21:11.461 INFO:teuthology.orchestra.run.smithi117.stdout:/bin/podman: stderr Copying config sha256:2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c 2024-03-19T03:21:11.461 INFO:teuthology.orchestra.run.smithi117.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-19T03:21:12.694 INFO:teuthology.orchestra.run.smithi117.stdout:ceph: stdout ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev) 2024-03-19T03:21:12.694 INFO:teuthology.orchestra.run.smithi117.stdout:Ceph version: ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev) 2024-03-19T03:21:12.694 INFO:teuthology.orchestra.run.smithi117.stdout:Extracting ceph user uid/gid from container image... 2024-03-19T03:21:13.778 INFO:teuthology.orchestra.run.smithi117.stdout:stat: stdout 167 167 2024-03-19T03:21:13.778 INFO:teuthology.orchestra.run.smithi117.stdout:Creating initial keys... 2024-03-19T03:21:14.761 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph-authtool: stdout AQCqBPllwmN1DBAAty9Abvw6m3xS4U9MFamRow== 2024-03-19T03:21:15.812 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph-authtool: stdout AQCrBPllKeBFCxAAz+kg8ZjY/KlCxMrMbhXMEw== 2024-03-19T03:21:25.781 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph-authtool: stdout AQCvBPllL4flJRAAJYSZJs1YWtIEowXHchZi+w== 2024-03-19T03:21:25.783 INFO:teuthology.orchestra.run.smithi117.stdout:Creating initial monmap... 2024-03-19T03:21:26.792 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-19T03:21:26.793 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-19T03:21:26.793 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:21:26.793 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-19T03:21:26.793 INFO:teuthology.orchestra.run.smithi117.stdout:monmaptool for a [v2:172.21.15.117:3300,v1:172.21.15.117:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-19T03:21:26.793 INFO:teuthology.orchestra.run.smithi117.stdout:setting min_mon_release = quincy 2024-03-19T03:21:26.794 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/monmaptool: set fsid to a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:21:26.794 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-19T03:21:26.794 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:21:26.794 INFO:teuthology.orchestra.run.smithi117.stdout:Creating mon... 2024-03-19T03:21:28.103 INFO:teuthology.orchestra.run.smithi117.stdout:create mon.a on 2024-03-19T03:21:28.441 INFO:teuthology.orchestra.run.smithi117.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-19T03:21:28.677 INFO:teuthology.orchestra.run.smithi117.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-19T03:21:28.981 INFO:teuthology.orchestra.run.smithi117.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-a3a82456-e59f-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-a3a82456-e59f-11ee-95c9-87774f69a715.target. 2024-03-19T03:21:28.982 INFO:teuthology.orchestra.run.smithi117.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-a3a82456-e59f-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-a3a82456-e59f-11ee-95c9-87774f69a715.target. 2024-03-19T03:21:29.359 INFO:teuthology.orchestra.run.smithi117.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mon.a 2024-03-19T03:21:29.359 INFO:teuthology.orchestra.run.smithi117.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mon.a.service: Unit ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mon.a.service not loaded. 2024-03-19T03:21:29.650 INFO:teuthology.orchestra.run.smithi117.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a3a82456-e59f-11ee-95c9-87774f69a715.target.wants/ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-a3a82456-e59f-11ee-95c9-87774f69a715@.service. 2024-03-19T03:21:30.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:30 smithi117 podman[31440]: 2024-03-19 03:21:30.080126806 +0000 UTC m=+0.172338185 container create 096660baf36df80b6b60218756b84b8b7721c7a90dda8043208c24d70afda853 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True) 2024-03-19T03:21:30.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:30 smithi117 podman[31440]: 2024-03-19 03:21:30.341626373 +0000 UTC m=+0.433837756 container init 096660baf36df80b6b60218756b84b8b7721c7a90dda8043208c24d70afda853 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-19T03:21:30.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:30 smithi117 podman[31440]: 2024-03-19 03:21:30.35153685 +0000 UTC m=+0.443748232 container start 096660baf36df80b6b60218756b84b8b7721c7a90dda8043208c24d70afda853 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-19T03:21:30.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:30 smithi117 ceph-mon[31458]: mkfs a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:21:30.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:30 smithi117 ceph-mon[31458]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-19T03:21:31.524 INFO:teuthology.orchestra.run.smithi117.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-19T03:21:31.525 INFO:teuthology.orchestra.run.smithi117.stdout:systemctl: stdout disabled 2024-03-19T03:21:31.546 INFO:teuthology.orchestra.run.smithi117.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-19T03:21:31.546 INFO:teuthology.orchestra.run.smithi117.stdout:systemctl: stdout inactive 2024-03-19T03:21:31.546 INFO:teuthology.orchestra.run.smithi117.stdout:firewalld.service is not enabled 2024-03-19T03:21:31.547 INFO:teuthology.orchestra.run.smithi117.stdout:Not possible to enable service . firewalld.service is not available 2024-03-19T03:21:31.548 INFO:teuthology.orchestra.run.smithi117.stdout:Waiting for mon to start... 2024-03-19T03:21:31.549 INFO:teuthology.orchestra.run.smithi117.stdout:Waiting for mon... 2024-03-19T03:21:31.705 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:31 smithi117 ceph-mon[31458]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-19T03:21:31.706 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:31 smithi117 ceph-mon[31458]: monmap epoch 1 2024-03-19T03:21:31.706 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:31 smithi117 ceph-mon[31458]: fsid a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:21:31.706 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:31 smithi117 ceph-mon[31458]: last_changed 2024-03-19T03:21:26.225562+0000 2024-03-19T03:21:31.706 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:31 smithi117 ceph-mon[31458]: created 2024-03-19T03:21:26.225562+0000 2024-03-19T03:21:31.706 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:31 smithi117 ceph-mon[31458]: min_mon_release 19 (squid) 2024-03-19T03:21:31.706 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:31 smithi117 ceph-mon[31458]: election_strategy: 1 2024-03-19T03:21:31.706 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:31 smithi117 ceph-mon[31458]: 0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-03-19T03:21:31.706 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:31 smithi117 ceph-mon[31458]: fsmap 2024-03-19T03:21:31.706 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:31 smithi117 ceph-mon[31458]: osdmap e1: 0 total, 0 up, 0 in 2024-03-19T03:21:31.706 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:31 smithi117 ceph-mon[31458]: mgrmap e1: no daemons active 2024-03-19T03:21:31.706 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:31 smithi117 bash[31440]: 096660baf36df80b6b60218756b84b8b7721c7a90dda8043208c24d70afda853 2024-03-19T03:21:31.706 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:31 smithi117 systemd[1]: Started Ceph mon.a for a3a82456-e59f-11ee-95c9-87774f69a715. 2024-03-19T03:21:32.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:32 smithi117 ceph-mon[31458]: from='client.? 172.21.15.117:0/3665886977' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-19T03:21:32.906 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout cluster: 2024-03-19T03:21:32.907 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout id: a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:21:32.907 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-19T03:21:32.907 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout 2024-03-19T03:21:32.908 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout services: 2024-03-19T03:21:32.908 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 1.81516s) 2024-03-19T03:21:32.908 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-19T03:21:32.913 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-19T03:21:32.913 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout 2024-03-19T03:21:32.913 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout data: 2024-03-19T03:21:32.913 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-19T03:21:32.913 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-19T03:21:32.914 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-19T03:21:32.914 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout pgs: 2024-03-19T03:21:32.914 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout 2024-03-19T03:21:32.914 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.146+0000 7f88f5000700 1 Processor -- start 2024-03-19T03:21:32.914 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.147+0000 7f88f5000700 1 -- start start 2024-03-19T03:21:32.914 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.147+0000 7f88f5000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f0106c20 0x7f88f0107040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:32.914 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.147+0000 7f88f5000700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f88f0107610 con 0x7f88f0106c20 2024-03-19T03:21:32.914 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.148+0000 7f88ede00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f0106c20 0x7f88f0107040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:21:32.914 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.148+0000 7f88ede00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f0106c20 0x7f88f0107040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:47768/0 (socket says 172.21.15.117:47768) 2024-03-19T03:21:32.914 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.148+0000 7f88ede00700 1 -- 172.21.15.117:0/4190053124 learned_addr learned my addr 172.21.15.117:0/4190053124 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:21:32.914 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.149+0000 7f88ede00700 1 -- 172.21.15.117:0/4190053124 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f0107dd0 con 0x7f88f0106c20 2024-03-19T03:21:32.914 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.149+0000 7f88ede00700 1 --2- 172.21.15.117:0/4190053124 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f0106c20 0x7f88f0107040 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f88e00098a0 tx=0x7f88e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6798c4bd1ed74824 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:21:32.914 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.149+0000 7f88eca00700 1 -- 172.21.15.117:0/4190053124 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88e0004160 con 0x7f88f0106c20 2024-03-19T03:21:32.915 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.149+0000 7f88eca00700 1 -- 172.21.15.117:0/4190053124 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88e0004300 con 0x7f88f0106c20 2024-03-19T03:21:32.915 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.149+0000 7f88eca00700 1 -- 172.21.15.117:0/4190053124 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88e00036a0 con 0x7f88f0106c20 2024-03-19T03:21:32.915 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.150+0000 7f88f5000700 1 -- 172.21.15.117:0/4190053124 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f0106c20 msgr2=0x7f88f0107040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:21:32.915 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.150+0000 7f88f5000700 1 --2- 172.21.15.117:0/4190053124 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f0106c20 0x7f88f0107040 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f88e00098a0 tx=0x7f88e0009bb0 comp rx=0 tx=0).stop 2024-03-19T03:21:32.915 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.150+0000 7f88f5000700 1 -- 172.21.15.117:0/4190053124 shutdown_connections 2024-03-19T03:21:32.915 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.150+0000 7f88f5000700 1 --2- 172.21.15.117:0/4190053124 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f0106c20 0x7f88f0107040 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:32.915 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.150+0000 7f88f5000700 1 -- 172.21.15.117:0/4190053124 >> 172.21.15.117:0/4190053124 conn(0x7f88f0102200 msgr2=0x7f88f0104620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:21:32.915 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.150+0000 7f88f5000700 1 -- 172.21.15.117:0/4190053124 shutdown_connections 2024-03-19T03:21:32.915 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.150+0000 7f88f5000700 1 -- 172.21.15.117:0/4190053124 wait complete. 2024-03-19T03:21:32.915 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.152+0000 7f88f5000700 1 Processor -- start 2024-03-19T03:21:32.915 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.152+0000 7f88f5000700 1 -- start start 2024-03-19T03:21:32.915 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.152+0000 7f88f5000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f0106c20 0x7f88f019d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:32.915 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.152+0000 7f88f5000700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f88f0108240 con 0x7f88f0106c20 2024-03-19T03:21:32.915 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.152+0000 7f88ede00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f0106c20 0x7f88f019d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:21:32.916 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.153+0000 7f88ede00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f0106c20 0x7f88f019d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:47772/0 (socket says 172.21.15.117:47772) 2024-03-19T03:21:32.916 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.153+0000 7f88ede00700 1 -- 172.21.15.117:0/3665886977 learned_addr learned my addr 172.21.15.117:0/3665886977 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:21:32.916 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.153+0000 7f88ede00700 1 -- 172.21.15.117:0/3665886977 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f019db40 con 0x7f88f0106c20 2024-03-19T03:21:32.916 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.153+0000 7f88ede00700 1 --2- 172.21.15.117:0/3665886977 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f0106c20 0x7f88f019d600 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f88e0000c00 tx=0x7f88e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:21:32.916 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.153+0000 7f88e6a00700 1 -- 172.21.15.117:0/3665886977 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88e0003db0 con 0x7f88f0106c20 2024-03-19T03:21:32.916 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.154+0000 7f88e6a00700 1 -- 172.21.15.117:0/3665886977 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88e0011630 con 0x7f88f0106c20 2024-03-19T03:21:32.916 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.154+0000 7f88f5000700 1 -- 172.21.15.117:0/3665886977 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88f019ddd0 con 0x7f88f0106c20 2024-03-19T03:21:32.916 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.154+0000 7f88e6a00700 1 -- 172.21.15.117:0/3665886977 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88e0011950 con 0x7f88f0106c20 2024-03-19T03:21:32.916 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.154+0000 7f88f5000700 1 -- 172.21.15.117:0/3665886977 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88f01a0a70 con 0x7f88f0106c20 2024-03-19T03:21:32.916 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.154+0000 7f88e6a00700 1 -- 172.21.15.117:0/3665886977 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f88e0011b80 con 0x7f88f0106c20 2024-03-19T03:21:32.916 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.154+0000 7f88e6a00700 1 -- 172.21.15.117:0/3665886977 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f88e001c450 con 0x7f88f0106c20 2024-03-19T03:21:32.916 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.155+0000 7f88f5000700 1 -- 172.21.15.117:0/3665886977 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88f0063940 con 0x7f88f0106c20 2024-03-19T03:21:32.916 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.159+0000 7f88e6a00700 1 -- 172.21.15.117:0/3665886977 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f88e0032020 con 0x7f88f0106c20 2024-03-19T03:21:32.916 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.239+0000 7f88f5000700 1 -- 172.21.15.117:0/3665886977 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f88f002ddb0 con 0x7f88f0106c20 2024-03-19T03:21:32.917 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.240+0000 7f88e6a00700 1 -- 172.21.15.117:0/3665886977 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+316 (secure 0 0 0) 0x7f88e0004260 con 0x7f88f0106c20 2024-03-19T03:21:32.917 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.241+0000 7f88f5000700 1 -- 172.21.15.117:0/3665886977 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f0106c20 msgr2=0x7f88f019d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:21:32.917 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.241+0000 7f88f5000700 1 --2- 172.21.15.117:0/3665886977 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f0106c20 0x7f88f019d600 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f88e0000c00 tx=0x7f88e0004750 comp rx=0 tx=0).stop 2024-03-19T03:21:32.917 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.241+0000 7f88f5000700 1 -- 172.21.15.117:0/3665886977 shutdown_connections 2024-03-19T03:21:32.917 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.241+0000 7f88f5000700 1 --2- 172.21.15.117:0/3665886977 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88f0106c20 0x7f88f019d600 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:32.917 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.241+0000 7f88f5000700 1 -- 172.21.15.117:0/3665886977 >> 172.21.15.117:0/3665886977 conn(0x7f88f0102200 msgr2=0x7f88f018fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:21:32.917 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.241+0000 7f88f5000700 1 -- 172.21.15.117:0/3665886977 shutdown_connections 2024-03-19T03:21:32.917 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:32.241+0000 7f88f5000700 1 -- 172.21.15.117:0/3665886977 wait complete. 2024-03-19T03:21:32.917 INFO:teuthology.orchestra.run.smithi117.stdout:mon is available 2024-03-19T03:21:32.917 INFO:teuthology.orchestra.run.smithi117.stdout:Assimilating anything we can from ceph.conf... 2024-03-19T03:21:33.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:33 smithi117 ceph-mon[31458]: from='client.? 172.21.15.117:0/2666039419' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-19T03:21:33.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:33 smithi117 ceph-mon[31458]: from='client.? 172.21.15.117:0/2666039419' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-19T03:21:34.172 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout 2024-03-19T03:21:34.173 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout [global] 2024-03-19T03:21:34.173 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout fsid = a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:21:34.174 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.117:3300,v1:172.21.15.117:6789] 2024-03-19T03:21:34.174 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-19T03:21:34.175 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-19T03:21:34.175 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-19T03:21:34.175 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-19T03:21:34.175 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout 2024-03-19T03:21:34.175 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-19T03:21:34.175 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-19T03:21:34.175 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout 2024-03-19T03:21:34.175 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout [osd] 2024-03-19T03:21:34.175 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-19T03:21:34.175 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-19T03:21:34.175 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-19T03:21:34.175 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-19T03:21:34.175 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-19T03:21:34.179 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.453+0000 7f5524200700 1 Processor -- start 2024-03-19T03:21:34.179 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.453+0000 7f5524200700 1 -- start start 2024-03-19T03:21:34.179 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.454+0000 7f5524200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f551c106c20 0x7f551c107040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:34.179 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.454+0000 7f5524200700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f551c107610 con 0x7f551c106c20 2024-03-19T03:21:34.180 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.454+0000 7f5521800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f551c106c20 0x7f551c107040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:21:34.180 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.454+0000 7f5521800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f551c106c20 0x7f551c107040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:47782/0 (socket says 172.21.15.117:47782) 2024-03-19T03:21:34.180 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.454+0000 7f5521800700 1 -- 172.21.15.117:0/4612135 learned_addr learned my addr 172.21.15.117:0/4612135 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:21:34.180 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.455+0000 7f5521800700 1 -- 172.21.15.117:0/4612135 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f551c107e20 con 0x7f551c106c20 2024-03-19T03:21:34.180 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.455+0000 7f5521800700 1 --2- 172.21.15.117:0/4612135 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f551c106c20 0x7f551c107040 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f55100098a0 tx=0x7f5510009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb427f61a86bcc32 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:21:34.180 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.455+0000 7f551be00700 1 -- 172.21.15.117:0/4612135 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5510004160 con 0x7f551c106c20 2024-03-19T03:21:34.180 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.455+0000 7f551be00700 1 -- 172.21.15.117:0/4612135 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5510004300 con 0x7f551c106c20 2024-03-19T03:21:34.180 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.455+0000 7f551be00700 1 -- 172.21.15.117:0/4612135 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55100036a0 con 0x7f551c106c20 2024-03-19T03:21:34.180 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.456+0000 7f5524200700 1 -- 172.21.15.117:0/4612135 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f551c106c20 msgr2=0x7f551c107040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:21:34.180 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.456+0000 7f5524200700 1 --2- 172.21.15.117:0/4612135 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f551c106c20 0x7f551c107040 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f55100098a0 tx=0x7f5510009bb0 comp rx=0 tx=0).stop 2024-03-19T03:21:34.180 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.456+0000 7f5524200700 1 -- 172.21.15.117:0/4612135 shutdown_connections 2024-03-19T03:21:34.180 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.456+0000 7f5524200700 1 --2- 172.21.15.117:0/4612135 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f551c106c20 0x7f551c107040 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:34.180 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.456+0000 7f5524200700 1 -- 172.21.15.117:0/4612135 >> 172.21.15.117:0/4612135 conn(0x7f551c1021c0 msgr2=0x7f551c104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:21:34.181 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.456+0000 7f5524200700 1 -- 172.21.15.117:0/4612135 shutdown_connections 2024-03-19T03:21:34.181 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.456+0000 7f5524200700 1 -- 172.21.15.117:0/4612135 wait complete. 2024-03-19T03:21:34.181 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.457+0000 7f5524200700 1 Processor -- start 2024-03-19T03:21:34.181 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.457+0000 7f5524200700 1 -- start start 2024-03-19T03:21:34.181 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.458+0000 7f5524200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f551c106c20 0x7f551c19d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:34.181 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.458+0000 7f5524200700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f551c108350 con 0x7f551c106c20 2024-03-19T03:21:34.181 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.458+0000 7f5521800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f551c106c20 0x7f551c19d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:21:34.181 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.458+0000 7f5521800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f551c106c20 0x7f551c19d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:47794/0 (socket says 172.21.15.117:47794) 2024-03-19T03:21:34.181 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.458+0000 7f5521800700 1 -- 172.21.15.117:0/2666039419 learned_addr learned my addr 172.21.15.117:0/2666039419 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:21:34.181 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.459+0000 7f5521800700 1 -- 172.21.15.117:0/2666039419 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f551c19db70 con 0x7f551c106c20 2024-03-19T03:21:34.181 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.459+0000 7f5521800700 1 --2- 172.21.15.117:0/2666039419 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f551c106c20 0x7f551c19d630 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f5510000c00 tx=0x7f5510004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:21:34.181 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.459+0000 7f551a000700 1 -- 172.21.15.117:0/2666039419 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5510003db0 con 0x7f551c106c20 2024-03-19T03:21:34.181 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.459+0000 7f551a000700 1 -- 172.21.15.117:0/2666039419 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5510011630 con 0x7f551c106c20 2024-03-19T03:21:34.181 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.459+0000 7f5524200700 1 -- 172.21.15.117:0/2666039419 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f551c19de00 con 0x7f551c106c20 2024-03-19T03:21:34.182 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.459+0000 7f551a000700 1 -- 172.21.15.117:0/2666039419 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5510011950 con 0x7f551c106c20 2024-03-19T03:21:34.182 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.459+0000 7f5524200700 1 -- 172.21.15.117:0/2666039419 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f551c1a0af0 con 0x7f551c106c20 2024-03-19T03:21:34.182 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.460+0000 7f551a000700 1 -- 172.21.15.117:0/2666039419 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5510011af0 con 0x7f551c106c20 2024-03-19T03:21:34.182 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.460+0000 7f551a000700 1 -- 172.21.15.117:0/2666039419 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f551001c450 con 0x7f551c106c20 2024-03-19T03:21:34.182 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.461+0000 7f5524200700 1 -- 172.21.15.117:0/2666039419 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f551c063940 con 0x7f551c106c20 2024-03-19T03:21:34.182 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.463+0000 7f551a000700 1 -- 172.21.15.117:0/2666039419 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f5510032020 con 0x7f551c106c20 2024-03-19T03:21:34.182 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.540+0000 7f5524200700 1 -- 172.21.15.117:0/2666039419 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f551c02ddc0 con 0x7f551c106c20 2024-03-19T03:21:34.183 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.547+0000 7f551a000700 1 -- 172.21.15.117:0/2666039419 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+431 (secure 0 0 0) 0x7f5510004260 con 0x7f551c106c20 2024-03-19T03:21:34.183 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.548+0000 7f5524200700 1 -- 172.21.15.117:0/2666039419 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f551c106c20 msgr2=0x7f551c19d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:21:34.183 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.548+0000 7f5524200700 1 --2- 172.21.15.117:0/2666039419 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f551c106c20 0x7f551c19d630 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f5510000c00 tx=0x7f5510004750 comp rx=0 tx=0).stop 2024-03-19T03:21:34.183 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.549+0000 7f5524200700 1 -- 172.21.15.117:0/2666039419 shutdown_connections 2024-03-19T03:21:34.184 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.549+0000 7f5524200700 1 --2- 172.21.15.117:0/2666039419 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f551c106c20 0x7f551c19d630 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:34.184 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.549+0000 7f5524200700 1 -- 172.21.15.117:0/2666039419 >> 172.21.15.117:0/2666039419 conn(0x7f551c1021c0 msgr2=0x7f551c194370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:21:34.184 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.549+0000 7f5524200700 1 -- 172.21.15.117:0/2666039419 shutdown_connections 2024-03-19T03:21:34.184 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:33.549+0000 7f5524200700 1 -- 172.21.15.117:0/2666039419 wait complete. 2024-03-19T03:21:34.184 INFO:teuthology.orchestra.run.smithi117.stdout:Generating new minimal ceph.conf... 2024-03-19T03:21:35.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:34 smithi117 ceph-mon[31458]: from='client.? 172.21.15.117:0/1583996366' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:21:35.439 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.775+0000 7fb4a5a00700 1 Processor -- start 2024-03-19T03:21:35.439 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.775+0000 7fb4a5a00700 1 -- start start 2024-03-19T03:21:35.440 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.775+0000 7fb4a5a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4a010a0e0 0x7fb4a010a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:35.441 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.775+0000 7fb4a5a00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a010aa40 con 0x7fb4a010a0e0 2024-03-19T03:21:35.441 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.775+0000 7fb49ea00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4a010a0e0 0x7fb4a010a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:21:35.441 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.775+0000 7fb49ea00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4a010a0e0 0x7fb4a010a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:47796/0 (socket says 172.21.15.117:47796) 2024-03-19T03:21:35.441 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.775+0000 7fb49ea00700 1 -- 172.21.15.117:0/854911038 learned_addr learned my addr 172.21.15.117:0/854911038 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:21:35.441 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.776+0000 7fb49ea00700 1 -- 172.21.15.117:0/854911038 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4a010b080 con 0x7fb4a010a0e0 2024-03-19T03:21:35.441 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.776+0000 7fb49ea00700 1 --2- 172.21.15.117:0/854911038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4a010a0e0 0x7fb4a010a500 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb4900098a0 tx=0x7fb490009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f288ca7f1b2d50a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:21:35.441 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.776+0000 7fb49d600700 1 -- 172.21.15.117:0/854911038 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb490004160 con 0x7fb4a010a0e0 2024-03-19T03:21:35.444 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.776+0000 7fb49d600700 1 -- 172.21.15.117:0/854911038 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb490004300 con 0x7fb4a010a0e0 2024-03-19T03:21:35.444 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.776+0000 7fb49d600700 1 -- 172.21.15.117:0/854911038 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb490011b50 con 0x7fb4a010a0e0 2024-03-19T03:21:35.445 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.777+0000 7fb4a5a00700 1 -- 172.21.15.117:0/854911038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4a010a0e0 msgr2=0x7fb4a010a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:21:35.445 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.777+0000 7fb4a5a00700 1 --2- 172.21.15.117:0/854911038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4a010a0e0 0x7fb4a010a500 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb4900098a0 tx=0x7fb490009bb0 comp rx=0 tx=0).stop 2024-03-19T03:21:35.445 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.777+0000 7fb4a5a00700 1 -- 172.21.15.117:0/854911038 shutdown_connections 2024-03-19T03:21:35.445 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.777+0000 7fb4a5a00700 1 --2- 172.21.15.117:0/854911038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4a010a0e0 0x7fb4a010a500 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:35.445 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.777+0000 7fb4a5a00700 1 -- 172.21.15.117:0/854911038 >> 172.21.15.117:0/854911038 conn(0x7fb4a007d310 msgr2=0x7fb4a007d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:21:35.445 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.777+0000 7fb4a5a00700 1 -- 172.21.15.117:0/854911038 shutdown_connections 2024-03-19T03:21:35.445 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.777+0000 7fb4a5a00700 1 -- 172.21.15.117:0/854911038 wait complete. 2024-03-19T03:21:35.445 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.778+0000 7fb4a5a00700 1 Processor -- start 2024-03-19T03:21:35.445 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.778+0000 7fb4a5a00700 1 -- start start 2024-03-19T03:21:35.445 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.778+0000 7fb4a5a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4a010a0e0 0x7fb4a019de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:35.445 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.778+0000 7fb4a5a00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a010e340 con 0x7fb4a010a0e0 2024-03-19T03:21:35.445 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.779+0000 7fb49ea00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4a010a0e0 0x7fb4a019de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:21:35.445 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.779+0000 7fb49ea00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4a010a0e0 0x7fb4a019de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:47804/0 (socket says 172.21.15.117:47804) 2024-03-19T03:21:35.445 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.779+0000 7fb49ea00700 1 -- 172.21.15.117:0/1583996366 learned_addr learned my addr 172.21.15.117:0/1583996366 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:21:35.446 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.779+0000 7fb49ea00700 1 -- 172.21.15.117:0/1583996366 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4a019e390 con 0x7fb4a010a0e0 2024-03-19T03:21:35.446 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.779+0000 7fb49ea00700 1 --2- 172.21.15.117:0/1583996366 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4a010a0e0 0x7fb4a019de50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb490011b60 tx=0x7fb490011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:21:35.446 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.780+0000 7fb497400700 1 -- 172.21.15.117:0/1583996366 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb490003b20 con 0x7fb4a010a0e0 2024-03-19T03:21:35.446 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.780+0000 7fb497400700 1 -- 172.21.15.117:0/1583996366 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb490003cc0 con 0x7fb4a010a0e0 2024-03-19T03:21:35.446 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.780+0000 7fb497400700 1 -- 172.21.15.117:0/1583996366 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb490024840 con 0x7fb4a010a0e0 2024-03-19T03:21:35.446 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.780+0000 7fb4a5a00700 1 -- 172.21.15.117:0/1583996366 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4a019e620 con 0x7fb4a010a0e0 2024-03-19T03:21:35.446 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.780+0000 7fb4a5a00700 1 -- 172.21.15.117:0/1583996366 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4a01a1310 con 0x7fb4a010a0e0 2024-03-19T03:21:35.446 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.780+0000 7fb497400700 1 -- 172.21.15.117:0/1583996366 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb490022020 con 0x7fb4a010a0e0 2024-03-19T03:21:35.446 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.780+0000 7fb497400700 1 -- 172.21.15.117:0/1583996366 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb490011e00 con 0x7fb4a010a0e0 2024-03-19T03:21:35.446 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.781+0000 7fb4a5a00700 1 -- 172.21.15.117:0/1583996366 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4a010e4c0 con 0x7fb4a010a0e0 2024-03-19T03:21:35.446 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.784+0000 7fb497400700 1 -- 172.21.15.117:0/1583996366 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb490043dd0 con 0x7fb4a010a0e0 2024-03-19T03:21:35.446 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.860+0000 7fb4a5a00700 1 -- 172.21.15.117:0/1583996366 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fb4a0063940 con 0x7fb4a010a0e0 2024-03-19T03:21:35.446 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.860+0000 7fb497400700 1 -- 172.21.15.117:0/1583996366 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+177 (secure 0 0 0) 0x7fb490032030 con 0x7fb4a010a0e0 2024-03-19T03:21:35.447 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.861+0000 7fb4a5a00700 1 -- 172.21.15.117:0/1583996366 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4a010a0e0 msgr2=0x7fb4a019de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:21:35.447 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.861+0000 7fb4a5a00700 1 --2- 172.21.15.117:0/1583996366 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4a010a0e0 0x7fb4a019de50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb490011b60 tx=0x7fb490011c40 comp rx=0 tx=0).stop 2024-03-19T03:21:35.447 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.861+0000 7fb4a5a00700 1 -- 172.21.15.117:0/1583996366 shutdown_connections 2024-03-19T03:21:35.447 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.861+0000 7fb4a5a00700 1 --2- 172.21.15.117:0/1583996366 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4a010a0e0 0x7fb4a019de50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:35.447 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.861+0000 7fb4a5a00700 1 -- 172.21.15.117:0/1583996366 >> 172.21.15.117:0/1583996366 conn(0x7fb4a007d310 msgr2=0x7fb4a0107cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:21:35.447 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.861+0000 7fb4a5a00700 1 -- 172.21.15.117:0/1583996366 shutdown_connections 2024-03-19T03:21:35.447 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:34.862+0000 7fb4a5a00700 1 -- 172.21.15.117:0/1583996366 wait complete. 2024-03-19T03:21:35.447 INFO:teuthology.orchestra.run.smithi117.stdout:Restarting the monitor... 2024-03-19T03:21:35.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:35 smithi117 systemd[1]: Stopping Ceph mon.a for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:21:36.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:35 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-a[31454]: 2024-03-19T03:21:35.919+0000 7fd7e5600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:21:36.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:35 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-a[31454]: 2024-03-19T03:21:35.919+0000 7fd7e5600700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-19T03:21:36.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:36 smithi117 podman[31702]: 2024-03-19 03:21:36.103622054 +0000 UTC m=+0.316724015 container died 096660baf36df80b6b60218756b84b8b7721c7a90dda8043208c24d70afda853 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-a, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T03:21:36.728 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:36 smithi117 podman[31702]: 2024-03-19 03:21:36.578691405 +0000 UTC m=+0.791793375 container cleanup 096660baf36df80b6b60218756b84b8b7721c7a90dda8043208c24d70afda853 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T03:21:36.728 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:36 smithi117 bash[31702]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-a 2024-03-19T03:21:37.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:36 smithi117 podman[31717]: 2024-03-19 03:21:36.89629977 +0000 UTC m=+0.777734255 container remove 096660baf36df80b6b60218756b84b8b7721c7a90dda8043208c24d70afda853 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-a, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T03:21:37.683 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 systemd[1]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-19T03:21:37.683 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 systemd[1]: Stopped Ceph mon.a for a3a82456-e59f-11ee-95c9-87774f69a715. 2024-03-19T03:21:37.683 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 systemd[1]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mon.a.service: Consumed 1.758s CPU time. 2024-03-19T03:21:37.683 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 systemd[1]: Starting Ceph mon.a for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:21:37.934 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 podman[31819]: 2024-03-19 03:21:37.610721447 +0000 UTC m=+0.034006176 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T03:21:37.934 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 podman[31819]: 2024-03-19 03:21:37.730334571 +0000 UTC m=+0.153619331 container create e14c366900a5afcce7dc85a3c5542c09342ca4aca90ffd51346751015653e315 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True) 2024-03-19T03:21:37.934 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 podman[31819]: 2024-03-19 03:21:37.933639206 +0000 UTC m=+0.356923972 container init e14c366900a5afcce7dc85a3c5542c09342ca4aca90ffd51346751015653e315 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-a, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:21:38.049 INFO:teuthology.orchestra.run.smithi117.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-19T03:21:38.199 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 podman[31819]: 2024-03-19 03:21:37.94364404 +0000 UTC m=+0.366928807 container start e14c366900a5afcce7dc85a3c5542c09342ca4aca90ffd51346751015653e315 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-a, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T03:21:38.199 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: set uid:gid to 167:167 (ceph:ceph) 2024-03-19T03:21:38.199 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev), process ceph-mon, pid 2 2024-03-19T03:21:38.199 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: pidfile_write: ignore empty --pid-file 2024-03-19T03:21:38.199 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: load: jerasure load: lrc 2024-03-19T03:21:38.199 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: RocksDB version: 7.9.2 2024-03-19T03:21:38.199 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Git sha 0 2024-03-19T03:21:38.199 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Compile date 2024-03-18 18:57:12 2024-03-19T03:21:38.199 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: DB SUMMARY 2024-03-19T03:21:38.200 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: DB Session ID: 8YJ8W503GHWIBNW77DNJ 2024-03-19T03:21:38.200 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: CURRENT file: CURRENT 2024-03-19T03:21:38.200 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: IDENTITY file: IDENTITY 2024-03-19T03:21:38.200 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-19T03:21:38.200 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-19T03:21:38.200 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105239 ; 2024-03-19T03:21:38.200 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.error_if_exists: 0 2024-03-19T03:21:38.200 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.create_if_missing: 0 2024-03-19T03:21:38.200 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.paranoid_checks: 1 2024-03-19T03:21:38.200 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-19T03:21:38.200 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-19T03:21:38.200 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-19T03:21:38.200 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.env: 0x55fee14e8c60 2024-03-19T03:21:38.200 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.fs: PosixFileSystem 2024-03-19T03:21:38.200 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.info_log: 0x55fee31c3b20 2024-03-19T03:21:38.201 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_file_opening_threads: 16 2024-03-19T03:21:38.201 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.statistics: (nil) 2024-03-19T03:21:38.201 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.use_fsync: 0 2024-03-19T03:21:38.201 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_log_file_size: 0 2024-03-19T03:21:38.201 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-19T03:21:38.201 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-19T03:21:38.201 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.keep_log_file_num: 1000 2024-03-19T03:21:38.201 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.recycle_log_file_num: 0 2024-03-19T03:21:38.201 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.allow_fallocate: 1 2024-03-19T03:21:38.201 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.allow_mmap_reads: 0 2024-03-19T03:21:38.202 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.allow_mmap_writes: 0 2024-03-19T03:21:38.202 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.use_direct_reads: 0 2024-03-19T03:21:38.202 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-19T03:21:38.202 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.create_missing_column_families: 0 2024-03-19T03:21:38.202 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.db_log_dir: 2024-03-19T03:21:38.202 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.wal_dir: 2024-03-19T03:21:38.202 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-19T03:21:38.202 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-19T03:21:38.202 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-19T03:21:38.202 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-19T03:21:38.202 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-19T03:21:38.202 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-19T03:21:38.202 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.advise_random_on_open: 1 2024-03-19T03:21:38.202 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.db_write_buffer_size: 0 2024-03-19T03:21:38.203 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.write_buffer_manager: 0x55fee31961e0 2024-03-19T03:21:38.203 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-19T03:21:38.203 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-19T03:21:38.203 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-19T03:21:38.203 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.rate_limiter: (nil) 2024-03-19T03:21:38.203 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-19T03:21:38.203 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.wal_recovery_mode: 2 2024-03-19T03:21:38.203 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.enable_thread_tracking: 0 2024-03-19T03:21:38.203 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.enable_pipelined_write: 0 2024-03-19T03:21:38.203 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.unordered_write: 0 2024-03-19T03:21:38.203 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-19T03:21:38.203 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-19T03:21:38.203 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-19T03:21:38.203 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-19T03:21:38.203 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.row_cache: None 2024-03-19T03:21:38.204 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.wal_filter: None 2024-03-19T03:21:38.204 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-19T03:21:38.204 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.allow_ingest_behind: 0 2024-03-19T03:21:38.204 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.two_write_queues: 0 2024-03-19T03:21:38.204 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.manual_wal_flush: 0 2024-03-19T03:21:38.204 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.wal_compression: 0 2024-03-19T03:21:38.204 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.atomic_flush: 0 2024-03-19T03:21:38.204 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-19T03:21:38.204 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-19T03:21:38.204 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-19T03:21:38.205 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.log_readahead_size: 0 2024-03-19T03:21:38.205 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-19T03:21:38.205 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.best_efforts_recovery: 0 2024-03-19T03:21:38.205 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-19T03:21:38.205 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-19T03:21:38.205 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.allow_data_in_errors: 0 2024-03-19T03:21:38.205 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.db_host_id: __hostname__ 2024-03-19T03:21:38.205 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-19T03:21:38.205 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_background_jobs: 2 2024-03-19T03:21:38.205 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_background_compactions: -1 2024-03-19T03:21:38.205 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_subcompactions: 1 2024-03-19T03:21:38.206 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-19T03:21:38.206 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-19T03:21:38.206 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-19T03:21:38.206 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_total_wal_size: 0 2024-03-19T03:21:38.206 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-19T03:21:38.206 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-19T03:21:38.206 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-19T03:21:38.206 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-19T03:21:38.206 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_open_files: -1 2024-03-19T03:21:38.206 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.bytes_per_sync: 0 2024-03-19T03:21:38.206 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-19T03:21:38.206 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-19T03:21:38.206 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.compaction_readahead_size: 0 2024-03-19T03:21:38.206 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_background_flushes: -1 2024-03-19T03:21:38.206 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Compression algorithms supported: 2024-03-19T03:21:38.207 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-19T03:21:38.207 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: kZSTD supported: 0 2024-03-19T03:21:38.207 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: kXpressCompression supported: 0 2024-03-19T03:21:38.207 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: kLZ4HCCompression supported: 1 2024-03-19T03:21:38.207 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: kZlibCompression supported: 1 2024-03-19T03:21:38.207 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: kSnappyCompression supported: 1 2024-03-19T03:21:38.207 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: kLZ4Compression supported: 1 2024-03-19T03:21:38.207 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: kBZip2Compression supported: 0 2024-03-19T03:21:38.207 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-19T03:21:38.207 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-19T03:21:38.207 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-19T03:21:38.207 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-19T03:21:38.207 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-19T03:21:38.207 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.merge_operator: 2024-03-19T03:21:38.207 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.compaction_filter: None 2024-03-19T03:21:38.208 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.compaction_filter_factory: None 2024-03-19T03:21:38.208 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.sst_partitioner_factory: None 2024-03-19T03:21:38.208 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-19T03:21:38.208 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-19T03:21:38.208 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fee31c3c40) 2024-03-19T03:21:38.208 INFO:journalctl@ceph.mon.a.smithi117.stdout: cache_index_and_filter_blocks: 1 2024-03-19T03:21:38.208 INFO:journalctl@ceph.mon.a.smithi117.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-19T03:21:38.208 INFO:journalctl@ceph.mon.a.smithi117.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-19T03:21:38.208 INFO:journalctl@ceph.mon.a.smithi117.stdout: pin_top_level_index_and_filter: 1 2024-03-19T03:21:38.208 INFO:journalctl@ceph.mon.a.smithi117.stdout: index_type: 0 2024-03-19T03:21:38.208 INFO:journalctl@ceph.mon.a.smithi117.stdout: data_block_index_type: 0 2024-03-19T03:21:38.208 INFO:journalctl@ceph.mon.a.smithi117.stdout: index_shortening: 1 2024-03-19T03:21:38.208 INFO:journalctl@ceph.mon.a.smithi117.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-19T03:21:38.208 INFO:journalctl@ceph.mon.a.smithi117.stdout: checksum: 4 2024-03-19T03:21:38.209 INFO:journalctl@ceph.mon.a.smithi117.stdout: no_block_cache: 0 2024-03-19T03:21:38.209 INFO:journalctl@ceph.mon.a.smithi117.stdout: block_cache: 0x55fee31c9090 2024-03-19T03:21:38.209 INFO:journalctl@ceph.mon.a.smithi117.stdout: block_cache_name: BinnedLRUCache 2024-03-19T03:21:38.209 INFO:journalctl@ceph.mon.a.smithi117.stdout: block_cache_options: 2024-03-19T03:21:38.209 INFO:journalctl@ceph.mon.a.smithi117.stdout: capacity : 536870912 2024-03-19T03:21:38.209 INFO:journalctl@ceph.mon.a.smithi117.stdout: num_shard_bits : 4 2024-03-19T03:21:38.209 INFO:journalctl@ceph.mon.a.smithi117.stdout: strict_capacity_limit : 0 2024-03-19T03:21:38.209 INFO:journalctl@ceph.mon.a.smithi117.stdout: high_pri_pool_ratio: 0.000 2024-03-19T03:21:38.209 INFO:journalctl@ceph.mon.a.smithi117.stdout: block_cache_compressed: (nil) 2024-03-19T03:21:38.209 INFO:journalctl@ceph.mon.a.smithi117.stdout: persistent_cache: (nil) 2024-03-19T03:21:38.209 INFO:journalctl@ceph.mon.a.smithi117.stdout: block_size: 4096 2024-03-19T03:21:38.209 INFO:journalctl@ceph.mon.a.smithi117.stdout: block_size_deviation: 10 2024-03-19T03:21:38.209 INFO:journalctl@ceph.mon.a.smithi117.stdout: block_restart_interval: 16 2024-03-19T03:21:38.210 INFO:journalctl@ceph.mon.a.smithi117.stdout: index_block_restart_interval: 1 2024-03-19T03:21:38.210 INFO:journalctl@ceph.mon.a.smithi117.stdout: metadata_block_size: 4096 2024-03-19T03:21:38.210 INFO:journalctl@ceph.mon.a.smithi117.stdout: partition_filters: 0 2024-03-19T03:21:38.210 INFO:journalctl@ceph.mon.a.smithi117.stdout: use_delta_encoding: 1 2024-03-19T03:21:38.210 INFO:journalctl@ceph.mon.a.smithi117.stdout: filter_policy: bloomfilter 2024-03-19T03:21:38.210 INFO:journalctl@ceph.mon.a.smithi117.stdout: whole_key_filtering: 1 2024-03-19T03:21:38.210 INFO:journalctl@ceph.mon.a.smithi117.stdout: verify_compression: 0 2024-03-19T03:21:38.210 INFO:journalctl@ceph.mon.a.smithi117.stdout: read_amp_bytes_per_bit: 0 2024-03-19T03:21:38.210 INFO:journalctl@ceph.mon.a.smithi117.stdout: format_version: 5 2024-03-19T03:21:38.210 INFO:journalctl@ceph.mon.a.smithi117.stdout: enable_index_compression: 1 2024-03-19T03:21:38.210 INFO:journalctl@ceph.mon.a.smithi117.stdout: block_align: 0 2024-03-19T03:21:38.210 INFO:journalctl@ceph.mon.a.smithi117.stdout: max_auto_readahead_size: 262144 2024-03-19T03:21:38.210 INFO:journalctl@ceph.mon.a.smithi117.stdout: prepopulate_block_cache: 0 2024-03-19T03:21:38.210 INFO:journalctl@ceph.mon.a.smithi117.stdout: initial_auto_readahead_size: 8192 2024-03-19T03:21:38.210 INFO:journalctl@ceph.mon.a.smithi117.stdout: num_file_reads_for_auto_readahead: 2 2024-03-19T03:21:38.210 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.write_buffer_size: 33554432 2024-03-19T03:21:38.211 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_write_buffer_number: 2 2024-03-19T03:21:38.211 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.compression: NoCompression 2024-03-19T03:21:38.211 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.bottommost_compression: Disabled 2024-03-19T03:21:38.211 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.prefix_extractor: nullptr 2024-03-19T03:21:38.211 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-19T03:21:38.211 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.num_levels: 7 2024-03-19T03:21:38.211 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-19T03:21:38.211 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-19T03:21:38.211 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-19T03:21:38.211 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-19T03:21:38.211 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-19T03:21:38.211 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-19T03:21:38.211 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-19T03:21:38.211 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-19T03:21:38.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-19T03:21:38.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-19T03:21:38.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-19T03:21:38.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-19T03:21:38.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-19T03:21:38.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.compression_opts.level: 32767 2024-03-19T03:21:38.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.compression_opts.strategy: 0 2024-03-19T03:21:38.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-19T03:21:38.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-19T03:21:38.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-19T03:21:38.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-19T03:21:38.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.compression_opts.enabled: false 2024-03-19T03:21:38.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-19T03:21:38.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-19T03:21:38.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-19T03:21:38.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-19T03:21:38.213 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.target_file_size_base: 67108864 2024-03-19T03:21:38.213 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-19T03:21:38.213 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-19T03:21:38.213 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-19T03:21:38.213 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-19T03:21:38.213 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-19T03:21:38.213 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-19T03:21:38.213 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-19T03:21:38.213 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-19T03:21:38.213 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-19T03:21:38.213 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-19T03:21:38.213 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-19T03:21:38.213 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-19T03:21:38.213 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-19T03:21:38.213 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-19T03:21:38.214 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.arena_block_size: 1048576 2024-03-19T03:21:38.214 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-19T03:21:38.214 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-19T03:21:38.214 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.disable_auto_compactions: 0 2024-03-19T03:21:38.215 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-19T03:21:38.215 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-19T03:21:38.215 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-19T03:21:38.215 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-19T03:21:38.215 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-19T03:21:38.215 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-19T03:21:38.215 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-19T03:21:38.215 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-19T03:21:38.215 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-19T03:21:38.215 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-19T03:21:38.215 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.table_properties_collectors: 2024-03-19T03:21:38.215 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.inplace_update_support: 0 2024-03-19T03:21:38.215 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-19T03:21:38.215 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-19T03:21:38.215 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-19T03:21:38.216 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-19T03:21:38.216 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.bloom_locality: 0 2024-03-19T03:21:38.216 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.max_successive_merges: 0 2024-03-19T03:21:38.216 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-19T03:21:38.216 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.paranoid_file_checks: 0 2024-03-19T03:21:38.216 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:37 smithi117 ceph-mon[31838]: rocksdb: Options.force_consistency_checks: 1 2024-03-19T03:21:38.216 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: Options.report_bg_io_stats: 0 2024-03-19T03:21:38.216 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: Options.ttl: 2592000 2024-03-19T03:21:38.216 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-19T03:21:38.216 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-19T03:21:38.216 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-19T03:21:38.216 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: Options.enable_blob_files: false 2024-03-19T03:21:38.216 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: Options.min_blob_size: 0 2024-03-19T03:21:38.216 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: Options.blob_file_size: 268435456 2024-03-19T03:21:38.216 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-19T03:21:38.217 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-19T03:21:38.217 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-19T03:21:38.217 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-19T03:21:38.217 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-19T03:21:38.217 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: Options.blob_file_starting_level: 0 2024-03-19T03:21:38.217 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-19T03:21:38.217 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-19T03:21:38.217 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-19T03:21:38.217 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: dc3bc8a4-308b-46ea-a050-cb512ef54f27 2024-03-19T03:21:38.217 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710818497999117, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-19T03:21:38.217 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-19T03:21:38.217 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710818498001117, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100240, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 330, "table_properties": {"data_size": 98152, "index_size": 306, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14138, "raw_average_key_size": 49, "raw_value_size": 90452, "raw_average_value_size": 317, "num_data_blocks": 12, "num_entries": 285, "num_filter_entries": 285, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710818497, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "dc3bc8a4-308b-46ea-a050-cb512ef54f27", "db_session_id": "8YJ8W503GHWIBNW77DNJ", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-19T03:21:38.217 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710818498001297, "job": 1, "event": "recovery_finished"} 2024-03-19T03:21:38.217 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-19T03:21:38.217 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-19T03:21:38.218 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fee32bc000 2024-03-19T03:21:38.218 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: DB pointer 0x55fee32a6000 2024-03-19T03:21:38.218 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-19T03:21:38.218 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-19T03:21:38.218 INFO:journalctl@ceph.mon.a.smithi117.stdout: ** DB Stats ** 2024-03-19T03:21:38.218 INFO:journalctl@ceph.mon.a.smithi117.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-19T03:21:38.218 INFO:journalctl@ceph.mon.a.smithi117.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-19T03:21:38.218 INFO:journalctl@ceph.mon.a.smithi117.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-19T03:21:38.218 INFO:journalctl@ceph.mon.a.smithi117.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-19T03:21:38.218 INFO:journalctl@ceph.mon.a.smithi117.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-19T03:21:38.218 INFO:journalctl@ceph.mon.a.smithi117.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-19T03:21:38.218 INFO:journalctl@ceph.mon.a.smithi117.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-19T03:21:38.218 INFO:journalctl@ceph.mon.a.smithi117.stdout: 2024-03-19T03:21:38.218 INFO:journalctl@ceph.mon.a.smithi117.stdout: ** Compaction Stats [default] ** 2024-03-19T03:21:38.218 INFO:journalctl@ceph.mon.a.smithi117.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-19T03:21:38.219 INFO:journalctl@ceph.mon.a.smithi117.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-19T03:21:38.219 INFO:journalctl@ceph.mon.a.smithi117.stdout: L0 2/0 99.73 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 68.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-19T03:21:38.219 INFO:journalctl@ceph.mon.a.smithi117.stdout: Sum 2/0 99.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 68.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-19T03:21:38.219 INFO:journalctl@ceph.mon.a.smithi117.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 68.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-19T03:21:38.219 INFO:journalctl@ceph.mon.a.smithi117.stdout: 2024-03-19T03:21:38.219 INFO:journalctl@ceph.mon.a.smithi117.stdout: ** Compaction Stats [default] ** 2024-03-19T03:21:38.219 INFO:journalctl@ceph.mon.a.smithi117.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-19T03:21:38.219 INFO:journalctl@ceph.mon.a.smithi117.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-19T03:21:38.219 INFO:journalctl@ceph.mon.a.smithi117.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 68.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-19T03:21:38.219 INFO:journalctl@ceph.mon.a.smithi117.stdout: 2024-03-19T03:21:38.219 INFO:journalctl@ceph.mon.a.smithi117.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-19T03:21:38.220 INFO:journalctl@ceph.mon.a.smithi117.stdout: 2024-03-19T03:21:38.220 INFO:journalctl@ceph.mon.a.smithi117.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-19T03:21:38.220 INFO:journalctl@ceph.mon.a.smithi117.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-19T03:21:38.220 INFO:journalctl@ceph.mon.a.smithi117.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-19T03:21:38.220 INFO:journalctl@ceph.mon.a.smithi117.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-19T03:21:38.220 INFO:journalctl@ceph.mon.a.smithi117.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-19T03:21:38.220 INFO:journalctl@ceph.mon.a.smithi117.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-19T03:21:38.220 INFO:journalctl@ceph.mon.a.smithi117.stdout: Cumulative compaction: 0.00 GB write, 13.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-19T03:21:38.220 INFO:journalctl@ceph.mon.a.smithi117.stdout: Interval compaction: 0.00 GB write, 13.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-19T03:21:38.220 INFO:journalctl@ceph.mon.a.smithi117.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-19T03:21:38.221 INFO:journalctl@ceph.mon.a.smithi117.stdout: Block cache BinnedLRUCache@0x55fee31c9090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-03-19T03:21:38.221 INFO:journalctl@ceph.mon.a.smithi117.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-19T03:21:38.221 INFO:journalctl@ceph.mon.a.smithi117.stdout: 2024-03-19T03:21:38.221 INFO:journalctl@ceph.mon.a.smithi117.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-19T03:21:38.221 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: starting mon.a rank 0 at public addrs [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] at bind addrs [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:21:38.221 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: mon.a@-1(???) e1 preinit fsid a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:21:38.221 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: mon.a@-1(???).mds e1 new map 2024-03-19T03:21:38.221 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: mon.a@-1(???).mds e1 print_map 2024-03-19T03:21:38.221 INFO:journalctl@ceph.mon.a.smithi117.stdout: e1 2024-03-19T03:21:38.221 INFO:journalctl@ceph.mon.a.smithi117.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-19T03:21:38.221 INFO:journalctl@ceph.mon.a.smithi117.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-19T03:21:38.221 INFO:journalctl@ceph.mon.a.smithi117.stdout: legacy client fscid: -1 2024-03-19T03:21:38.221 INFO:journalctl@ceph.mon.a.smithi117.stdout: 2024-03-19T03:21:38.221 INFO:journalctl@ceph.mon.a.smithi117.stdout: No filesystems configured 2024-03-19T03:21:38.221 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-19T03:21:38.222 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T03:21:38.222 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T03:21:38.222 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T03:21:38.222 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-19T03:21:38.222 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 bash[31819]: e14c366900a5afcce7dc85a3c5542c09342ca4aca90ffd51346751015653e315 2024-03-19T03:21:38.222 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 systemd[1]: Started Ceph mon.a for a3a82456-e59f-11ee-95c9-87774f69a715. 2024-03-19T03:21:38.222 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-19T03:21:38.222 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: monmap epoch 1 2024-03-19T03:21:38.222 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: fsid a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:21:38.222 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: last_changed 2024-03-19T03:21:26.225562+0000 2024-03-19T03:21:38.222 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: created 2024-03-19T03:21:26.225562+0000 2024-03-19T03:21:38.222 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: min_mon_release 19 (squid) 2024-03-19T03:21:38.222 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: election_strategy: 1 2024-03-19T03:21:38.222 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: 0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-03-19T03:21:38.222 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: fsmap 2024-03-19T03:21:38.223 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: osdmap e1: 0 total, 0 up, 0 in 2024-03-19T03:21:38.223 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:38 smithi117 ceph-mon[31838]: mgrmap e1: no daemons active 2024-03-19T03:21:39.405 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.709+0000 7f1453600700 1 Processor -- start 2024-03-19T03:21:39.405 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.709+0000 7f1453600700 1 -- start start 2024-03-19T03:21:39.406 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.709+0000 7f1453600700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f144c106c20 0x7f144c107040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:39.406 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.709+0000 7f1453600700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f144c107610 con 0x7f144c106c20 2024-03-19T03:21:39.406 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.710+0000 7f1450c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f144c106c20 0x7f144c107040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:21:39.406 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.710+0000 7f1450c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f144c106c20 0x7f144c107040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:58526/0 (socket says 172.21.15.117:58526) 2024-03-19T03:21:39.407 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.710+0000 7f1450c00700 1 -- 172.21.15.117:0/1741120745 learned_addr learned my addr 172.21.15.117:0/1741120745 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:21:39.407 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.710+0000 7f1450c00700 1 -- 172.21.15.117:0/1741120745 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f144c107e20 con 0x7f144c106c20 2024-03-19T03:21:39.407 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.711+0000 7f1450c00700 1 --2- 172.21.15.117:0/1741120745 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f144c106c20 0x7f144c107040 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f14400098a0 tx=0x7f1440009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b57e7dda3106058 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:21:39.408 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.711+0000 7f144b400700 1 -- 172.21.15.117:0/1741120745 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1440004160 con 0x7f144c106c20 2024-03-19T03:21:39.408 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.711+0000 7f144b400700 1 -- 172.21.15.117:0/1741120745 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1440004300 con 0x7f144c106c20 2024-03-19T03:21:39.408 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.711+0000 7f144b400700 1 -- 172.21.15.117:0/1741120745 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1440011b50 con 0x7f144c106c20 2024-03-19T03:21:39.408 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.712+0000 7f1453600700 1 -- 172.21.15.117:0/1741120745 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f144c106c20 msgr2=0x7f144c107040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:21:39.408 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.712+0000 7f1453600700 1 --2- 172.21.15.117:0/1741120745 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f144c106c20 0x7f144c107040 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f14400098a0 tx=0x7f1440009bb0 comp rx=0 tx=0).stop 2024-03-19T03:21:39.408 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.712+0000 7f1453600700 1 -- 172.21.15.117:0/1741120745 shutdown_connections 2024-03-19T03:21:39.408 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.712+0000 7f1453600700 1 --2- 172.21.15.117:0/1741120745 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f144c106c20 0x7f144c107040 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:39.408 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.712+0000 7f1453600700 1 -- 172.21.15.117:0/1741120745 >> 172.21.15.117:0/1741120745 conn(0x7f144c1021c0 msgr2=0x7f144c104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:21:39.409 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.712+0000 7f1453600700 1 -- 172.21.15.117:0/1741120745 shutdown_connections 2024-03-19T03:21:39.409 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.712+0000 7f1453600700 1 -- 172.21.15.117:0/1741120745 wait complete. 2024-03-19T03:21:39.410 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.713+0000 7f1453600700 1 Processor -- start 2024-03-19T03:21:39.410 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.713+0000 7f1453600700 1 -- start start 2024-03-19T03:21:39.410 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.713+0000 7f1453600700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f144c106c20 0x7f144c19ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:39.410 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.713+0000 7f1453600700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f144c07af60 con 0x7f144c106c20 2024-03-19T03:21:39.410 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.714+0000 7f1450c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f144c106c20 0x7f144c19ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:21:39.410 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.714+0000 7f1450c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f144c106c20 0x7f144c19ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:58538/0 (socket says 172.21.15.117:58538) 2024-03-19T03:21:39.410 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.714+0000 7f1450c00700 1 -- 172.21.15.117:0/535010063 learned_addr learned my addr 172.21.15.117:0/535010063 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:21:39.410 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.714+0000 7f1450c00700 1 -- 172.21.15.117:0/535010063 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f144c19e410 con 0x7f144c106c20 2024-03-19T03:21:39.412 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.714+0000 7f1450c00700 1 --2- 172.21.15.117:0/535010063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f144c106c20 0x7f144c19ded0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1440000c00 tx=0x7f1440011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:21:39.412 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.715+0000 7f1449600700 1 -- 172.21.15.117:0/535010063 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1440003b20 con 0x7f144c106c20 2024-03-19T03:21:39.412 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.715+0000 7f1453600700 1 -- 172.21.15.117:0/535010063 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f144c19e6a0 con 0x7f144c106c20 2024-03-19T03:21:39.412 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.715+0000 7f1449600700 1 -- 172.21.15.117:0/535010063 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1440003cc0 con 0x7f144c106c20 2024-03-19T03:21:39.412 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.715+0000 7f1449600700 1 -- 172.21.15.117:0/535010063 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1440024910 con 0x7f144c106c20 2024-03-19T03:21:39.412 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.715+0000 7f1453600700 1 -- 172.21.15.117:0/535010063 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f144c1a1390 con 0x7f144c106c20 2024-03-19T03:21:39.412 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.715+0000 7f1449600700 1 -- 172.21.15.117:0/535010063 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1440022020 con 0x7f144c106c20 2024-03-19T03:21:39.412 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.715+0000 7f1449600700 1 -- 172.21.15.117:0/535010063 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1440011e00 con 0x7f144c106c20 2024-03-19T03:21:39.412 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.716+0000 7f1453600700 1 -- 172.21.15.117:0/535010063 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f144c063940 con 0x7f144c106c20 2024-03-19T03:21:39.412 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.719+0000 7f1449600700 1 -- 172.21.15.117:0/535010063 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f1440042dd0 con 0x7f144c106c20 2024-03-19T03:21:39.412 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.795+0000 7f1453600700 1 -- 172.21.15.117:0/535010063 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f144c197430 con 0x7f144c106c20 2024-03-19T03:21:39.412 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.798+0000 7f1449600700 1 -- 172.21.15.117:0/535010063 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f144c197430 con 0x7f144c106c20 2024-03-19T03:21:39.413 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.800+0000 7f1453600700 1 -- 172.21.15.117:0/535010063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f144c106c20 msgr2=0x7f144c19ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:21:39.413 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.800+0000 7f1453600700 1 --2- 172.21.15.117:0/535010063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f144c106c20 0x7f144c19ded0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1440000c00 tx=0x7f1440011c10 comp rx=0 tx=0).stop 2024-03-19T03:21:39.413 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.800+0000 7f1453600700 1 -- 172.21.15.117:0/535010063 shutdown_connections 2024-03-19T03:21:39.413 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.800+0000 7f1453600700 1 --2- 172.21.15.117:0/535010063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f144c106c20 0x7f144c19ded0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:39.413 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.800+0000 7f1453600700 1 -- 172.21.15.117:0/535010063 >> 172.21.15.117:0/535010063 conn(0x7f144c1021c0 msgr2=0x7f144c194510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:21:39.413 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.800+0000 7f1453600700 1 -- 172.21.15.117:0/535010063 shutdown_connections 2024-03-19T03:21:39.414 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:38.800+0000 7f1453600700 1 -- 172.21.15.117:0/535010063 wait complete. 2024-03-19T03:21:39.414 INFO:teuthology.orchestra.run.smithi117.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-19T03:21:39.430 INFO:teuthology.orchestra.run.smithi117.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-19T03:21:39.430 INFO:teuthology.orchestra.run.smithi117.stdout:Creating mgr... 2024-03-19T03:21:39.432 INFO:teuthology.orchestra.run.smithi117.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-19T03:21:39.433 INFO:teuthology.orchestra.run.smithi117.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-19T03:21:39.870 INFO:teuthology.orchestra.run.smithi117.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mgr.x 2024-03-19T03:21:39.870 INFO:teuthology.orchestra.run.smithi117.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mgr.x.service: Unit ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mgr.x.service not loaded. 2024-03-19T03:21:40.133 INFO:teuthology.orchestra.run.smithi117.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a3a82456-e59f-11ee-95c9-87774f69a715.target.wants/ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-a3a82456-e59f-11ee-95c9-87774f69a715@.service. 2024-03-19T03:21:40.164 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:39 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/535010063' entity='client.admin' 2024-03-19T03:21:40.508 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:40 smithi117 systemd[1]: Starting Ceph mgr.x for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:21:40.792 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:40 smithi117 podman[32049]: 2024-03-19 03:21:40.507469722 +0000 UTC m=+0.119292563 container create fb84b6bcbbed166d71066578e1d5c4bcb6e842ee31c071ca68a87523d9e76dc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-19T03:21:40.792 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:40 smithi117 podman[32049]: 2024-03-19 03:21:40.420173957 +0000 UTC m=+0.031996789 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T03:21:40.792 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:40 smithi117 podman[32049]: 2024-03-19 03:21:40.718985436 +0000 UTC m=+0.330808311 container init fb84b6bcbbed166d71066578e1d5c4bcb6e842ee31c071ca68a87523d9e76dc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:21:40.792 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:40 smithi117 podman[32049]: 2024-03-19 03:21:40.728729537 +0000 UTC m=+0.340552411 container start fb84b6bcbbed166d71066578e1d5c4bcb6e842ee31c071ca68a87523d9e76dc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-19T03:21:40.838 INFO:teuthology.orchestra.run.smithi117.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-19T03:21:40.838 INFO:teuthology.orchestra.run.smithi117.stdout:systemctl: stdout disabled 2024-03-19T03:21:40.854 INFO:teuthology.orchestra.run.smithi117.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-19T03:21:40.854 INFO:teuthology.orchestra.run.smithi117.stdout:systemctl: stdout inactive 2024-03-19T03:21:40.854 INFO:teuthology.orchestra.run.smithi117.stdout:firewalld.service is not enabled 2024-03-19T03:21:40.854 INFO:teuthology.orchestra.run.smithi117.stdout:Not possible to enable service . firewalld.service is not available 2024-03-19T03:21:40.875 INFO:teuthology.orchestra.run.smithi117.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-19T03:21:40.876 INFO:teuthology.orchestra.run.smithi117.stdout:systemctl: stdout disabled 2024-03-19T03:21:40.897 INFO:teuthology.orchestra.run.smithi117.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-19T03:21:40.898 INFO:teuthology.orchestra.run.smithi117.stdout:systemctl: stdout inactive 2024-03-19T03:21:40.898 INFO:teuthology.orchestra.run.smithi117.stdout:firewalld.service is not enabled 2024-03-19T03:21:40.898 INFO:teuthology.orchestra.run.smithi117.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-19T03:21:40.899 INFO:teuthology.orchestra.run.smithi117.stdout:Waiting for mgr to start... 2024-03-19T03:21:40.899 INFO:teuthology.orchestra.run.smithi117.stdout:Waiting for mgr... 2024-03-19T03:21:41.103 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:40 smithi117 bash[32049]: fb84b6bcbbed166d71066578e1d5c4bcb6e842ee31c071ca68a87523d9e76dc8 2024-03-19T03:21:41.103 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:40 smithi117 systemd[1]: Started Ceph mgr.x for a3a82456-e59f-11ee-95c9-87774f69a715. 2024-03-19T03:21:41.633 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:41 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:41.509+0000 7f6bb0965200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T03:21:41.908 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:41 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/478957938' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-19T03:21:41.908 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:41 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:41.632+0000 7f6bb0965200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T03:21:42.285 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout 2024-03-19T03:21:42.288 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout { 2024-03-19T03:21:42.288 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "fsid": "a3a82456-e59f-11ee-95c9-87774f69a715", 2024-03-19T03:21:42.289 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "health": { 2024-03-19T03:21:42.289 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-19T03:21:42.289 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-19T03:21:42.289 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-19T03:21:42.289 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:42.289 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-19T03:21:42.289 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-19T03:21:42.289 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout 0 2024-03-19T03:21:42.289 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout ], 2024-03-19T03:21:42.289 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-19T03:21:42.289 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "a" 2024-03-19T03:21:42.289 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout ], 2024-03-19T03:21:42.290 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-19T03:21:42.290 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-19T03:21:42.290 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:21:42.290 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-19T03:21:42.290 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-19T03:21:42.290 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:42.290 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-19T03:21:42.290 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:21:42.290 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-19T03:21:42.290 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-19T03:21:42.290 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-19T03:21:42.299 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-19T03:21:42.299 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-19T03:21:42.299 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-19T03:21:42.299 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:42.299 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-19T03:21:42.299 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-19T03:21:42.299 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-19T03:21:42.299 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-19T03:21:42.299 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-19T03:21:42.299 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-19T03:21:42.299 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-19T03:21:42.300 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-19T03:21:42.300 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-19T03:21:42.300 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:42.300 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-19T03:21:42.300 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:21:42.300 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-19T03:21:42.300 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-19T03:21:42.300 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:42.300 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-19T03:21:42.300 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-19T03:21:42.300 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-19T03:21:42.300 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-19T03:21:42.300 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-19T03:21:42.300 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-19T03:21:42.301 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "restful" 2024-03-19T03:21:42.301 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout ], 2024-03-19T03:21:42.301 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T03:21:42.301 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:42.301 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-19T03:21:42.301 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:21:42.301 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "modified": "2024-03-19T03:21:30.427711+0000", 2024-03-19T03:21:42.301 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T03:21:42.301 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:42.301 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-19T03:21:42.301 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout } 2024-03-19T03:21:42.301 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.498+0000 7f75da800700 1 Processor -- start 2024-03-19T03:21:42.301 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.498+0000 7f75da800700 1 -- start start 2024-03-19T03:21:42.301 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.499+0000 7f75da800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75d407dd70 0x7f75d407c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:42.301 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.499+0000 7f75da800700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f75d407c7b0 con 0x7f75d407dd70 2024-03-19T03:21:42.302 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.499+0000 7f75d3400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75d407dd70 0x7f75d407c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:21:42.302 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.499+0000 7f75d3400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75d407dd70 0x7f75d407c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:58556/0 (socket says 172.21.15.117:58556) 2024-03-19T03:21:42.302 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.499+0000 7f75d3400700 1 -- 172.21.15.117:0/1343694435 learned_addr learned my addr 172.21.15.117:0/1343694435 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:21:42.302 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.500+0000 7f75d3400700 1 -- 172.21.15.117:0/1343694435 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75d407c930 con 0x7f75d407dd70 2024-03-19T03:21:42.302 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.500+0000 7f75d3400700 1 --2- 172.21.15.117:0/1343694435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75d407dd70 0x7f75d407c270 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f75c40098a0 tx=0x7f75c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7343ba4afa175cf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:21:42.302 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.500+0000 7f75d2000700 1 -- 172.21.15.117:0/1343694435 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75c4004160 con 0x7f75d407dd70 2024-03-19T03:21:42.302 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.500+0000 7f75d2000700 1 -- 172.21.15.117:0/1343694435 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f75c4004300 con 0x7f75d407dd70 2024-03-19T03:21:42.302 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.500+0000 7f75d2000700 1 -- 172.21.15.117:0/1343694435 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75c4011b50 con 0x7f75d407dd70 2024-03-19T03:21:42.302 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.501+0000 7f75da800700 1 -- 172.21.15.117:0/1343694435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75d407dd70 msgr2=0x7f75d407c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:21:42.302 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.501+0000 7f75da800700 1 --2- 172.21.15.117:0/1343694435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75d407dd70 0x7f75d407c270 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f75c40098a0 tx=0x7f75c4009bb0 comp rx=0 tx=0).stop 2024-03-19T03:21:42.302 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.501+0000 7f75da800700 1 -- 172.21.15.117:0/1343694435 shutdown_connections 2024-03-19T03:21:42.302 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.501+0000 7f75da800700 1 --2- 172.21.15.117:0/1343694435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75d407dd70 0x7f75d407c270 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:42.302 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.501+0000 7f75da800700 1 -- 172.21.15.117:0/1343694435 >> 172.21.15.117:0/1343694435 conn(0x7f75d4103310 msgr2=0x7f75d4105750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:21:42.303 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.502+0000 7f75da800700 1 -- 172.21.15.117:0/1343694435 shutdown_connections 2024-03-19T03:21:42.303 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.502+0000 7f75da800700 1 -- 172.21.15.117:0/1343694435 wait complete. 2024-03-19T03:21:42.303 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.503+0000 7f75da800700 1 Processor -- start 2024-03-19T03:21:42.303 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.503+0000 7f75da800700 1 -- start start 2024-03-19T03:21:42.303 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.503+0000 7f75da800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75d407dd70 0x7f75d419de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:42.303 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.503+0000 7f75da800700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f75d410e220 con 0x7f75d407dd70 2024-03-19T03:21:42.303 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.504+0000 7f75d3400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75d407dd70 0x7f75d419de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:21:42.303 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.504+0000 7f75d3400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75d407dd70 0x7f75d419de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:58560/0 (socket says 172.21.15.117:58560) 2024-03-19T03:21:42.303 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.504+0000 7f75d3400700 1 -- 172.21.15.117:0/478957938 learned_addr learned my addr 172.21.15.117:0/478957938 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:21:42.303 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.504+0000 7f75d3400700 1 -- 172.21.15.117:0/478957938 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75d419e340 con 0x7f75d407dd70 2024-03-19T03:21:42.303 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.504+0000 7f75d3400700 1 --2- 172.21.15.117:0/478957938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75d407dd70 0x7f75d419de00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f75c4011b60 tx=0x7f75c4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:21:42.303 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.505+0000 7f75cbe00700 1 -- 172.21.15.117:0/478957938 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75c4003b20 con 0x7f75d407dd70 2024-03-19T03:21:42.303 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.505+0000 7f75cbe00700 1 -- 172.21.15.117:0/478957938 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f75c4003cc0 con 0x7f75d407dd70 2024-03-19T03:21:42.303 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.505+0000 7f75cbe00700 1 -- 172.21.15.117:0/478957938 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75c4024840 con 0x7f75d407dd70 2024-03-19T03:21:42.304 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.505+0000 7f75da800700 1 -- 172.21.15.117:0/478957938 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75d419e5d0 con 0x7f75d407dd70 2024-03-19T03:21:42.304 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.505+0000 7f75da800700 1 -- 172.21.15.117:0/478957938 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75d41a12c0 con 0x7f75d407dd70 2024-03-19T03:21:42.304 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.505+0000 7f75cbe00700 1 -- 172.21.15.117:0/478957938 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f75c4022020 con 0x7f75d407dd70 2024-03-19T03:21:42.304 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.506+0000 7f75cbe00700 1 -- 172.21.15.117:0/478957938 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f75c4011e00 con 0x7f75d407dd70 2024-03-19T03:21:42.304 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.506+0000 7f75da800700 1 -- 172.21.15.117:0/478957938 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75d4063940 con 0x7f75d407dd70 2024-03-19T03:21:42.304 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.509+0000 7f75cbe00700 1 -- 172.21.15.117:0/478957938 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f75c401a430 con 0x7f75d407dd70 2024-03-19T03:21:42.304 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.588+0000 7f75da800700 1 -- 172.21.15.117:0/478957938 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f75d41a1530 con 0x7f75d407dd70 2024-03-19T03:21:42.304 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.589+0000 7f75cbe00700 1 -- 172.21.15.117:0/478957938 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f75c401aa60 con 0x7f75d407dd70 2024-03-19T03:21:42.304 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.590+0000 7f75da800700 1 -- 172.21.15.117:0/478957938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75d407dd70 msgr2=0x7f75d419de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:21:42.304 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.590+0000 7f75da800700 1 --2- 172.21.15.117:0/478957938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75d407dd70 0x7f75d419de00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f75c4011b60 tx=0x7f75c4011c40 comp rx=0 tx=0).stop 2024-03-19T03:21:42.304 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.590+0000 7f75da800700 1 -- 172.21.15.117:0/478957938 shutdown_connections 2024-03-19T03:21:42.304 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.590+0000 7f75da800700 1 --2- 172.21.15.117:0/478957938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75d407dd70 0x7f75d419de00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:42.304 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.590+0000 7f75da800700 1 -- 172.21.15.117:0/478957938 >> 172.21.15.117:0/478957938 conn(0x7f75d4103310 msgr2=0x7f75d4103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:21:42.304 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.591+0000 7f75da800700 1 -- 172.21.15.117:0/478957938 shutdown_connections 2024-03-19T03:21:42.305 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:41.591+0000 7f75da800700 1 -- 172.21.15.117:0/478957938 wait complete. 2024-03-19T03:21:42.305 INFO:teuthology.orchestra.run.smithi117.stdout:mgr not available, waiting (1/15)... 2024-03-19T03:21:42.542 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:42 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:42.443+0000 7f6bb0965200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T03:21:42.860 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:42 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:42.563+0000 7f6bb0965200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T03:21:42.861 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:42 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:42.738+0000 7f6bb0965200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T03:21:42.861 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:42 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:42.860+0000 7f6bb0965200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T03:21:43.291 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:42 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:42.966+0000 7f6bb0965200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T03:21:43.664 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:43 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:43.293+0000 7f6bb0965200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T03:21:43.664 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:43 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:43.431+0000 7f6bb0965200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T03:21:43.664 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:43 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:43.542+0000 7f6bb0965200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T03:21:44.042 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:43 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:43.663+0000 7f6bb0965200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T03:21:44.042 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:43 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:43.875+0000 7f6bb0965200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T03:21:45.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:45 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/936950957' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-19T03:21:45.292 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:45 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:45.195+0000 7f6bb0965200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T03:21:45.705 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:45 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:45.367+0000 7f6bb0965200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T03:21:45.715 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout 2024-03-19T03:21:45.716 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout { 2024-03-19T03:21:45.717 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "fsid": "a3a82456-e59f-11ee-95c9-87774f69a715", 2024-03-19T03:21:45.717 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "health": { 2024-03-19T03:21:45.717 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-19T03:21:45.717 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-19T03:21:45.717 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-19T03:21:45.717 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:45.717 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-19T03:21:45.717 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-19T03:21:45.717 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout 0 2024-03-19T03:21:45.717 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout ], 2024-03-19T03:21:45.719 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-19T03:21:45.719 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "a" 2024-03-19T03:21:45.719 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout ], 2024-03-19T03:21:45.719 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-19T03:21:45.719 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-19T03:21:45.720 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:21:45.720 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-19T03:21:45.720 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-19T03:21:45.720 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:45.720 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-19T03:21:45.720 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:21:45.720 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-19T03:21:45.720 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-19T03:21:45.720 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-19T03:21:45.730 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-19T03:21:45.730 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-19T03:21:45.730 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-19T03:21:45.730 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:45.730 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-19T03:21:45.730 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-19T03:21:45.730 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-19T03:21:45.730 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-19T03:21:45.730 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-19T03:21:45.730 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-19T03:21:45.731 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-19T03:21:45.731 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-19T03:21:45.731 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-19T03:21:45.731 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:45.731 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-19T03:21:45.731 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:21:45.731 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-19T03:21:45.731 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-19T03:21:45.731 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:45.731 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-19T03:21:45.731 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-19T03:21:45.731 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-19T03:21:45.732 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-19T03:21:45.732 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-19T03:21:45.732 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-19T03:21:45.732 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "restful" 2024-03-19T03:21:45.732 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout ], 2024-03-19T03:21:45.732 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T03:21:45.732 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:45.732 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-19T03:21:45.732 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:21:45.732 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "modified": "2024-03-19T03:21:30.427711+0000", 2024-03-19T03:21:45.732 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T03:21:45.732 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:45.732 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-19T03:21:45.733 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout } 2024-03-19T03:21:45.733 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.871+0000 7f4e75a00700 1 Processor -- start 2024-03-19T03:21:45.733 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.871+0000 7f4e75a00700 1 -- start start 2024-03-19T03:21:45.733 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.871+0000 7f4e75a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e7007de70 0x7f4e7007c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:45.733 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.871+0000 7f4e75a00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7007c8a0 con 0x7f4e7007de70 2024-03-19T03:21:45.733 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.872+0000 7f4e6fe00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e7007de70 0x7f4e7007c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:21:45.733 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.872+0000 7f4e6fe00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e7007de70 0x7f4e7007c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:58572/0 (socket says 172.21.15.117:58572) 2024-03-19T03:21:45.733 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.872+0000 7f4e6fe00700 1 -- 172.21.15.117:0/2068718987 learned_addr learned my addr 172.21.15.117:0/2068718987 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:21:45.733 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.872+0000 7f4e6fe00700 1 -- 172.21.15.117:0/2068718987 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e7007ca20 con 0x7f4e7007de70 2024-03-19T03:21:45.733 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.873+0000 7f4e6fe00700 1 --2- 172.21.15.117:0/2068718987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e7007de70 0x7f4e7007c2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4e600098a0 tx=0x7f4e60009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15c32937288166df server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:21:45.733 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.873+0000 7f4e6ea00700 1 -- 172.21.15.117:0/2068718987 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e60004160 con 0x7f4e7007de70 2024-03-19T03:21:45.733 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.873+0000 7f4e6ea00700 1 -- 172.21.15.117:0/2068718987 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4e60004300 con 0x7f4e7007de70 2024-03-19T03:21:45.734 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.873+0000 7f4e6ea00700 1 -- 172.21.15.117:0/2068718987 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e60011b50 con 0x7f4e7007de70 2024-03-19T03:21:45.734 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.874+0000 7f4e75a00700 1 -- 172.21.15.117:0/2068718987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e7007de70 msgr2=0x7f4e7007c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:21:45.734 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.874+0000 7f4e75a00700 1 --2- 172.21.15.117:0/2068718987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e7007de70 0x7f4e7007c2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4e600098a0 tx=0x7f4e60009bb0 comp rx=0 tx=0).stop 2024-03-19T03:21:45.734 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.874+0000 7f4e75a00700 1 -- 172.21.15.117:0/2068718987 shutdown_connections 2024-03-19T03:21:45.734 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.874+0000 7f4e75a00700 1 --2- 172.21.15.117:0/2068718987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e7007de70 0x7f4e7007c2d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:45.734 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.874+0000 7f4e75a00700 1 -- 172.21.15.117:0/2068718987 >> 172.21.15.117:0/2068718987 conn(0x7f4e70103080 msgr2=0x7f4e70105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:21:45.734 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.874+0000 7f4e75a00700 1 -- 172.21.15.117:0/2068718987 shutdown_connections 2024-03-19T03:21:45.734 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.874+0000 7f4e75a00700 1 -- 172.21.15.117:0/2068718987 wait complete. 2024-03-19T03:21:45.734 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.875+0000 7f4e75a00700 1 Processor -- start 2024-03-19T03:21:45.734 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.875+0000 7f4e75a00700 1 -- start start 2024-03-19T03:21:45.734 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.876+0000 7f4e75a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e7007de70 0x7f4e7019dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:45.735 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.876+0000 7f4e75a00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7010e0e0 con 0x7f4e7007de70 2024-03-19T03:21:45.735 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.876+0000 7f4e6fe00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e7007de70 0x7f4e7019dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:21:45.735 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.876+0000 7f4e6fe00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e7007de70 0x7f4e7019dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:58576/0 (socket says 172.21.15.117:58576) 2024-03-19T03:21:45.735 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.876+0000 7f4e6fe00700 1 -- 172.21.15.117:0/936950957 learned_addr learned my addr 172.21.15.117:0/936950957 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:21:45.735 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.876+0000 7f4e6fe00700 1 -- 172.21.15.117:0/936950957 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e7019e240 con 0x7f4e7007de70 2024-03-19T03:21:45.735 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.877+0000 7f4e6fe00700 1 --2- 172.21.15.117:0/936950957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e7007de70 0x7f4e7019dd00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4e60000c00 tx=0x7f4e60011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:21:45.735 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.877+0000 7f4e6cc00700 1 -- 172.21.15.117:0/936950957 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e60003b20 con 0x7f4e7007de70 2024-03-19T03:21:45.735 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.877+0000 7f4e6cc00700 1 -- 172.21.15.117:0/936950957 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4e60003cc0 con 0x7f4e7007de70 2024-03-19T03:21:45.735 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.877+0000 7f4e75a00700 1 -- 172.21.15.117:0/936950957 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e7019e4d0 con 0x7f4e7007de70 2024-03-19T03:21:45.735 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.877+0000 7f4e6cc00700 1 -- 172.21.15.117:0/936950957 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e600248a0 con 0x7f4e7007de70 2024-03-19T03:21:45.735 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.878+0000 7f4e6cc00700 1 -- 172.21.15.117:0/936950957 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4e60022020 con 0x7f4e7007de70 2024-03-19T03:21:45.735 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.878+0000 7f4e75a00700 1 -- 172.21.15.117:0/936950957 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e701a11b0 con 0x7f4e7007de70 2024-03-19T03:21:45.736 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.878+0000 7f4e6cc00700 1 -- 172.21.15.117:0/936950957 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4e60011e00 con 0x7f4e7007de70 2024-03-19T03:21:45.736 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.879+0000 7f4e75a00700 1 -- 172.21.15.117:0/936950957 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e70063940 con 0x7f4e7007de70 2024-03-19T03:21:45.736 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.881+0000 7f4e6cc00700 1 -- 172.21.15.117:0/936950957 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4e6001ac10 con 0x7f4e7007de70 2024-03-19T03:21:45.736 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.957+0000 7f4e75a00700 1 -- 172.21.15.117:0/936950957 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f4e701a1380 con 0x7f4e7007de70 2024-03-19T03:21:45.736 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.958+0000 7f4e6cc00700 1 -- 172.21.15.117:0/936950957 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f4e6001a430 con 0x7f4e7007de70 2024-03-19T03:21:45.736 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.959+0000 7f4e75a00700 1 -- 172.21.15.117:0/936950957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e7007de70 msgr2=0x7f4e7019dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:21:45.736 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.959+0000 7f4e75a00700 1 --2- 172.21.15.117:0/936950957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e7007de70 0x7f4e7019dd00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4e60000c00 tx=0x7f4e60011c10 comp rx=0 tx=0).stop 2024-03-19T03:21:45.736 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.959+0000 7f4e75a00700 1 -- 172.21.15.117:0/936950957 shutdown_connections 2024-03-19T03:21:45.736 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.959+0000 7f4e75a00700 1 --2- 172.21.15.117:0/936950957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e7007de70 0x7f4e7019dd00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:45.736 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.959+0000 7f4e75a00700 1 -- 172.21.15.117:0/936950957 >> 172.21.15.117:0/936950957 conn(0x7f4e70103080 msgr2=0x7f4e70103ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:21:45.736 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.959+0000 7f4e75a00700 1 -- 172.21.15.117:0/936950957 shutdown_connections 2024-03-19T03:21:45.736 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:44.959+0000 7f4e75a00700 1 -- 172.21.15.117:0/936950957 wait complete. 2024-03-19T03:21:45.736 INFO:teuthology.orchestra.run.smithi117.stdout:mgr not available, waiting (2/15)... 2024-03-19T03:21:46.042 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:45 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:45.786+0000 7f6bb0965200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T03:21:46.042 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:45 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:45.901+0000 7f6bb0965200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T03:21:46.542 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:46 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:46.153+0000 7f6bb0965200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T03:21:46.542 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:46 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:46.393+0000 7f6bb0965200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T03:21:47.041 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:46 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:46.755+0000 7f6bb0965200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T03:21:47.373 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:47 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:47.066+0000 7f6bb0965200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T03:21:48.041 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:47 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:47.372+0000 7f6bb0965200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T03:21:48.745 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:48 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1607737686' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-19T03:21:48.745 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:48 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:48.421+0000 7f6bb0965200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T03:21:49.072 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout 2024-03-19T03:21:49.072 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout { 2024-03-19T03:21:49.072 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "fsid": "a3a82456-e59f-11ee-95c9-87774f69a715", 2024-03-19T03:21:49.072 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "health": { 2024-03-19T03:21:49.072 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-19T03:21:49.072 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-19T03:21:49.072 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-19T03:21:49.072 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:49.072 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-19T03:21:49.072 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-19T03:21:49.073 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout 0 2024-03-19T03:21:49.073 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout ], 2024-03-19T03:21:49.073 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-19T03:21:49.073 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "a" 2024-03-19T03:21:49.073 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout ], 2024-03-19T03:21:49.073 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-19T03:21:49.073 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-19T03:21:49.073 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:21:49.073 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-19T03:21:49.073 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-19T03:21:49.073 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:49.073 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-19T03:21:49.073 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:21:49.073 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-19T03:21:49.073 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-19T03:21:49.074 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-19T03:21:49.081 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-19T03:21:49.081 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-19T03:21:49.082 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-19T03:21:49.082 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:49.082 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-19T03:21:49.082 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-19T03:21:49.082 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-19T03:21:49.082 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-19T03:21:49.082 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-19T03:21:49.082 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-19T03:21:49.082 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-19T03:21:49.082 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-19T03:21:49.082 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-19T03:21:49.082 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:49.082 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-19T03:21:49.082 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:21:49.083 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-19T03:21:49.083 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-19T03:21:49.083 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:49.083 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-19T03:21:49.083 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-19T03:21:49.083 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-19T03:21:49.083 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-19T03:21:49.083 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-19T03:21:49.083 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-19T03:21:49.083 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "restful" 2024-03-19T03:21:49.083 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout ], 2024-03-19T03:21:49.083 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T03:21:49.083 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:49.083 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-19T03:21:49.084 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:21:49.084 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "modified": "2024-03-19T03:21:30.427711+0000", 2024-03-19T03:21:49.084 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T03:21:49.084 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:49.084 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-19T03:21:49.084 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout } 2024-03-19T03:21:49.084 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.308+0000 7fd26b600700 1 Processor -- start 2024-03-19T03:21:49.084 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.308+0000 7fd26b600700 1 -- start start 2024-03-19T03:21:49.084 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.309+0000 7fd26b600700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd264108830 0x7fd26410ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:49.084 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.309+0000 7fd26b600700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd26407bda0 con 0x7fd264108830 2024-03-19T03:21:49.084 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.309+0000 7fd268c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd264108830 0x7fd26410ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:21:49.084 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.309+0000 7fd268c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd264108830 0x7fd26410ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:54914/0 (socket says 172.21.15.117:54914) 2024-03-19T03:21:49.084 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.309+0000 7fd268c00700 1 -- 172.21.15.117:0/4027985097 learned_addr learned my addr 172.21.15.117:0/4027985097 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:21:49.084 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.309+0000 7fd268c00700 1 -- 172.21.15.117:0/4027985097 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd26410b160 con 0x7fd264108830 2024-03-19T03:21:49.085 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.310+0000 7fd268c00700 1 --2- 172.21.15.117:0/4027985097 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd264108830 0x7fd26410ac20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd2580098a0 tx=0x7fd258009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36e675374c9a0428 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:21:49.085 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.310+0000 7fd263400700 1 -- 172.21.15.117:0/4027985097 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd258004160 con 0x7fd264108830 2024-03-19T03:21:49.085 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.310+0000 7fd263400700 1 -- 172.21.15.117:0/4027985097 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd258004300 con 0x7fd264108830 2024-03-19T03:21:49.085 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.310+0000 7fd263400700 1 -- 172.21.15.117:0/4027985097 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd258011b50 con 0x7fd264108830 2024-03-19T03:21:49.085 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.311+0000 7fd26b600700 1 -- 172.21.15.117:0/4027985097 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd264108830 msgr2=0x7fd26410ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:21:49.085 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.311+0000 7fd26b600700 1 --2- 172.21.15.117:0/4027985097 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd264108830 0x7fd26410ac20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd2580098a0 tx=0x7fd258009bb0 comp rx=0 tx=0).stop 2024-03-19T03:21:49.085 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.311+0000 7fd26b600700 1 -- 172.21.15.117:0/4027985097 shutdown_connections 2024-03-19T03:21:49.085 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.311+0000 7fd26b600700 1 --2- 172.21.15.117:0/4027985097 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd264108830 0x7fd26410ac20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:49.085 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.311+0000 7fd26b600700 1 -- 172.21.15.117:0/4027985097 >> 172.21.15.117:0/4027985097 conn(0x7fd2641021c0 msgr2=0x7fd264104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:21:49.085 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.311+0000 7fd26b600700 1 -- 172.21.15.117:0/4027985097 shutdown_connections 2024-03-19T03:21:49.085 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.311+0000 7fd26b600700 1 -- 172.21.15.117:0/4027985097 wait complete. 2024-03-19T03:21:49.085 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.312+0000 7fd26b600700 1 Processor -- start 2024-03-19T03:21:49.085 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.312+0000 7fd26b600700 1 -- start start 2024-03-19T03:21:49.085 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.313+0000 7fd26b600700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd264108830 0x7fd26419de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:49.085 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.313+0000 7fd26b600700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd26410c0b0 con 0x7fd264108830 2024-03-19T03:21:49.086 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.313+0000 7fd268c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd264108830 0x7fd26419de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:21:49.086 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.313+0000 7fd268c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd264108830 0x7fd26419de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:54926/0 (socket says 172.21.15.117:54926) 2024-03-19T03:21:49.086 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.313+0000 7fd268c00700 1 -- 172.21.15.117:0/1607737686 learned_addr learned my addr 172.21.15.117:0/1607737686 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:21:49.086 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.313+0000 7fd268c00700 1 -- 172.21.15.117:0/1607737686 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd26419e3b0 con 0x7fd264108830 2024-03-19T03:21:49.086 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.314+0000 7fd268c00700 1 --2- 172.21.15.117:0/1607737686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd264108830 0x7fd26419de70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd258011b60 tx=0x7fd258011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:21:49.086 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.314+0000 7fd261600700 1 -- 172.21.15.117:0/1607737686 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd258003b20 con 0x7fd264108830 2024-03-19T03:21:49.086 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.314+0000 7fd261600700 1 -- 172.21.15.117:0/1607737686 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd258003cc0 con 0x7fd264108830 2024-03-19T03:21:49.086 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.314+0000 7fd261600700 1 -- 172.21.15.117:0/1607737686 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd258024840 con 0x7fd264108830 2024-03-19T03:21:49.086 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.314+0000 7fd26b600700 1 -- 172.21.15.117:0/1607737686 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd26419e640 con 0x7fd264108830 2024-03-19T03:21:49.086 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.314+0000 7fd26b600700 1 -- 172.21.15.117:0/1607737686 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2641a1330 con 0x7fd264108830 2024-03-19T03:21:49.086 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.314+0000 7fd261600700 1 -- 172.21.15.117:0/1607737686 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd258022020 con 0x7fd264108830 2024-03-19T03:21:49.086 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.314+0000 7fd261600700 1 -- 172.21.15.117:0/1607737686 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd258011e00 con 0x7fd264108830 2024-03-19T03:21:49.086 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.315+0000 7fd26b600700 1 -- 172.21.15.117:0/1607737686 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd264063940 con 0x7fd264108830 2024-03-19T03:21:49.086 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.319+0000 7fd261600700 1 -- 172.21.15.117:0/1607737686 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fd25802d3f0 con 0x7fd264108830 2024-03-19T03:21:49.087 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.398+0000 7fd26b600700 1 -- 172.21.15.117:0/1607737686 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fd2641a1500 con 0x7fd264108830 2024-03-19T03:21:49.087 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.399+0000 7fd261600700 1 -- 172.21.15.117:0/1607737686 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7fd25802de20 con 0x7fd264108830 2024-03-19T03:21:49.087 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.400+0000 7fd26b600700 1 -- 172.21.15.117:0/1607737686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd264108830 msgr2=0x7fd26419de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:21:49.087 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.400+0000 7fd26b600700 1 --2- 172.21.15.117:0/1607737686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd264108830 0x7fd26419de70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd258011b60 tx=0x7fd258011c40 comp rx=0 tx=0).stop 2024-03-19T03:21:49.087 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.400+0000 7fd26b600700 1 -- 172.21.15.117:0/1607737686 shutdown_connections 2024-03-19T03:21:49.087 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.400+0000 7fd26b600700 1 --2- 172.21.15.117:0/1607737686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd264108830 0x7fd26419de70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:49.087 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.400+0000 7fd26b600700 1 -- 172.21.15.117:0/1607737686 >> 172.21.15.117:0/1607737686 conn(0x7fd2641021c0 msgr2=0x7fd264102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:21:49.087 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.400+0000 7fd26b600700 1 -- 172.21.15.117:0/1607737686 shutdown_connections 2024-03-19T03:21:49.087 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:48.400+0000 7fd26b600700 1 -- 172.21.15.117:0/1607737686 wait complete. 2024-03-19T03:21:49.087 INFO:teuthology.orchestra.run.smithi117.stdout:mgr not available, waiting (3/15)... 2024-03-19T03:21:49.525 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:49 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:49.121+0000 7f6bb0965200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T03:21:49.525 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:49 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:49.248+0000 7f6bb0965200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T03:21:49.792 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:49 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:49.524+0000 7f6bb0965200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T03:21:49.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:49 smithi117 ceph-mon[31838]: Activating manager daemon x 2024-03-19T03:21:49.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:49 smithi117 ceph-mon[31838]: mgrmap e2: x(active, starting, since 0.0114474s) 2024-03-19T03:21:49.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:49 smithi117 ceph-mon[31838]: from='mgr.14100 172.21.15.117:0/3252130481' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-19T03:21:49.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:49 smithi117 ceph-mon[31838]: from='mgr.14100 172.21.15.117:0/3252130481' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-19T03:21:49.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:49 smithi117 ceph-mon[31838]: from='mgr.14100 172.21.15.117:0/3252130481' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-19T03:21:49.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:49 smithi117 ceph-mon[31838]: from='mgr.14100 172.21.15.117:0/3252130481' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T03:21:49.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:49 smithi117 ceph-mon[31838]: from='mgr.14100 172.21.15.117:0/3252130481' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-19T03:21:49.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:49 smithi117 ceph-mon[31838]: Manager daemon x is now available 2024-03-19T03:21:49.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:49 smithi117 ceph-mon[31838]: from='mgr.14100 172.21.15.117:0/3252130481' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-19T03:21:49.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:49 smithi117 ceph-mon[31838]: from='mgr.14100 172.21.15.117:0/3252130481' entity='mgr.x' 2024-03-19T03:21:49.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:49 smithi117 ceph-mon[31838]: from='mgr.14100 172.21.15.117:0/3252130481' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-19T03:21:49.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:49 smithi117 ceph-mon[31838]: from='mgr.14100 172.21.15.117:0/3252130481' entity='mgr.x' 2024-03-19T03:21:49.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:49 smithi117 ceph-mon[31838]: from='mgr.14100 172.21.15.117:0/3252130481' entity='mgr.x' 2024-03-19T03:21:51.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:51 smithi117 ceph-mon[31838]: mgrmap e3: x(active, since 1.01966s) 2024-03-19T03:21:52.612 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout 2024-03-19T03:21:52.612 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout { 2024-03-19T03:21:52.612 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "fsid": "a3a82456-e59f-11ee-95c9-87774f69a715", 2024-03-19T03:21:52.612 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "health": { 2024-03-19T03:21:52.612 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-19T03:21:52.612 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-19T03:21:52.612 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-19T03:21:52.613 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:52.613 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-19T03:21:52.613 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-19T03:21:52.613 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout 0 2024-03-19T03:21:52.613 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout ], 2024-03-19T03:21:52.613 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-19T03:21:52.613 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "a" 2024-03-19T03:21:52.613 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout ], 2024-03-19T03:21:52.613 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-03-19T03:21:52.613 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-19T03:21:52.613 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:21:52.613 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-19T03:21:52.613 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-19T03:21:52.613 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:52.613 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-19T03:21:52.613 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:21:52.613 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-19T03:21:52.614 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-19T03:21:52.614 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-19T03:21:52.614 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-19T03:21:52.622 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-19T03:21:52.622 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-19T03:21:52.622 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:52.622 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-19T03:21:52.622 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-19T03:21:52.622 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-19T03:21:52.622 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-19T03:21:52.622 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-19T03:21:52.623 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-19T03:21:52.623 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-19T03:21:52.623 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-19T03:21:52.623 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-19T03:21:52.623 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:52.623 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-19T03:21:52.623 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:21:52.623 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-19T03:21:52.623 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-19T03:21:52.623 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:52.623 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-19T03:21:52.623 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-19T03:21:52.623 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-19T03:21:52.623 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-19T03:21:52.623 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-19T03:21:52.623 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-19T03:21:52.624 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "restful" 2024-03-19T03:21:52.624 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout ], 2024-03-19T03:21:52.624 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T03:21:52.624 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:52.624 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-19T03:21:52.624 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T03:21:52.624 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "modified": "2024-03-19T03:21:30.427711+0000", 2024-03-19T03:21:52.624 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T03:21:52.624 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout }, 2024-03-19T03:21:52.624 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-19T03:21:52.624 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout } 2024-03-19T03:21:52.624 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.650+0000 7f26c0c00700 1 Processor -- start 2024-03-19T03:21:52.624 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.650+0000 7f26c0c00700 1 -- start start 2024-03-19T03:21:52.624 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.650+0000 7f26c0c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26bc10a0e0 0x7f26bc10a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:52.624 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.650+0000 7f26c0c00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f26bc10aa40 con 0x7f26bc10a0e0 2024-03-19T03:21:52.624 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.651+0000 7f26b9e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26bc10a0e0 0x7f26bc10a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:21:52.625 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.651+0000 7f26b9e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26bc10a0e0 0x7f26bc10a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:55014/0 (socket says 172.21.15.117:55014) 2024-03-19T03:21:52.625 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.651+0000 7f26b9e00700 1 -- 172.21.15.117:0/4184999594 learned_addr learned my addr 172.21.15.117:0/4184999594 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:21:52.625 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.651+0000 7f26b9e00700 1 -- 172.21.15.117:0/4184999594 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26bc10b080 con 0x7f26bc10a0e0 2024-03-19T03:21:52.625 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.652+0000 7f26b9e00700 1 --2- 172.21.15.117:0/4184999594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26bc10a0e0 0x7f26bc10a500 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f26ac0098a0 tx=0x7f26ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5166008db9c5eb7e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:21:52.625 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.652+0000 7f26b8a00700 1 -- 172.21.15.117:0/4184999594 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26ac004160 con 0x7f26bc10a0e0 2024-03-19T03:21:52.625 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.652+0000 7f26b8a00700 1 -- 172.21.15.117:0/4184999594 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f26ac004300 con 0x7f26bc10a0e0 2024-03-19T03:21:52.625 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.652+0000 7f26b8a00700 1 -- 172.21.15.117:0/4184999594 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26ac011b50 con 0x7f26bc10a0e0 2024-03-19T03:21:52.625 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.653+0000 7f26c0c00700 1 -- 172.21.15.117:0/4184999594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26bc10a0e0 msgr2=0x7f26bc10a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:21:52.625 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.653+0000 7f26c0c00700 1 --2- 172.21.15.117:0/4184999594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26bc10a0e0 0x7f26bc10a500 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f26ac0098a0 tx=0x7f26ac009bb0 comp rx=0 tx=0).stop 2024-03-19T03:21:52.625 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.653+0000 7f26c0c00700 1 -- 172.21.15.117:0/4184999594 shutdown_connections 2024-03-19T03:21:52.625 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.653+0000 7f26c0c00700 1 --2- 172.21.15.117:0/4184999594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26bc10a0e0 0x7f26bc10a500 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:52.625 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.653+0000 7f26c0c00700 1 -- 172.21.15.117:0/4184999594 >> 172.21.15.117:0/4184999594 conn(0x7f26bc07d310 msgr2=0x7f26bc07d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:21:52.625 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.653+0000 7f26c0c00700 1 -- 172.21.15.117:0/4184999594 shutdown_connections 2024-03-19T03:21:52.625 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.653+0000 7f26c0c00700 1 -- 172.21.15.117:0/4184999594 wait complete. 2024-03-19T03:21:52.625 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.654+0000 7f26c0c00700 1 Processor -- start 2024-03-19T03:21:52.626 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.654+0000 7f26c0c00700 1 -- start start 2024-03-19T03:21:52.626 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.654+0000 7f26c0c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26bc10a0e0 0x7f26bc19de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:52.626 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.654+0000 7f26c0c00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f26bc10e340 con 0x7f26bc10a0e0 2024-03-19T03:21:52.626 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.654+0000 7f26b9e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26bc10a0e0 0x7f26bc19de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:21:52.626 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.655+0000 7f26b9e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26bc10a0e0 0x7f26bc19de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:55024/0 (socket says 172.21.15.117:55024) 2024-03-19T03:21:52.626 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.655+0000 7f26b9e00700 1 -- 172.21.15.117:0/689806097 learned_addr learned my addr 172.21.15.117:0/689806097 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:21:52.626 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.655+0000 7f26b9e00700 1 -- 172.21.15.117:0/689806097 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26bc19e390 con 0x7f26bc10a0e0 2024-03-19T03:21:52.626 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.655+0000 7f26b9e00700 1 --2- 172.21.15.117:0/689806097 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26bc10a0e0 0x7f26bc19de50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f26ac011b60 tx=0x7f26ac011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:21:52.626 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.655+0000 7f26b2a00700 1 -- 172.21.15.117:0/689806097 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26ac003b20 con 0x7f26bc10a0e0 2024-03-19T03:21:52.626 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.655+0000 7f26b2a00700 1 -- 172.21.15.117:0/689806097 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f26ac003cc0 con 0x7f26bc10a0e0 2024-03-19T03:21:52.626 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.655+0000 7f26b2a00700 1 -- 172.21.15.117:0/689806097 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26ac024840 con 0x7f26bc10a0e0 2024-03-19T03:21:52.626 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.655+0000 7f26c0c00700 1 -- 172.21.15.117:0/689806097 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26bc19e620 con 0x7f26bc10a0e0 2024-03-19T03:21:52.626 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.656+0000 7f26c0c00700 1 -- 172.21.15.117:0/689806097 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26bc1a1310 con 0x7f26bc10a0e0 2024-03-19T03:21:52.626 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.657+0000 7f26b2a00700 1 -- 172.21.15.117:0/689806097 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f26ac022020 con 0x7f26bc10a0e0 2024-03-19T03:21:52.626 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.657+0000 7f26c0c00700 1 -- 172.21.15.117:0/689806097 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26bc10e4c0 con 0x7f26bc10a0e0 2024-03-19T03:21:52.627 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.657+0000 7f26b2a00700 1 --2- 172.21.15.117:0/689806097 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7f26a0038f20 0x7f26a003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:52.627 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.657+0000 7f26b2a00700 1 -- 172.21.15.117:0/689806097 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f26ac011e00 con 0x7f26bc10a0e0 2024-03-19T03:21:52.627 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.657+0000 7f26b9400700 1 --2- 172.21.15.117:0/689806097 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7f26a0038f20 0x7f26a003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:21:52.627 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.658+0000 7f26b9400700 1 --2- 172.21.15.117:0/689806097 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7f26a0038f20 0x7f26a003b3e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f26a8006fd0 tx=0x7f26a8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:21:52.627 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.664+0000 7f26b2a00700 1 -- 172.21.15.117:0/689806097 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f26ac00b3e0 con 0x7f26bc10a0e0 2024-03-19T03:21:52.627 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.970+0000 7f26c0c00700 1 -- 172.21.15.117:0/689806097 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f26bc1a1960 con 0x7f26bc10a0e0 2024-03-19T03:21:52.627 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.972+0000 7f26b2a00700 1 -- 172.21.15.117:0/689806097 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f26ac016260 con 0x7f26bc10a0e0 2024-03-19T03:21:52.627 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.974+0000 7f26c0c00700 1 -- 172.21.15.117:0/689806097 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7f26a0038f20 msgr2=0x7f26a003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:21:52.627 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.974+0000 7f26c0c00700 1 --2- 172.21.15.117:0/689806097 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7f26a0038f20 0x7f26a003b3e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f26a8006fd0 tx=0x7f26a8006e40 comp rx=0 tx=0).stop 2024-03-19T03:21:52.627 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.974+0000 7f26c0c00700 1 -- 172.21.15.117:0/689806097 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26bc10a0e0 msgr2=0x7f26bc19de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:21:52.627 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.974+0000 7f26c0c00700 1 --2- 172.21.15.117:0/689806097 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26bc10a0e0 0x7f26bc19de50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f26ac011b60 tx=0x7f26ac011c40 comp rx=0 tx=0).stop 2024-03-19T03:21:52.627 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.974+0000 7f26c0c00700 1 -- 172.21.15.117:0/689806097 shutdown_connections 2024-03-19T03:21:52.627 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.974+0000 7f26c0c00700 1 --2- 172.21.15.117:0/689806097 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7f26a0038f20 0x7f26a003b3e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:52.627 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.974+0000 7f26c0c00700 1 --2- 172.21.15.117:0/689806097 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26bc10a0e0 0x7f26bc19de50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:52.627 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.974+0000 7f26c0c00700 1 -- 172.21.15.117:0/689806097 >> 172.21.15.117:0/689806097 conn(0x7f26bc07d310 msgr2=0x7f26bc107cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:21:52.627 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.975+0000 7f26c0c00700 1 -- 172.21.15.117:0/689806097 shutdown_connections 2024-03-19T03:21:52.628 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:51.975+0000 7f26c0c00700 1 -- 172.21.15.117:0/689806097 wait complete. 2024-03-19T03:21:52.628 INFO:teuthology.orchestra.run.smithi117.stdout:mgr is available 2024-03-19T03:21:52.922 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:52 smithi117 ceph-mon[31838]: mgrmap e4: x(active, since 2s) 2024-03-19T03:21:52.923 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:52 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/689806097' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-19T03:21:53.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:53 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1188658472' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-19T03:21:54.128 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout 2024-03-19T03:21:54.128 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout [global] 2024-03-19T03:21:54.128 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout fsid = a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:21:54.128 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-19T03:21:54.128 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-19T03:21:54.128 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-19T03:21:54.128 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-19T03:21:54.128 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout 2024-03-19T03:21:54.128 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-19T03:21:54.129 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-19T03:21:54.129 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout 2024-03-19T03:21:54.129 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout [osd] 2024-03-19T03:21:54.134 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-19T03:21:54.134 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-19T03:21:54.134 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-19T03:21:54.134 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-19T03:21:54.134 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-19T03:21:54.134 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.202+0000 7f1f73000700 1 Processor -- start 2024-03-19T03:21:54.135 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.202+0000 7f1f73000700 1 -- start start 2024-03-19T03:21:54.135 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.202+0000 7f1f73000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f6c07eb00 0x7f1f6c07ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:54.135 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.202+0000 7f1f73000700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1f6c07f4f0 con 0x7f1f6c07eb00 2024-03-19T03:21:54.135 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.203+0000 7f1f71c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f6c07eb00 0x7f1f6c07ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:21:54.135 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.203+0000 7f1f71c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f6c07eb00 0x7f1f6c07ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:55026/0 (socket says 172.21.15.117:55026) 2024-03-19T03:21:54.135 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.203+0000 7f1f71c00700 1 -- 172.21.15.117:0/1883088299 learned_addr learned my addr 172.21.15.117:0/1883088299 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:21:54.135 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.203+0000 7f1f71c00700 1 -- 172.21.15.117:0/1883088299 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f6c07fd50 con 0x7f1f6c07eb00 2024-03-19T03:21:54.135 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.204+0000 7f1f71c00700 1 --2- 172.21.15.117:0/1883088299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f6c07eb00 0x7f1f6c07ef20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1f600098a0 tx=0x7f1f60009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96a96fed4143493d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:21:54.135 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.204+0000 7f1f6be00700 1 -- 172.21.15.117:0/1883088299 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f60004160 con 0x7f1f6c07eb00 2024-03-19T03:21:54.135 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.204+0000 7f1f6be00700 1 -- 172.21.15.117:0/1883088299 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1f60004300 con 0x7f1f6c07eb00 2024-03-19T03:21:54.135 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.204+0000 7f1f6be00700 1 -- 172.21.15.117:0/1883088299 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f60011b50 con 0x7f1f6c07eb00 2024-03-19T03:21:54.135 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.205+0000 7f1f73000700 1 -- 172.21.15.117:0/1883088299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f6c07eb00 msgr2=0x7f1f6c07ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:21:54.135 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.205+0000 7f1f73000700 1 --2- 172.21.15.117:0/1883088299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f6c07eb00 0x7f1f6c07ef20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1f600098a0 tx=0x7f1f60009bb0 comp rx=0 tx=0).stop 2024-03-19T03:21:54.135 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.205+0000 7f1f73000700 1 -- 172.21.15.117:0/1883088299 shutdown_connections 2024-03-19T03:21:54.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.205+0000 7f1f73000700 1 --2- 172.21.15.117:0/1883088299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f6c07eb00 0x7f1f6c07ef20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:54.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.205+0000 7f1f73000700 1 -- 172.21.15.117:0/1883088299 >> 172.21.15.117:0/1883088299 conn(0x7f1f6c07d130 msgr2=0x7f1f6c07d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:21:54.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.205+0000 7f1f73000700 1 -- 172.21.15.117:0/1883088299 shutdown_connections 2024-03-19T03:21:54.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.205+0000 7f1f73000700 1 -- 172.21.15.117:0/1883088299 wait complete. 2024-03-19T03:21:54.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.206+0000 7f1f73000700 1 Processor -- start 2024-03-19T03:21:54.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.206+0000 7f1f73000700 1 -- start start 2024-03-19T03:21:54.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.206+0000 7f1f73000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f6c07eb00 0x7f1f6c1a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:54.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.206+0000 7f1f73000700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1f6c081110 con 0x7f1f6c07eb00 2024-03-19T03:21:54.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.207+0000 7f1f71c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f6c07eb00 0x7f1f6c1a20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:21:54.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.207+0000 7f1f71c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f6c07eb00 0x7f1f6c1a20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:55038/0 (socket says 172.21.15.117:55038) 2024-03-19T03:21:54.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.207+0000 7f1f71c00700 1 -- 172.21.15.117:0/1188658472 learned_addr learned my addr 172.21.15.117:0/1188658472 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:21:54.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.207+0000 7f1f71c00700 1 -- 172.21.15.117:0/1188658472 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f6c1a2630 con 0x7f1f6c07eb00 2024-03-19T03:21:54.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.207+0000 7f1f71c00700 1 --2- 172.21.15.117:0/1188658472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f6c07eb00 0x7f1f6c1a20f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1f60009fb0 tx=0x7f1f60011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:21:54.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.207+0000 7f1f6a000700 1 -- 172.21.15.117:0/1188658472 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f600038b0 con 0x7f1f6c07eb00 2024-03-19T03:21:54.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.208+0000 7f1f6a000700 1 -- 172.21.15.117:0/1188658472 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1f60011e50 con 0x7f1f6c07eb00 2024-03-19T03:21:54.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.208+0000 7f1f6a000700 1 -- 172.21.15.117:0/1188658472 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f60019420 con 0x7f1f6c07eb00 2024-03-19T03:21:54.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.208+0000 7f1f73000700 1 -- 172.21.15.117:0/1188658472 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f6c1a28c0 con 0x7f1f6c07eb00 2024-03-19T03:21:54.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.208+0000 7f1f73000700 1 -- 172.21.15.117:0/1188658472 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f6c1a55b0 con 0x7f1f6c07eb00 2024-03-19T03:21:54.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.209+0000 7f1f6a000700 1 -- 172.21.15.117:0/1188658472 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f1f60003a50 con 0x7f1f6c07eb00 2024-03-19T03:21:54.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.209+0000 7f1f73000700 1 -- 172.21.15.117:0/1188658472 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f6c063940 con 0x7f1f6c07eb00 2024-03-19T03:21:54.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.209+0000 7f1f6a000700 1 --2- 172.21.15.117:0/1188658472 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7f1f54038ed0 0x7f1f5403b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:54.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.209+0000 7f1f6a000700 1 -- 172.21.15.117:0/1188658472 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1f6004cfe0 con 0x7f1f6c07eb00 2024-03-19T03:21:54.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.209+0000 7f1f71200700 1 --2- 172.21.15.117:0/1188658472 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7f1f54038ed0 0x7f1f5403b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:21:54.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.210+0000 7f1f71200700 1 --2- 172.21.15.117:0/1188658472 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7f1f54038ed0 0x7f1f5403b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1f58006fd0 tx=0x7f1f58006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:21:54.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.215+0000 7f1f6a000700 1 -- 172.21.15.117:0/1188658472 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1f60023390 con 0x7f1f6c07eb00 2024-03-19T03:21:54.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.434+0000 7f1f73000700 1 -- 172.21.15.117:0/1188658472 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f1f6c0008d0 con 0x7f1f6c07eb00 2024-03-19T03:21:54.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.436+0000 7f1f6a000700 1 -- 172.21.15.117:0/1188658472 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7f1f60019660 con 0x7f1f6c07eb00 2024-03-19T03:21:54.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.439+0000 7f1f73000700 1 -- 172.21.15.117:0/1188658472 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7f1f54038ed0 msgr2=0x7f1f5403b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:21:54.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.439+0000 7f1f73000700 1 --2- 172.21.15.117:0/1188658472 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7f1f54038ed0 0x7f1f5403b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1f58006fd0 tx=0x7f1f58006e40 comp rx=0 tx=0).stop 2024-03-19T03:21:54.138 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.439+0000 7f1f73000700 1 -- 172.21.15.117:0/1188658472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f6c07eb00 msgr2=0x7f1f6c1a20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:21:54.139 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.439+0000 7f1f73000700 1 --2- 172.21.15.117:0/1188658472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f6c07eb00 0x7f1f6c1a20f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1f60009fb0 tx=0x7f1f60011bf0 comp rx=0 tx=0).stop 2024-03-19T03:21:54.139 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.439+0000 7f1f73000700 1 -- 172.21.15.117:0/1188658472 shutdown_connections 2024-03-19T03:21:54.139 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.439+0000 7f1f73000700 1 --2- 172.21.15.117:0/1188658472 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7f1f54038ed0 0x7f1f5403b390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:54.139 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.439+0000 7f1f73000700 1 --2- 172.21.15.117:0/1188658472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f6c07eb00 0x7f1f6c1a20f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:54.139 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.439+0000 7f1f73000700 1 -- 172.21.15.117:0/1188658472 >> 172.21.15.117:0/1188658472 conn(0x7f1f6c07d130 msgr2=0x7f1f6c1087d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:21:54.139 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.439+0000 7f1f73000700 1 -- 172.21.15.117:0/1188658472 shutdown_connections 2024-03-19T03:21:54.139 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:53.439+0000 7f1f73000700 1 -- 172.21.15.117:0/1188658472 wait complete. 2024-03-19T03:21:54.139 INFO:teuthology.orchestra.run.smithi117.stdout:Enabling cephadm module... 2024-03-19T03:21:55.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:55 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1691645642' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-19T03:21:55.292 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:55 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: ignoring --setuser ceph since I am not root 2024-03-19T03:21:55.292 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:55 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: ignoring --setgroup ceph since I am not root 2024-03-19T03:21:55.785 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.737+0000 7fa65ea00700 1 Processor -- start 2024-03-19T03:21:55.785 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.737+0000 7fa65ea00700 1 -- start start 2024-03-19T03:21:55.785 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.737+0000 7fa65ea00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa65810a0e0 0x7fa65810a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:55.785 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.737+0000 7fa65ea00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa65810aa40 con 0x7fa65810a0e0 2024-03-19T03:21:55.785 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.738+0000 7fa657e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa65810a0e0 0x7fa65810a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:21:55.785 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.738+0000 7fa657e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa65810a0e0 0x7fa65810a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:55040/0 (socket says 172.21.15.117:55040) 2024-03-19T03:21:55.785 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.738+0000 7fa657e00700 1 -- 172.21.15.117:0/522286723 learned_addr learned my addr 172.21.15.117:0/522286723 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:21:55.785 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.738+0000 7fa657e00700 1 -- 172.21.15.117:0/522286723 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa65810b080 con 0x7fa65810a0e0 2024-03-19T03:21:55.785 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.738+0000 7fa657e00700 1 --2- 172.21.15.117:0/522286723 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa65810a0e0 0x7fa65810a500 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa6480098a0 tx=0x7fa648009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8676ad699935f6f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:21:55.785 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.739+0000 7fa656a00700 1 -- 172.21.15.117:0/522286723 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa648004160 con 0x7fa65810a0e0 2024-03-19T03:21:55.786 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.739+0000 7fa656a00700 1 -- 172.21.15.117:0/522286723 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa648004300 con 0x7fa65810a0e0 2024-03-19T03:21:55.786 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.739+0000 7fa656a00700 1 -- 172.21.15.117:0/522286723 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa648011b50 con 0x7fa65810a0e0 2024-03-19T03:21:55.786 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.739+0000 7fa65ea00700 1 -- 172.21.15.117:0/522286723 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa65810a0e0 msgr2=0x7fa65810a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:21:55.786 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.739+0000 7fa65ea00700 1 --2- 172.21.15.117:0/522286723 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa65810a0e0 0x7fa65810a500 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa6480098a0 tx=0x7fa648009bb0 comp rx=0 tx=0).stop 2024-03-19T03:21:55.786 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.739+0000 7fa65ea00700 1 -- 172.21.15.117:0/522286723 shutdown_connections 2024-03-19T03:21:55.786 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.739+0000 7fa65ea00700 1 --2- 172.21.15.117:0/522286723 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa65810a0e0 0x7fa65810a500 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:55.786 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.739+0000 7fa65ea00700 1 -- 172.21.15.117:0/522286723 >> 172.21.15.117:0/522286723 conn(0x7fa65807d310 msgr2=0x7fa65807d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:21:55.786 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.740+0000 7fa65ea00700 1 -- 172.21.15.117:0/522286723 shutdown_connections 2024-03-19T03:21:55.786 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.740+0000 7fa65ea00700 1 -- 172.21.15.117:0/522286723 wait complete. 2024-03-19T03:21:55.786 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.741+0000 7fa65ea00700 1 Processor -- start 2024-03-19T03:21:55.786 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.741+0000 7fa65ea00700 1 -- start start 2024-03-19T03:21:55.786 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.741+0000 7fa65ea00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa65810a0e0 0x7fa65819de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:55.786 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.741+0000 7fa65ea00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa65810e340 con 0x7fa65810a0e0 2024-03-19T03:21:55.786 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.741+0000 7fa657e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa65810a0e0 0x7fa65819de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:21:55.787 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.741+0000 7fa657e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa65810a0e0 0x7fa65819de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:55056/0 (socket says 172.21.15.117:55056) 2024-03-19T03:21:55.787 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.741+0000 7fa657e00700 1 -- 172.21.15.117:0/1691645642 learned_addr learned my addr 172.21.15.117:0/1691645642 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:21:55.787 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.742+0000 7fa657e00700 1 -- 172.21.15.117:0/1691645642 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa65819e390 con 0x7fa65810a0e0 2024-03-19T03:21:55.787 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.742+0000 7fa657e00700 1 --2- 172.21.15.117:0/1691645642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa65810a0e0 0x7fa65819de50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa648011b60 tx=0x7fa648011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:21:55.787 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.742+0000 7fa654c00700 1 -- 172.21.15.117:0/1691645642 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa648003b20 con 0x7fa65810a0e0 2024-03-19T03:21:55.787 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.742+0000 7fa654c00700 1 -- 172.21.15.117:0/1691645642 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa648003cc0 con 0x7fa65810a0e0 2024-03-19T03:21:55.787 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.742+0000 7fa654c00700 1 -- 172.21.15.117:0/1691645642 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa648024840 con 0x7fa65810a0e0 2024-03-19T03:21:55.787 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.742+0000 7fa65ea00700 1 -- 172.21.15.117:0/1691645642 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa65819e620 con 0x7fa65810a0e0 2024-03-19T03:21:55.787 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.743+0000 7fa65ea00700 1 -- 172.21.15.117:0/1691645642 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6581a1310 con 0x7fa65810a0e0 2024-03-19T03:21:55.787 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.744+0000 7fa654c00700 1 -- 172.21.15.117:0/1691645642 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7fa648022020 con 0x7fa65810a0e0 2024-03-19T03:21:55.787 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.744+0000 7fa65ea00700 1 -- 172.21.15.117:0/1691645642 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa65810e4c0 con 0x7fa65810a0e0 2024-03-19T03:21:55.787 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.744+0000 7fa654c00700 1 --2- 172.21.15.117:0/1691645642 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7fa63c038f70 0x7fa63c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:55.787 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.744+0000 7fa654c00700 1 -- 172.21.15.117:0/1691645642 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa648011e00 con 0x7fa65810a0e0 2024-03-19T03:21:55.787 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.744+0000 7fa657400700 1 --2- 172.21.15.117:0/1691645642 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7fa63c038f70 0x7fa63c03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:21:55.788 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.745+0000 7fa657400700 1 --2- 172.21.15.117:0/1691645642 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7fa63c038f70 0x7fa63c03b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa644006fd0 tx=0x7fa644006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:21:55.788 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:54.749+0000 7fa654c00700 1 -- 172.21.15.117:0/1691645642 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa64800b3e0 con 0x7fa65810a0e0 2024-03-19T03:21:55.788 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:55.028+0000 7fa65ea00700 1 -- 172.21.15.117:0/1691645642 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fa6581a1960 con 0x7fa65810a0e0 2024-03-19T03:21:55.788 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:55.082+0000 7fa654c00700 1 -- 172.21.15.117:0/1691645642 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fa648016260 con 0x7fa65810a0e0 2024-03-19T03:21:55.788 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:55.086+0000 7fa65ea00700 1 -- 172.21.15.117:0/1691645642 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7fa63c038f70 msgr2=0x7fa63c03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:21:55.788 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:55.087+0000 7fa65ea00700 1 --2- 172.21.15.117:0/1691645642 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7fa63c038f70 0x7fa63c03b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa644006fd0 tx=0x7fa644006e40 comp rx=0 tx=0).stop 2024-03-19T03:21:55.788 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:55.087+0000 7fa65ea00700 1 -- 172.21.15.117:0/1691645642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa65810a0e0 msgr2=0x7fa65819de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:21:55.788 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:55.087+0000 7fa654c00700 1 -- 172.21.15.117:0/1691645642 <== mon.0 v2:172.21.15.117:3300/0 8 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7fa64802c350 con 0x7fa65810a0e0 2024-03-19T03:21:55.788 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:55.087+0000 7fa65ea00700 1 --2- 172.21.15.117:0/1691645642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa65810a0e0 0x7fa65819de50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa648011b60 tx=0x7fa648011c40 comp rx=0 tx=0).stop 2024-03-19T03:21:55.788 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:55.087+0000 7fa65ea00700 1 -- 172.21.15.117:0/1691645642 shutdown_connections 2024-03-19T03:21:55.788 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:55.087+0000 7fa65ea00700 1 --2- 172.21.15.117:0/1691645642 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7fa63c038f70 0x7fa63c03b430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:55.788 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:55.087+0000 7fa65ea00700 1 --2- 172.21.15.117:0/1691645642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa65810a0e0 0x7fa65819de50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:55.788 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:55.087+0000 7fa65ea00700 1 -- 172.21.15.117:0/1691645642 >> 172.21.15.117:0/1691645642 conn(0x7fa65807d310 msgr2=0x7fa658107cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:21:55.788 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:55.088+0000 7fa65ea00700 1 -- 172.21.15.117:0/1691645642 shutdown_connections 2024-03-19T03:21:55.788 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:55.088+0000 7fa65ea00700 1 -- 172.21.15.117:0/1691645642 wait complete. 2024-03-19T03:21:56.041 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:55 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:55.894+0000 7f9f44f16200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T03:21:56.042 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:56 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:56.016+0000 7f9f44f16200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T03:21:56.347 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:56 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1691645642' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-19T03:21:56.347 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:56 smithi117 ceph-mon[31838]: mgrmap e5: x(active, since 5s) 2024-03-19T03:21:56.942 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:56 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:56.819+0000 7f9f44f16200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T03:21:57.247 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:21:57 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/244712239' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-19T03:21:57.247 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:56 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:56.939+0000 7f9f44f16200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T03:21:57.247 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:57 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:57.123+0000 7f9f44f16200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T03:21:57.247 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:57 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:57.245+0000 7f9f44f16200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T03:21:57.389 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout { 2024-03-19T03:21:57.396 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-19T03:21:57.396 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-19T03:21:57.396 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-19T03:21:57.396 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-19T03:21:57.396 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout } 2024-03-19T03:21:57.396 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.407+0000 7f3910200700 1 Processor -- start 2024-03-19T03:21:57.396 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.408+0000 7f3910200700 1 -- start start 2024-03-19T03:21:57.396 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.408+0000 7f3910200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f390807ddb0 0x7f390807c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:57.396 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.408+0000 7f3910200700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f390807c7f0 con 0x7f390807ddb0 2024-03-19T03:21:57.396 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.408+0000 7f390d800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f390807ddb0 0x7f390807c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:21:57.397 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.409+0000 7f390d800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f390807ddb0 0x7f390807c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:55710/0 (socket says 172.21.15.117:55710) 2024-03-19T03:21:57.397 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.409+0000 7f390d800700 1 -- 172.21.15.117:0/3047189490 learned_addr learned my addr 172.21.15.117:0/3047189490 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:21:57.397 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.409+0000 7f390d800700 1 -- 172.21.15.117:0/3047189490 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f390807c970 con 0x7f390807ddb0 2024-03-19T03:21:57.397 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.409+0000 7f390d800700 1 --2- 172.21.15.117:0/3047189490 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f390807ddb0 0x7f390807c2b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f38fc0098a0 tx=0x7f38fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f2543f5928def4e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:21:57.397 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.409+0000 7f3907e00700 1 -- 172.21.15.117:0/3047189490 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f38fc004160 con 0x7f390807ddb0 2024-03-19T03:21:57.397 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.410+0000 7f3907e00700 1 -- 172.21.15.117:0/3047189490 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f38fc004300 con 0x7f390807ddb0 2024-03-19T03:21:57.397 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.410+0000 7f3907e00700 1 -- 172.21.15.117:0/3047189490 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f38fc011b50 con 0x7f390807ddb0 2024-03-19T03:21:57.397 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.410+0000 7f3910200700 1 -- 172.21.15.117:0/3047189490 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f390807ddb0 msgr2=0x7f390807c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:21:57.397 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.410+0000 7f3910200700 1 --2- 172.21.15.117:0/3047189490 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f390807ddb0 0x7f390807c2b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f38fc0098a0 tx=0x7f38fc009bb0 comp rx=0 tx=0).stop 2024-03-19T03:21:57.397 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.410+0000 7f3910200700 1 -- 172.21.15.117:0/3047189490 shutdown_connections 2024-03-19T03:21:57.397 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.410+0000 7f3910200700 1 --2- 172.21.15.117:0/3047189490 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f390807ddb0 0x7f390807c2b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:57.397 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.410+0000 7f3910200700 1 -- 172.21.15.117:0/3047189490 >> 172.21.15.117:0/3047189490 conn(0x7f3908103310 msgr2=0x7f3908105750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:21:57.397 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.411+0000 7f3910200700 1 -- 172.21.15.117:0/3047189490 shutdown_connections 2024-03-19T03:21:57.397 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.411+0000 7f3910200700 1 -- 172.21.15.117:0/3047189490 wait complete. 2024-03-19T03:21:57.398 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.412+0000 7f3910200700 1 Processor -- start 2024-03-19T03:21:57.398 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.412+0000 7f3910200700 1 -- start start 2024-03-19T03:21:57.398 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.412+0000 7f3910200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f390807ddb0 0x7f390819de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:57.398 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.412+0000 7f3910200700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f390810e240 con 0x7f390807ddb0 2024-03-19T03:21:57.398 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.413+0000 7f390d800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f390807ddb0 0x7f390819de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:21:57.398 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.413+0000 7f390d800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f390807ddb0 0x7f390819de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:55726/0 (socket says 172.21.15.117:55726) 2024-03-19T03:21:57.398 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.413+0000 7f390d800700 1 -- 172.21.15.117:0/244712239 learned_addr learned my addr 172.21.15.117:0/244712239 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:21:57.398 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.413+0000 7f390d800700 1 -- 172.21.15.117:0/244712239 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f390819e340 con 0x7f390807ddb0 2024-03-19T03:21:57.398 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.413+0000 7f390d800700 1 --2- 172.21.15.117:0/244712239 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f390807ddb0 0x7f390819de00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f38fc011b60 tx=0x7f38fc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:21:57.398 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.414+0000 7f3906000700 1 -- 172.21.15.117:0/244712239 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f38fc003b20 con 0x7f390807ddb0 2024-03-19T03:21:57.398 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.414+0000 7f3906000700 1 -- 172.21.15.117:0/244712239 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f38fc003cc0 con 0x7f390807ddb0 2024-03-19T03:21:57.398 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.414+0000 7f3910200700 1 -- 172.21.15.117:0/244712239 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f390819e5d0 con 0x7f390807ddb0 2024-03-19T03:21:57.398 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.414+0000 7f3906000700 1 -- 172.21.15.117:0/244712239 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f38fc024840 con 0x7f390807ddb0 2024-03-19T03:21:57.398 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.414+0000 7f3910200700 1 -- 172.21.15.117:0/244712239 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39081a12c0 con 0x7f390807ddb0 2024-03-19T03:21:57.399 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.415+0000 7f3910200700 1 -- 172.21.15.117:0/244712239 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3908063940 con 0x7f390807ddb0 2024-03-19T03:21:57.399 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.415+0000 7f3906000700 1 -- 172.21.15.117:0/244712239 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7f38fc022020 con 0x7f390807ddb0 2024-03-19T03:21:57.399 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.416+0000 7f3906000700 1 --2- 172.21.15.117:0/244712239 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7f38f0038f70 0x7f38f003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:21:57.399 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.416+0000 7f3906000700 1 -- 172.21.15.117:0/244712239 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f38fc011e00 con 0x7f390807ddb0 2024-03-19T03:21:57.399 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.416+0000 7f390ce00700 1 -- 172.21.15.117:0/244712239 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7f38f0038f70 msgr2=0x7f38f003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:6800/1570899077 2024-03-19T03:21:57.399 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.416+0000 7f390ce00700 1 --2- 172.21.15.117:0/244712239 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7f38f0038f70 0x7f38f003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T03:21:57.399 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.421+0000 7f3906000700 1 -- 172.21.15.117:0/244712239 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f38fc04c950 con 0x7f390807ddb0 2024-03-19T03:21:57.399 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.617+0000 7f390ce00700 1 -- 172.21.15.117:0/244712239 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7f38f0038f70 msgr2=0x7f38f003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:6800/1570899077 2024-03-19T03:21:57.399 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.617+0000 7f390ce00700 1 --2- 172.21.15.117:0/244712239 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7f38f0038f70 0x7f38f003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T03:21:57.399 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.703+0000 7f3910200700 1 -- 172.21.15.117:0/244712239 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f39081a1490 con 0x7f390807ddb0 2024-03-19T03:21:57.399 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.704+0000 7f3906000700 1 -- 172.21.15.117:0/244712239 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f39081a1490 con 0x7f390807ddb0 2024-03-19T03:21:57.399 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.706+0000 7f3910200700 1 -- 172.21.15.117:0/244712239 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7f38f0038f70 msgr2=0x7f38f003b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T03:21:57.399 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.706+0000 7f3910200700 1 --2- 172.21.15.117:0/244712239 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7f38f0038f70 0x7f38f003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:57.399 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.706+0000 7f3910200700 1 -- 172.21.15.117:0/244712239 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f390807ddb0 msgr2=0x7f390819de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:21:57.400 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.706+0000 7f3910200700 1 --2- 172.21.15.117:0/244712239 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f390807ddb0 0x7f390819de00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f38fc011b60 tx=0x7f38fc011c40 comp rx=0 tx=0).stop 2024-03-19T03:21:57.400 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.707+0000 7f3910200700 1 -- 172.21.15.117:0/244712239 shutdown_connections 2024-03-19T03:21:57.400 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.707+0000 7f3910200700 1 --2- 172.21.15.117:0/244712239 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7f38f0038f70 0x7f38f003b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:57.400 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.707+0000 7f3910200700 1 --2- 172.21.15.117:0/244712239 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f390807ddb0 0x7f390819de00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:21:57.400 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.707+0000 7f3910200700 1 -- 172.21.15.117:0/244712239 >> 172.21.15.117:0/244712239 conn(0x7f3908103310 msgr2=0x7f3908103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:21:57.400 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.707+0000 7f3910200700 1 -- 172.21.15.117:0/244712239 shutdown_connections 2024-03-19T03:21:57.400 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:56.707+0000 7f3910200700 1 -- 172.21.15.117:0/244712239 wait complete. 2024-03-19T03:21:57.400 INFO:teuthology.orchestra.run.smithi117.stdout:Waiting for the mgr to restart... 2024-03-19T03:21:57.400 INFO:teuthology.orchestra.run.smithi117.stdout:Waiting for mgr epoch 5... 2024-03-19T03:21:57.538 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:57 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:57.354+0000 7f9f44f16200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T03:21:57.791 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:57 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:57.703+0000 7f9f44f16200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T03:21:58.092 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:57 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:57.848+0000 7f9f44f16200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T03:21:58.092 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:57 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:57.963+0000 7f9f44f16200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T03:21:58.092 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:58 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:58.090+0000 7f9f44f16200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T03:21:58.359 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:58 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:58.312+0000 7f9f44f16200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T03:22:00.042 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:59 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:59.563+0000 7f9f44f16200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T03:22:00.042 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:21:59 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:21:59.732+0000 7f9f44f16200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T03:22:00.537 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:00 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:00.161+0000 7f9f44f16200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T03:22:00.537 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:00 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:00.278+0000 7f9f44f16200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T03:22:00.791 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:00 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:00.535+0000 7f9f44f16200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T03:22:00.792 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:00 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:00.779+0000 7f9f44f16200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T03:22:01.493 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:01 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:01.153+0000 7f9f44f16200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T03:22:01.792 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:01 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:01.491+0000 7f9f44f16200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T03:22:02.294 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:01 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:01.794+0000 7f9f44f16200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T03:22:03.291 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:02 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:02.802+0000 7f9f44f16200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T03:22:03.792 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:03 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:03.489+0000 7f9f44f16200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T03:22:03.792 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:03 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:03.621+0000 7f9f44f16200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T03:22:04.292 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:03 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:03.901+0000 7f9f44f16200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T03:22:04.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:03 smithi117 ceph-mon[31838]: Active manager daemon x restarted 2024-03-19T03:22:04.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:03 smithi117 ceph-mon[31838]: Activating manager daemon x 2024-03-19T03:22:04.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:03 smithi117 ceph-mon[31838]: osdmap e2: 0 total, 0 up, 0 in 2024-03-19T03:22:04.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:03 smithi117 ceph-mon[31838]: mgrmap e6: x(active, starting, since 0.00967513s) 2024-03-19T03:22:04.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:03 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T03:22:04.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:03 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-19T03:22:04.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:03 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-19T03:22:04.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:03 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-19T03:22:04.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:03 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-19T03:22:04.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:03 smithi117 ceph-mon[31838]: Manager daemon x is now available 2024-03-19T03:22:04.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:03 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' 2024-03-19T03:22:04.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:03 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' 2024-03-19T03:22:05.196 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:04 smithi117 ceph-mon[31838]: Found migration_current of "None". Setting to last migration. 2024-03-19T03:22:05.197 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:04 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:22:05.197 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:04 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:22:05.197 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:04 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:22:05.197 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:04 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-19T03:22:05.197 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:04 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-19T03:22:05.197 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:04 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' 2024-03-19T03:22:05.197 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:04 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' 2024-03-19T03:22:05.197 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:04 smithi117 ceph-mon[31838]: mgrmap e7: x(active, since 1.01163s) 2024-03-19T03:22:05.656 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout { 2024-03-19T03:22:05.656 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-19T03:22:05.656 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-19T03:22:05.656 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout } 2024-03-19T03:22:05.656 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.149+0000 7fce3be00700 1 Processor -- start 2024-03-19T03:22:05.656 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.149+0000 7fce3be00700 1 -- start start 2024-03-19T03:22:05.656 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.150+0000 7fce3be00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce3c07e5f0 0x7fce3c07cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:05.656 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.150+0000 7fce3be00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fce3c07d0a0 con 0x7fce3c07e5f0 2024-03-19T03:22:05.656 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.150+0000 7fce3aa00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce3c07e5f0 0x7fce3c07cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:05.656 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.150+0000 7fce3aa00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce3c07e5f0 0x7fce3c07cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:55736/0 (socket says 172.21.15.117:55736) 2024-03-19T03:22:05.657 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.150+0000 7fce3aa00700 1 -- 172.21.15.117:0/2843577633 learned_addr learned my addr 172.21.15.117:0/2843577633 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:05.657 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.150+0000 7fce3aa00700 1 -- 172.21.15.117:0/2843577633 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce3c07d220 con 0x7fce3c07e5f0 2024-03-19T03:22:05.657 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.151+0000 7fce3aa00700 1 --2- 172.21.15.117:0/2843577633 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce3c07e5f0 0x7fce3c07cb60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fce2c0098a0 tx=0x7fce2c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92be58ffd2931c11 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:05.657 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.151+0000 7fce39600700 1 -- 172.21.15.117:0/2843577633 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fce2c004160 con 0x7fce3c07e5f0 2024-03-19T03:22:05.657 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.151+0000 7fce39600700 1 -- 172.21.15.117:0/2843577633 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fce2c004300 con 0x7fce3c07e5f0 2024-03-19T03:22:05.657 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.151+0000 7fce39600700 1 -- 172.21.15.117:0/2843577633 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fce2c011b50 con 0x7fce3c07e5f0 2024-03-19T03:22:05.657 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.152+0000 7fce3be00700 1 -- 172.21.15.117:0/2843577633 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce3c07e5f0 msgr2=0x7fce3c07cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:05.657 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.152+0000 7fce3be00700 1 --2- 172.21.15.117:0/2843577633 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce3c07e5f0 0x7fce3c07cb60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fce2c0098a0 tx=0x7fce2c009bb0 comp rx=0 tx=0).stop 2024-03-19T03:22:05.657 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.152+0000 7fce3be00700 1 -- 172.21.15.117:0/2843577633 shutdown_connections 2024-03-19T03:22:05.657 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.152+0000 7fce3be00700 1 --2- 172.21.15.117:0/2843577633 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce3c07e5f0 0x7fce3c07cb60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:05.657 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.152+0000 7fce3be00700 1 -- 172.21.15.117:0/2843577633 >> 172.21.15.117:0/2843577633 conn(0x7fce3c103210 msgr2=0x7fce3c105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:05.657 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.152+0000 7fce3be00700 1 -- 172.21.15.117:0/2843577633 shutdown_connections 2024-03-19T03:22:05.657 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.152+0000 7fce3be00700 1 -- 172.21.15.117:0/2843577633 wait complete. 2024-03-19T03:22:05.658 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.154+0000 7fce3be00700 1 Processor -- start 2024-03-19T03:22:05.658 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.154+0000 7fce3be00700 1 -- start start 2024-03-19T03:22:05.658 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.154+0000 7fce3be00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce3c07e5f0 0x7fce3c1a20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:05.658 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.154+0000 7fce3be00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fce3c109c00 con 0x7fce3c07e5f0 2024-03-19T03:22:05.658 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.154+0000 7fce3aa00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce3c07e5f0 0x7fce3c1a20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:05.658 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.154+0000 7fce3aa00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce3c07e5f0 0x7fce3c1a20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:55748/0 (socket says 172.21.15.117:55748) 2024-03-19T03:22:05.658 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.154+0000 7fce3aa00700 1 -- 172.21.15.117:0/1030512541 learned_addr learned my addr 172.21.15.117:0/1030512541 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:05.658 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.155+0000 7fce3aa00700 1 -- 172.21.15.117:0/1030512541 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce3c1a2620 con 0x7fce3c07e5f0 2024-03-19T03:22:05.658 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.155+0000 7fce3aa00700 1 --2- 172.21.15.117:0/1030512541 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce3c07e5f0 0x7fce3c1a20e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fce2c011b60 tx=0x7fce2c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:05.658 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.155+0000 7fce33400700 1 -- 172.21.15.117:0/1030512541 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fce2c003970 con 0x7fce3c07e5f0 2024-03-19T03:22:05.658 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.155+0000 7fce33400700 1 -- 172.21.15.117:0/1030512541 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fce2c003b10 con 0x7fce3c07e5f0 2024-03-19T03:22:05.658 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.155+0000 7fce33400700 1 -- 172.21.15.117:0/1030512541 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fce2c024610 con 0x7fce3c07e5f0 2024-03-19T03:22:05.658 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.155+0000 7fce3be00700 1 -- 172.21.15.117:0/1030512541 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce3c1a28b0 con 0x7fce3c07e5f0 2024-03-19T03:22:05.658 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.155+0000 7fce3be00700 1 -- 172.21.15.117:0/1030512541 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce3c1a55a0 con 0x7fce3c07e5f0 2024-03-19T03:22:05.658 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.156+0000 7fce33400700 1 -- 172.21.15.117:0/1030512541 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7fce2c0248c0 con 0x7fce3c07e5f0 2024-03-19T03:22:05.659 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.157+0000 7fce33400700 1 --2- 172.21.15.117:0/1030512541 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7fce20038f70 0x7fce2003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:05.659 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.157+0000 7fce3be00700 1 -- 172.21.15.117:0/1030512541 --> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fce3c063940 con 0x7fce20038f70 2024-03-19T03:22:05.659 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.157+0000 7fce33400700 1 -- 172.21.15.117:0/1030512541 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fce2c011e00 con 0x7fce3c07e5f0 2024-03-19T03:22:05.659 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.157+0000 7fce3a000700 1 -- 172.21.15.117:0/1030512541 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7fce20038f70 msgr2=0x7fce2003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:6800/1570899077 2024-03-19T03:22:05.659 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.157+0000 7fce3a000700 1 --2- 172.21.15.117:0/1030512541 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7fce20038f70 0x7fce2003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T03:22:05.659 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.358+0000 7fce3a000700 1 -- 172.21.15.117:0/1030512541 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7fce20038f70 msgr2=0x7fce2003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:6800/1570899077 2024-03-19T03:22:05.659 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.358+0000 7fce3a000700 1 --2- 172.21.15.117:0/1030512541 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7fce20038f70 0x7fce2003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T03:22:05.659 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.759+0000 7fce3a000700 1 -- 172.21.15.117:0/1030512541 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7fce20038f70 msgr2=0x7fce2003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:6800/1570899077 2024-03-19T03:22:05.659 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:58.759+0000 7fce3a000700 1 --2- 172.21.15.117:0/1030512541 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7fce20038f70 0x7fce2003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-19T03:22:05.659 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:59.560+0000 7fce3a000700 1 -- 172.21.15.117:0/1030512541 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7fce20038f70 msgr2=0x7fce2003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:6800/1570899077 2024-03-19T03:22:05.659 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:21:59.560+0000 7fce3a000700 1 --2- 172.21.15.117:0/1030512541 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7fce20038f70 0x7fce2003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-19T03:22:05.659 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:01.162+0000 7fce3a000700 1 -- 172.21.15.117:0/1030512541 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7fce20038f70 msgr2=0x7fce2003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:6800/1570899077 2024-03-19T03:22:05.659 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:01.162+0000 7fce3a000700 1 --2- 172.21.15.117:0/1030512541 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7fce20038f70 0x7fce2003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-19T03:22:05.659 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:03.910+0000 7fce33400700 1 -- 172.21.15.117:0/1030512541 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mgrmap(e 6) ==== 45873+0+0 (secure 0 0 0) 0x7fce2c02d910 con 0x7fce3c07e5f0 2024-03-19T03:22:05.660 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:03.910+0000 7fce33400700 1 -- 172.21.15.117:0/1030512541 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7fce20038f70 msgr2=0x7fce2003b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T03:22:05.660 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:03.910+0000 7fce33400700 1 --2- 172.21.15.117:0/1030512541 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7fce20038f70 0x7fce2003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:05.660 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:04.914+0000 7fce33400700 1 -- 172.21.15.117:0/1030512541 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mgrmap(e 7) ==== 46000+0+0 (secure 0 0 0) 0x7fce2c04ef10 con 0x7fce3c07e5f0 2024-03-19T03:22:05.660 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:04.915+0000 7fce33400700 1 --2- 172.21.15.117:0/1030512541 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7fce2003c670 0x7fce2003ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:05.660 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:04.915+0000 7fce33400700 1 -- 172.21.15.117:0/1030512541 --> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fce3c063940 con 0x7fce2003c670 2024-03-19T03:22:05.660 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:04.917+0000 7fce3a000700 1 --2- 172.21.15.117:0/1030512541 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7fce2003c670 0x7fce2003ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:05.660 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:04.921+0000 7fce3a000700 1 --2- 172.21.15.117:0/1030512541 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7fce2003c670 0x7fce2003ea60 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fce28003a10 tx=0x7fce280092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:05.660 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:04.923+0000 7fce33400700 1 -- 172.21.15.117:0/1030512541 <== mgr.14120 v2:172.21.15.117:6800/2788067603 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fce3c063940 con 0x7fce2003c670 2024-03-19T03:22:05.660 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:04.932+0000 7fce3be00700 1 -- 172.21.15.117:0/1030512541 --> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fce3c062750 con 0x7fce2003c670 2024-03-19T03:22:05.660 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:04.933+0000 7fce33400700 1 -- 172.21.15.117:0/1030512541 <== mgr.14120 v2:172.21.15.117:6800/2788067603 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fce3c062750 con 0x7fce2003c670 2024-03-19T03:22:05.660 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:04.933+0000 7fce3be00700 1 -- 172.21.15.117:0/1030512541 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7fce2003c670 msgr2=0x7fce2003ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:05.660 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:04.933+0000 7fce3be00700 1 --2- 172.21.15.117:0/1030512541 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7fce2003c670 0x7fce2003ea60 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fce28003a10 tx=0x7fce280092b0 comp rx=0 tx=0).stop 2024-03-19T03:22:05.660 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:04.933+0000 7fce3be00700 1 -- 172.21.15.117:0/1030512541 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce3c07e5f0 msgr2=0x7fce3c1a20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:05.661 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:04.933+0000 7fce3be00700 1 --2- 172.21.15.117:0/1030512541 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce3c07e5f0 0x7fce3c1a20e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fce2c011b60 tx=0x7fce2c011c40 comp rx=0 tx=0).stop 2024-03-19T03:22:05.661 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:04.934+0000 7fce3be00700 1 -- 172.21.15.117:0/1030512541 shutdown_connections 2024-03-19T03:22:05.661 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:04.934+0000 7fce3be00700 1 --2- 172.21.15.117:0/1030512541 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7fce2003c670 0x7fce2003ea60 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:05.661 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:04.934+0000 7fce3be00700 1 --2- 172.21.15.117:0/1030512541 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce3c07e5f0 0x7fce3c1a20e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:05.661 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:04.934+0000 7fce3be00700 1 --2- 172.21.15.117:0/1030512541 >> [v2:172.21.15.117:6800/1570899077,v1:172.21.15.117:6801/1570899077] conn(0x7fce20038f70 0x7fce2003b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:05.661 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:04.934+0000 7fce3be00700 1 -- 172.21.15.117:0/1030512541 >> 172.21.15.117:0/1030512541 conn(0x7fce3c103210 msgr2=0x7fce3c105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:05.661 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:04.934+0000 7fce3be00700 1 -- 172.21.15.117:0/1030512541 shutdown_connections 2024-03-19T03:22:05.661 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:04.934+0000 7fce3be00700 1 -- 172.21.15.117:0/1030512541 wait complete. 2024-03-19T03:22:05.661 INFO:teuthology.orchestra.run.smithi117.stdout:mgr epoch 5 is available 2024-03-19T03:22:05.661 INFO:teuthology.orchestra.run.smithi117.stdout:Setting orchestrator backend to cephadm... 2024-03-19T03:22:06.069 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:05 smithi117 ceph-mon[31838]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-19T03:22:06.069 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:05 smithi117 ceph-mon[31838]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-19T03:22:06.069 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:05 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:22:07.182 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:06 smithi117 ceph-mon[31838]: [19/Mar/2024:03:22:05] ENGINE Bus STARTING 2024-03-19T03:22:07.182 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:06 smithi117 ceph-mon[31838]: [19/Mar/2024:03:22:05] ENGINE Serving on https://172.21.15.117:7150 2024-03-19T03:22:07.182 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:06 smithi117 ceph-mon[31838]: [19/Mar/2024:03:22:05] ENGINE Serving on http://172.21.15.117:8765 2024-03-19T03:22:07.182 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:06 smithi117 ceph-mon[31838]: [19/Mar/2024:03:22:05] ENGINE Bus STARTED 2024-03-19T03:22:07.182 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:06 smithi117 ceph-mon[31838]: mgrmap e8: x(active, since 2s) 2024-03-19T03:22:07.182 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:06 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' 2024-03-19T03:22:07.182 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:06 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:22:07.281 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.231+0000 7f03d7a00700 1 Processor -- start 2024-03-19T03:22:07.281 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.232+0000 7f03d7a00700 1 -- start start 2024-03-19T03:22:07.281 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.232+0000 7f03d7a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03d010a0e0 0x7f03d010a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:07.281 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.232+0000 7f03d7a00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f03d010aa40 con 0x7f03d010a0e0 2024-03-19T03:22:07.281 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.232+0000 7f03d5000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03d010a0e0 0x7f03d010a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:07.281 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.232+0000 7f03d5000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03d010a0e0 0x7f03d010a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:45414/0 (socket says 172.21.15.117:45414) 2024-03-19T03:22:07.281 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.232+0000 7f03d5000700 1 -- 172.21.15.117:0/2748515656 learned_addr learned my addr 172.21.15.117:0/2748515656 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:07.285 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.233+0000 7f03d5000700 1 -- 172.21.15.117:0/2748515656 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03d010b080 con 0x7f03d010a0e0 2024-03-19T03:22:07.285 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.233+0000 7f03d5000700 1 --2- 172.21.15.117:0/2748515656 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03d010a0e0 0x7f03d010a500 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f03c40098a0 tx=0x7f03c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bb78e5790e69bef server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:07.285 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.233+0000 7f03cf400700 1 -- 172.21.15.117:0/2748515656 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03c4004160 con 0x7f03d010a0e0 2024-03-19T03:22:07.285 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.233+0000 7f03cf400700 1 -- 172.21.15.117:0/2748515656 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f03c4004300 con 0x7f03d010a0e0 2024-03-19T03:22:07.285 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.233+0000 7f03cf400700 1 -- 172.21.15.117:0/2748515656 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03c4011b50 con 0x7f03d010a0e0 2024-03-19T03:22:07.285 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.234+0000 7f03d7a00700 1 -- 172.21.15.117:0/2748515656 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03d010a0e0 msgr2=0x7f03d010a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:07.285 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.234+0000 7f03d7a00700 1 --2- 172.21.15.117:0/2748515656 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03d010a0e0 0x7f03d010a500 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f03c40098a0 tx=0x7f03c4009bb0 comp rx=0 tx=0).stop 2024-03-19T03:22:07.285 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.234+0000 7f03d7a00700 1 -- 172.21.15.117:0/2748515656 shutdown_connections 2024-03-19T03:22:07.285 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.234+0000 7f03d7a00700 1 --2- 172.21.15.117:0/2748515656 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03d010a0e0 0x7f03d010a500 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:07.285 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.234+0000 7f03d7a00700 1 -- 172.21.15.117:0/2748515656 >> 172.21.15.117:0/2748515656 conn(0x7f03d007d310 msgr2=0x7f03d007d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:07.285 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.234+0000 7f03d7a00700 1 -- 172.21.15.117:0/2748515656 shutdown_connections 2024-03-19T03:22:07.285 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.234+0000 7f03d7a00700 1 -- 172.21.15.117:0/2748515656 wait complete. 2024-03-19T03:22:07.285 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.235+0000 7f03d7a00700 1 Processor -- start 2024-03-19T03:22:07.285 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.235+0000 7f03d7a00700 1 -- start start 2024-03-19T03:22:07.286 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.236+0000 7f03d7a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03d010a0e0 0x7f03d019de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:07.286 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.236+0000 7f03d7a00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f03d010e340 con 0x7f03d010a0e0 2024-03-19T03:22:07.286 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.236+0000 7f03d5000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03d010a0e0 0x7f03d019de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:07.286 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.236+0000 7f03d5000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03d010a0e0 0x7f03d019de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:45420/0 (socket says 172.21.15.117:45420) 2024-03-19T03:22:07.286 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.236+0000 7f03d5000700 1 -- 172.21.15.117:0/641587523 learned_addr learned my addr 172.21.15.117:0/641587523 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:07.286 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.236+0000 7f03d5000700 1 -- 172.21.15.117:0/641587523 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03d019e390 con 0x7f03d010a0e0 2024-03-19T03:22:07.286 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.237+0000 7f03d5000700 1 --2- 172.21.15.117:0/641587523 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03d010a0e0 0x7f03d019de50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f03c4011b60 tx=0x7f03c4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:07.286 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.237+0000 7f03cd600700 1 -- 172.21.15.117:0/641587523 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03c4003b20 con 0x7f03d010a0e0 2024-03-19T03:22:07.286 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.237+0000 7f03cd600700 1 -- 172.21.15.117:0/641587523 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f03c4003cc0 con 0x7f03d010a0e0 2024-03-19T03:22:07.286 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.237+0000 7f03d7a00700 1 -- 172.21.15.117:0/641587523 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03d019e620 con 0x7f03d010a0e0 2024-03-19T03:22:07.286 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.237+0000 7f03cd600700 1 -- 172.21.15.117:0/641587523 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03c4024840 con 0x7f03d010a0e0 2024-03-19T03:22:07.286 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.237+0000 7f03d7a00700 1 -- 172.21.15.117:0/641587523 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03d01a1310 con 0x7f03d010a0e0 2024-03-19T03:22:07.286 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.238+0000 7f03cd600700 1 -- 172.21.15.117:0/641587523 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f03c4022020 con 0x7f03d010a0e0 2024-03-19T03:22:07.286 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.238+0000 7f03d7a00700 1 -- 172.21.15.117:0/641587523 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03d010e4c0 con 0x7f03d010a0e0 2024-03-19T03:22:07.288 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.239+0000 7f03cd600700 1 --2- 172.21.15.117:0/641587523 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f03b8038f20 0x7f03b803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:07.288 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.239+0000 7f03cd600700 1 -- 172.21.15.117:0/641587523 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f03c4011e00 con 0x7f03d010a0e0 2024-03-19T03:22:07.289 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.239+0000 7f03cfe00700 1 --2- 172.21.15.117:0/641587523 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f03b8038f20 0x7f03b803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:07.289 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.239+0000 7f03cfe00700 1 --2- 172.21.15.117:0/641587523 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f03b8038f20 0x7f03b803b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f03bc006fd0 tx=0x7f03bc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:07.289 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.244+0000 7f03cd600700 1 -- 172.21.15.117:0/641587523 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f03c404c980 con 0x7f03d010a0e0 2024-03-19T03:22:07.289 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.482+0000 7f03d7a00700 1 -- 172.21.15.117:0/641587523 --> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f03d01083e0 con 0x7f03b8038f20 2024-03-19T03:22:07.289 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.496+0000 7f03cd600700 1 -- 172.21.15.117:0/641587523 <== mgr.14120 v2:172.21.15.117:6800/2788067603 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f03d01083e0 con 0x7f03b8038f20 2024-03-19T03:22:07.289 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.499+0000 7f03d7a00700 1 -- 172.21.15.117:0/641587523 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f03b8038f20 msgr2=0x7f03b803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:07.289 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.499+0000 7f03d7a00700 1 --2- 172.21.15.117:0/641587523 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f03b8038f20 0x7f03b803b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f03bc006fd0 tx=0x7f03bc006e40 comp rx=0 tx=0).stop 2024-03-19T03:22:07.289 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.499+0000 7f03d7a00700 1 -- 172.21.15.117:0/641587523 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03d010a0e0 msgr2=0x7f03d019de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:07.289 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.499+0000 7f03d7a00700 1 --2- 172.21.15.117:0/641587523 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03d010a0e0 0x7f03d019de50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f03c4011b60 tx=0x7f03c4011c40 comp rx=0 tx=0).stop 2024-03-19T03:22:07.289 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.500+0000 7f03d7a00700 1 -- 172.21.15.117:0/641587523 shutdown_connections 2024-03-19T03:22:07.289 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.500+0000 7f03d7a00700 1 --2- 172.21.15.117:0/641587523 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f03b8038f20 0x7f03b803b3e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:07.289 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.500+0000 7f03d7a00700 1 --2- 172.21.15.117:0/641587523 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03d010a0e0 0x7f03d019de50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:07.289 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.500+0000 7f03d7a00700 1 -- 172.21.15.117:0/641587523 >> 172.21.15.117:0/641587523 conn(0x7f03d007d310 msgr2=0x7f03d0107cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:07.290 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.500+0000 7f03d7a00700 1 -- 172.21.15.117:0/641587523 shutdown_connections 2024-03-19T03:22:07.290 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:06.500+0000 7f03d7a00700 1 -- 172.21.15.117:0/641587523 wait complete. 2024-03-19T03:22:08.291 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:07 smithi117 ceph-mon[31838]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:22:08.681 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-19T03:22:08.682 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.840+0000 7f94cfe00700 1 Processor -- start 2024-03-19T03:22:08.682 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.840+0000 7f94cfe00700 1 -- start start 2024-03-19T03:22:08.682 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.841+0000 7f94cfe00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f94d007de30 0x7f94d007c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:08.682 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.841+0000 7f94cfe00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f94d007c860 con 0x7f94d007de30 2024-03-19T03:22:08.682 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.841+0000 7f94cea00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f94d007de30 0x7f94d007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:08.682 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.841+0000 7f94cea00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f94d007de30 0x7f94d007c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:45426/0 (socket says 172.21.15.117:45426) 2024-03-19T03:22:08.687 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.841+0000 7f94cea00700 1 -- 172.21.15.117:0/2858417055 learned_addr learned my addr 172.21.15.117:0/2858417055 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:08.687 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.842+0000 7f94cea00700 1 -- 172.21.15.117:0/2858417055 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94d007c9e0 con 0x7f94d007de30 2024-03-19T03:22:08.687 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.842+0000 7f94cea00700 1 --2- 172.21.15.117:0/2858417055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f94d007de30 0x7f94d007c290 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f94c00098a0 tx=0x7f94c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e709da63539d88bd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:08.687 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.842+0000 7f94cd600700 1 -- 172.21.15.117:0/2858417055 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94c0004160 con 0x7f94d007de30 2024-03-19T03:22:08.687 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.842+0000 7f94cd600700 1 -- 172.21.15.117:0/2858417055 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f94c0004300 con 0x7f94d007de30 2024-03-19T03:22:08.688 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.842+0000 7f94cd600700 1 -- 172.21.15.117:0/2858417055 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94c0011b50 con 0x7f94d007de30 2024-03-19T03:22:08.688 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.843+0000 7f94cfe00700 1 -- 172.21.15.117:0/2858417055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f94d007de30 msgr2=0x7f94d007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:08.688 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.843+0000 7f94cfe00700 1 --2- 172.21.15.117:0/2858417055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f94d007de30 0x7f94d007c290 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f94c00098a0 tx=0x7f94c0009bb0 comp rx=0 tx=0).stop 2024-03-19T03:22:08.688 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.843+0000 7f94cfe00700 1 -- 172.21.15.117:0/2858417055 shutdown_connections 2024-03-19T03:22:08.688 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.843+0000 7f94cfe00700 1 --2- 172.21.15.117:0/2858417055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f94d007de30 0x7f94d007c290 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:08.688 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.843+0000 7f94cfe00700 1 -- 172.21.15.117:0/2858417055 >> 172.21.15.117:0/2858417055 conn(0x7f94d0103080 msgr2=0x7f94d0105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:08.688 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.843+0000 7f94cfe00700 1 -- 172.21.15.117:0/2858417055 shutdown_connections 2024-03-19T03:22:08.688 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.843+0000 7f94cfe00700 1 -- 172.21.15.117:0/2858417055 wait complete. 2024-03-19T03:22:08.688 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.844+0000 7f94cfe00700 1 Processor -- start 2024-03-19T03:22:08.688 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.844+0000 7f94cfe00700 1 -- start start 2024-03-19T03:22:08.688 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.845+0000 7f94cfe00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f94d007de30 0x7f94d019dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:08.688 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.845+0000 7f94cfe00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f94d010e0d0 con 0x7f94d007de30 2024-03-19T03:22:08.688 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.845+0000 7f94cea00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f94d007de30 0x7f94d019dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:08.688 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.845+0000 7f94cea00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f94d007de30 0x7f94d019dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:45430/0 (socket says 172.21.15.117:45430) 2024-03-19T03:22:08.689 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.845+0000 7f94cea00700 1 -- 172.21.15.117:0/1218858347 learned_addr learned my addr 172.21.15.117:0/1218858347 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:08.689 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.845+0000 7f94cea00700 1 -- 172.21.15.117:0/1218858347 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94d019e210 con 0x7f94d007de30 2024-03-19T03:22:08.689 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.846+0000 7f94cea00700 1 --2- 172.21.15.117:0/1218858347 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f94d007de30 0x7f94d019dcd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f94c0000c00 tx=0x7f94c0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:08.689 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.846+0000 7f94c7400700 1 -- 172.21.15.117:0/1218858347 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94c0003b20 con 0x7f94d007de30 2024-03-19T03:22:08.689 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.846+0000 7f94c7400700 1 -- 172.21.15.117:0/1218858347 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f94c0003cc0 con 0x7f94d007de30 2024-03-19T03:22:08.689 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.846+0000 7f94cfe00700 1 -- 172.21.15.117:0/1218858347 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94d019e4a0 con 0x7f94d007de30 2024-03-19T03:22:08.689 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.846+0000 7f94c7400700 1 -- 172.21.15.117:0/1218858347 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94c0024910 con 0x7f94d007de30 2024-03-19T03:22:08.689 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.846+0000 7f94cfe00700 1 -- 172.21.15.117:0/1218858347 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94d01a1180 con 0x7f94d007de30 2024-03-19T03:22:08.689 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.847+0000 7f94c7400700 1 -- 172.21.15.117:0/1218858347 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f94c0022020 con 0x7f94d007de30 2024-03-19T03:22:08.689 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.847+0000 7f94cfe00700 1 -- 172.21.15.117:0/1218858347 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94d0063940 con 0x7f94d007de30 2024-03-19T03:22:08.689 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.848+0000 7f94c7400700 1 --2- 172.21.15.117:0/1218858347 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f94b4038f20 0x7f94b403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:08.689 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.848+0000 7f94c7400700 1 -- 172.21.15.117:0/1218858347 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f94c0011e00 con 0x7f94d007de30 2024-03-19T03:22:08.689 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.848+0000 7f94ce000700 1 --2- 172.21.15.117:0/1218858347 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f94b4038f20 0x7f94b403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:08.689 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.848+0000 7f94ce000700 1 --2- 172.21.15.117:0/1218858347 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f94b4038f20 0x7f94b403b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f94bc006fd0 tx=0x7f94bc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:08.689 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:07.857+0000 7f94c7400700 1 -- 172.21.15.117:0/1218858347 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f94c004caf0 con 0x7f94d007de30 2024-03-19T03:22:08.690 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:08.074+0000 7f94cfe00700 1 -- 172.21.15.117:0/1218858347 --> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f94d01043b0 con 0x7f94b4038f20 2024-03-19T03:22:08.690 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:08.075+0000 7f94c7400700 1 -- 172.21.15.117:0/1218858347 <== mgr.14120 v2:172.21.15.117:6800/2788067603 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f94d01043b0 con 0x7f94b4038f20 2024-03-19T03:22:08.690 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:08.077+0000 7f94cfe00700 1 -- 172.21.15.117:0/1218858347 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f94b4038f20 msgr2=0x7f94b403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:08.690 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:08.077+0000 7f94cfe00700 1 --2- 172.21.15.117:0/1218858347 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f94b4038f20 0x7f94b403b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f94bc006fd0 tx=0x7f94bc006e40 comp rx=0 tx=0).stop 2024-03-19T03:22:08.690 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:08.077+0000 7f94cfe00700 1 -- 172.21.15.117:0/1218858347 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f94d007de30 msgr2=0x7f94d019dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:08.690 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:08.077+0000 7f94cfe00700 1 --2- 172.21.15.117:0/1218858347 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f94d007de30 0x7f94d019dcd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f94c0000c00 tx=0x7f94c0011c10 comp rx=0 tx=0).stop 2024-03-19T03:22:08.690 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:08.077+0000 7f94cfe00700 1 -- 172.21.15.117:0/1218858347 shutdown_connections 2024-03-19T03:22:08.690 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:08.077+0000 7f94cfe00700 1 --2- 172.21.15.117:0/1218858347 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f94b4038f20 0x7f94b403b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:08.690 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:08.077+0000 7f94cfe00700 1 --2- 172.21.15.117:0/1218858347 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f94d007de30 0x7f94d019dcd0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:08.690 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:08.077+0000 7f94cfe00700 1 -- 172.21.15.117:0/1218858347 >> 172.21.15.117:0/1218858347 conn(0x7f94d0103080 msgr2=0x7f94d0103ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:08.690 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:08.077+0000 7f94cfe00700 1 -- 172.21.15.117:0/1218858347 shutdown_connections 2024-03-19T03:22:08.690 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:08.077+0000 7f94cfe00700 1 -- 172.21.15.117:0/1218858347 wait complete. 2024-03-19T03:22:08.690 INFO:teuthology.orchestra.run.smithi117.stdout:Generating ssh key... 2024-03-19T03:22:09.291 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:08 smithi117 ceph-mon[31838]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:22:10.513 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:10 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: Generating public/private rsa key pair. 2024-03-19T03:22:10.513 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:10 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: Your identification has been saved in /tmp/tmp75d27suv/key. 2024-03-19T03:22:10.514 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:10 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: Your public key has been saved in /tmp/tmp75d27suv/key.pub. 2024-03-19T03:22:10.514 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:10 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: The key fingerprint is: 2024-03-19T03:22:10.514 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:10 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: SHA256:S+NrGwmsNznCdttXvy8NYNO+8AMHyZsgAGSwPLcmT18 ceph-a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:22:10.514 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:10 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: The key's randomart image is: 2024-03-19T03:22:10.514 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:10 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: +---[RSA 3072]----+ 2024-03-19T03:22:10.514 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:10 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: | .o+. | 2024-03-19T03:22:10.514 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:10 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: | . o . | 2024-03-19T03:22:10.514 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:10 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: | + . . . o | 2024-03-19T03:22:10.514 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:10 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: | o o . . B . | 2024-03-19T03:22:10.514 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:10 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: | . + o E. o B | 2024-03-19T03:22:10.514 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:10 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: | * o * + * + | 2024-03-19T03:22:10.514 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:10 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: | * B = . * + | 2024-03-19T03:22:10.514 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:10 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: | . + =.o. * .| 2024-03-19T03:22:10.515 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:10 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: | ..+o .=.| 2024-03-19T03:22:10.515 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:10 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: +----[SHA256]-----+ 2024-03-19T03:22:10.819 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.290+0000 7f5ae3200700 1 Processor -- start 2024-03-19T03:22:10.824 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.290+0000 7f5ae3200700 1 -- start start 2024-03-19T03:22:10.824 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.291+0000 7f5ae3200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae407ddf0 0x7f5ae407c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:10.824 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.291+0000 7f5ae3200700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae407c820 con 0x7f5ae407ddf0 2024-03-19T03:22:10.824 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.291+0000 7f5ae1e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae407ddf0 0x7f5ae407c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:10.824 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.291+0000 7f5ae1e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae407ddf0 0x7f5ae407c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:45442/0 (socket says 172.21.15.117:45442) 2024-03-19T03:22:10.824 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.291+0000 7f5ae1e00700 1 -- 172.21.15.117:0/3618384699 learned_addr learned my addr 172.21.15.117:0/3618384699 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:10.824 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.292+0000 7f5ae1e00700 1 -- 172.21.15.117:0/3618384699 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae407c9a0 con 0x7f5ae407ddf0 2024-03-19T03:22:10.825 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.292+0000 7f5ae1e00700 1 --2- 172.21.15.117:0/3618384699 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae407ddf0 0x7f5ae407c250 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5ad40098a0 tx=0x7f5ad4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa3ad8675217a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:10.825 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.292+0000 7f5ae0a00700 1 -- 172.21.15.117:0/3618384699 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ad4004160 con 0x7f5ae407ddf0 2024-03-19T03:22:10.825 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.292+0000 7f5ae0a00700 1 -- 172.21.15.117:0/3618384699 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5ad4004300 con 0x7f5ae407ddf0 2024-03-19T03:22:10.825 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.292+0000 7f5ae0a00700 1 -- 172.21.15.117:0/3618384699 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ad4011b50 con 0x7f5ae407ddf0 2024-03-19T03:22:10.825 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.293+0000 7f5ae3200700 1 -- 172.21.15.117:0/3618384699 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae407ddf0 msgr2=0x7f5ae407c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:10.825 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.293+0000 7f5ae3200700 1 --2- 172.21.15.117:0/3618384699 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae407ddf0 0x7f5ae407c250 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5ad40098a0 tx=0x7f5ad4009bb0 comp rx=0 tx=0).stop 2024-03-19T03:22:10.825 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.293+0000 7f5ae3200700 1 -- 172.21.15.117:0/3618384699 shutdown_connections 2024-03-19T03:22:10.825 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.293+0000 7f5ae3200700 1 --2- 172.21.15.117:0/3618384699 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae407ddf0 0x7f5ae407c250 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:10.825 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.293+0000 7f5ae3200700 1 -- 172.21.15.117:0/3618384699 >> 172.21.15.117:0/3618384699 conn(0x7f5ae4103080 msgr2=0x7f5ae4105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:10.825 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.293+0000 7f5ae3200700 1 -- 172.21.15.117:0/3618384699 shutdown_connections 2024-03-19T03:22:10.825 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.293+0000 7f5ae3200700 1 -- 172.21.15.117:0/3618384699 wait complete. 2024-03-19T03:22:10.825 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.294+0000 7f5ae3200700 1 Processor -- start 2024-03-19T03:22:10.825 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.295+0000 7f5ae3200700 1 -- start start 2024-03-19T03:22:10.825 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.295+0000 7f5ae3200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae407ddf0 0x7f5ae419dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:10.826 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.295+0000 7f5ae3200700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae410e0b0 con 0x7f5ae407ddf0 2024-03-19T03:22:10.826 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.295+0000 7f5ae1e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae407ddf0 0x7f5ae419dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:10.826 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.295+0000 7f5ae1e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae407ddf0 0x7f5ae419dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:45448/0 (socket says 172.21.15.117:45448) 2024-03-19T03:22:10.826 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.295+0000 7f5ae1e00700 1 -- 172.21.15.117:0/4250715163 learned_addr learned my addr 172.21.15.117:0/4250715163 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:10.826 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.296+0000 7f5ae1e00700 1 -- 172.21.15.117:0/4250715163 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae419e1e0 con 0x7f5ae407ddf0 2024-03-19T03:22:10.826 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.296+0000 7f5ae1e00700 1 --2- 172.21.15.117:0/4250715163 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae407ddf0 0x7f5ae419dca0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4011b60 tx=0x7f5ad4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:10.826 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.296+0000 7f5adaa00700 1 -- 172.21.15.117:0/4250715163 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ad4003b20 con 0x7f5ae407ddf0 2024-03-19T03:22:10.826 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.296+0000 7f5adaa00700 1 -- 172.21.15.117:0/4250715163 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5ad4003cc0 con 0x7f5ae407ddf0 2024-03-19T03:22:10.826 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.296+0000 7f5ae3200700 1 -- 172.21.15.117:0/4250715163 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ae419e470 con 0x7f5ae407ddf0 2024-03-19T03:22:10.826 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.296+0000 7f5adaa00700 1 -- 172.21.15.117:0/4250715163 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ad4024840 con 0x7f5ae407ddf0 2024-03-19T03:22:10.826 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.296+0000 7f5ae3200700 1 -- 172.21.15.117:0/4250715163 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ae41a1160 con 0x7f5ae407ddf0 2024-03-19T03:22:10.826 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.297+0000 7f5adaa00700 1 -- 172.21.15.117:0/4250715163 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f5ad4022020 con 0x7f5ae407ddf0 2024-03-19T03:22:10.826 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.298+0000 7f5acfe00700 1 -- 172.21.15.117:0/4250715163 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ae4063940 con 0x7f5ae407ddf0 2024-03-19T03:22:10.827 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.298+0000 7f5adaa00700 1 --2- 172.21.15.117:0/4250715163 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f5ac8038f20 0x7f5ac803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:10.827 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.298+0000 7f5adaa00700 1 -- 172.21.15.117:0/4250715163 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5ad4011e00 con 0x7f5ae407ddf0 2024-03-19T03:22:10.827 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.298+0000 7f5ae1400700 1 --2- 172.21.15.117:0/4250715163 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f5ac8038f20 0x7f5ac803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:10.827 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.299+0000 7f5ae1400700 1 --2- 172.21.15.117:0/4250715163 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f5ac8038f20 0x7f5ac803b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0006fd0 tx=0x7f5ad0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:10.827 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.304+0000 7f5adaa00700 1 -- 172.21.15.117:0/4250715163 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5ad404cd90 con 0x7f5ae407ddf0 2024-03-19T03:22:10.827 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:09.524+0000 7f5acfe00700 1 -- 172.21.15.117:0/4250715163 --> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f5ae40630b0 con 0x7f5ac8038f20 2024-03-19T03:22:10.827 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:10.199+0000 7f5adaa00700 1 -- 172.21.15.117:0/4250715163 <== mgr.14120 v2:172.21.15.117:6800/2788067603 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5ae40630b0 con 0x7f5ac8038f20 2024-03-19T03:22:10.827 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:10.202+0000 7f5acfe00700 1 -- 172.21.15.117:0/4250715163 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f5ac8038f20 msgr2=0x7f5ac803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:10.827 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:10.202+0000 7f5acfe00700 1 --2- 172.21.15.117:0/4250715163 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f5ac8038f20 0x7f5ac803b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0006fd0 tx=0x7f5ad0006e40 comp rx=0 tx=0).stop 2024-03-19T03:22:10.827 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:10.202+0000 7f5acfe00700 1 -- 172.21.15.117:0/4250715163 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae407ddf0 msgr2=0x7f5ae419dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:10.827 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:10.202+0000 7f5acfe00700 1 --2- 172.21.15.117:0/4250715163 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae407ddf0 0x7f5ae419dca0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4011b60 tx=0x7f5ad4011c40 comp rx=0 tx=0).stop 2024-03-19T03:22:10.827 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:10.202+0000 7f5acfe00700 1 -- 172.21.15.117:0/4250715163 shutdown_connections 2024-03-19T03:22:10.827 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:10.202+0000 7f5acfe00700 1 --2- 172.21.15.117:0/4250715163 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f5ac8038f20 0x7f5ac803b3e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:10.827 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:10.202+0000 7f5acfe00700 1 --2- 172.21.15.117:0/4250715163 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae407ddf0 0x7f5ae419dca0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:10.828 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:10.202+0000 7f5acfe00700 1 -- 172.21.15.117:0/4250715163 >> 172.21.15.117:0/4250715163 conn(0x7f5ae4103080 msgr2=0x7f5ae4103c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:10.828 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:10.202+0000 7f5acfe00700 1 -- 172.21.15.117:0/4250715163 shutdown_connections 2024-03-19T03:22:10.828 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:10.202+0000 7f5acfe00700 1 -- 172.21.15.117:0/4250715163 wait complete. 2024-03-19T03:22:11.379 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:11 smithi117 ceph-mon[31838]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:22:11.379 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:11 smithi117 ceph-mon[31838]: Generating ssh key... 2024-03-19T03:22:11.379 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:11 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' 2024-03-19T03:22:11.379 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:11 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' 2024-03-19T03:22:12.270 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUgcFuirggaJ0n2PfVLR57W9HZ8bjQ6pWJJ1sctYDlL6TJkJZYjU+l1tNatnr2YBM65g1XMG0zwxmbALvvbLRgIz50e0F3JFIq6uo2aNVGc9mdF8ZcgZSumbefUKJKHFWcfLemYOYOGdokGW09zyC5nGqjgG4u00uOyeQItp2VHYz04G4EOOjOVp/w5rPRGuhvAlpk/X/WvrjWNNYxRn/xIRFnA9CNTmm+6D5aMTPfGv6um+k4MYaASXjpdY8F/v66lkBYJzkuxKzDjtMC1Stsk5v5FSmiFUVO3tYDZZFOPAZF0tSiwBHzbA7Ri7rR42oj9Fv6aLOfMUFa8HBK0t6kxGAtHTJPst2vtyr+Ewd1gydqwsTXoIBunaFLQdMMrY5GMb6l3DZWAhWF9uEF1W+qZmj7T44XF7KTVuQp6Dlvr+LnzOZBg+QnlRGmTmLtxIX7M9UjE1uGR4PIrjN/IxChq+UgoydBzU6oLRkHkEouITuKErisRQwUzR/aPhJ+aFU= ceph-a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:22:12.270 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.377+0000 7f5926000700 1 Processor -- start 2024-03-19T03:22:12.271 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.377+0000 7f5926000700 1 -- start start 2024-03-19T03:22:12.271 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.377+0000 7f5926000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5920081d00 0x7f5920082120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:12.271 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.377+0000 7f5926000700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f59200826f0 con 0x7f5920081d00 2024-03-19T03:22:12.271 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.378+0000 7f591f400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5920081d00 0x7f5920082120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:12.271 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.378+0000 7f591f400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5920081d00 0x7f5920082120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:45450/0 (socket says 172.21.15.117:45450) 2024-03-19T03:22:12.271 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.378+0000 7f591f400700 1 -- 172.21.15.117:0/3800588676 learned_addr learned my addr 172.21.15.117:0/3800588676 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:12.271 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.379+0000 7f591f400700 1 -- 172.21.15.117:0/3800588676 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f592010f590 con 0x7f5920081d00 2024-03-19T03:22:12.271 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.379+0000 7f591f400700 1 --2- 172.21.15.117:0/3800588676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5920081d00 0x7f5920082120 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f59100098a0 tx=0x7f5910009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=748eade6b4408f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:12.271 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.379+0000 7f591ea00700 1 -- 172.21.15.117:0/3800588676 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5910004160 con 0x7f5920081d00 2024-03-19T03:22:12.271 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.379+0000 7f591ea00700 1 -- 172.21.15.117:0/3800588676 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5910004300 con 0x7f5920081d00 2024-03-19T03:22:12.271 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.379+0000 7f591ea00700 1 -- 172.21.15.117:0/3800588676 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5910011b50 con 0x7f5920081d00 2024-03-19T03:22:12.271 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.380+0000 7f5926000700 1 -- 172.21.15.117:0/3800588676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5920081d00 msgr2=0x7f5920082120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:12.271 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.380+0000 7f5926000700 1 --2- 172.21.15.117:0/3800588676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5920081d00 0x7f5920082120 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f59100098a0 tx=0x7f5910009bb0 comp rx=0 tx=0).stop 2024-03-19T03:22:12.271 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.380+0000 7f5926000700 1 -- 172.21.15.117:0/3800588676 shutdown_connections 2024-03-19T03:22:12.272 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.380+0000 7f5926000700 1 --2- 172.21.15.117:0/3800588676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5920081d00 0x7f5920082120 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:12.272 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.380+0000 7f5926000700 1 -- 172.21.15.117:0/3800588676 >> 172.21.15.117:0/3800588676 conn(0x7f592007d310 msgr2=0x7f592007f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:12.272 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.380+0000 7f5926000700 1 -- 172.21.15.117:0/3800588676 shutdown_connections 2024-03-19T03:22:12.272 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.380+0000 7f5926000700 1 -- 172.21.15.117:0/3800588676 wait complete. 2024-03-19T03:22:12.272 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.381+0000 7f5926000700 1 Processor -- start 2024-03-19T03:22:12.272 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.381+0000 7f5926000700 1 -- start start 2024-03-19T03:22:12.272 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.382+0000 7f5926000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5920081d00 0x7f59201a2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:12.272 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.382+0000 7f5926000700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f59201127c0 con 0x7f5920081d00 2024-03-19T03:22:12.272 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.382+0000 7f591f400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5920081d00 0x7f59201a2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:12.272 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.382+0000 7f591f400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5920081d00 0x7f59201a2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:45454/0 (socket says 172.21.15.117:45454) 2024-03-19T03:22:12.272 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.382+0000 7f591f400700 1 -- 172.21.15.117:0/2131400896 learned_addr learned my addr 172.21.15.117:0/2131400896 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:12.272 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.382+0000 7f591f400700 1 -- 172.21.15.117:0/2131400896 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59201a27c0 con 0x7f5920081d00 2024-03-19T03:22:12.272 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.383+0000 7f591f400700 1 --2- 172.21.15.117:0/2131400896 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5920081d00 0x7f59201a2280 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5910011670 tx=0x7f5910011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:12.272 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.383+0000 7f591cc00700 1 -- 172.21.15.117:0/2131400896 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59100039c0 con 0x7f5920081d00 2024-03-19T03:22:12.273 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.383+0000 7f591cc00700 1 -- 172.21.15.117:0/2131400896 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5910011e50 con 0x7f5920081d00 2024-03-19T03:22:12.273 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.383+0000 7f5926000700 1 -- 172.21.15.117:0/2131400896 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59201a2a50 con 0x7f5920081d00 2024-03-19T03:22:12.273 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.383+0000 7f591cc00700 1 -- 172.21.15.117:0/2131400896 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f591001a5c0 con 0x7f5920081d00 2024-03-19T03:22:12.273 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.383+0000 7f5926000700 1 -- 172.21.15.117:0/2131400896 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59201a5740 con 0x7f5920081d00 2024-03-19T03:22:12.273 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.384+0000 7f591cc00700 1 -- 172.21.15.117:0/2131400896 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f5910024430 con 0x7f5920081d00 2024-03-19T03:22:12.273 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.384+0000 7f5926000700 1 -- 172.21.15.117:0/2131400896 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5920112940 con 0x7f5920081d00 2024-03-19T03:22:12.273 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.385+0000 7f591cc00700 1 --2- 172.21.15.117:0/2131400896 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f5904038ed0 0x7f590403b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:12.273 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.385+0000 7f591cc00700 1 -- 172.21.15.117:0/2131400896 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f591004d3d0 con 0x7f5920081d00 2024-03-19T03:22:12.273 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.385+0000 7f5917e00700 1 --2- 172.21.15.117:0/2131400896 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f5904038ed0 0x7f590403b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:12.273 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.386+0000 7f5917e00700 1 --2- 172.21.15.117:0/2131400896 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f5904038ed0 0x7f590403b390 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5908006fd0 tx=0x7f5908006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:12.273 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.390+0000 7f591cc00700 1 -- 172.21.15.117:0/2131400896 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f591004cd20 con 0x7f5920081d00 2024-03-19T03:22:12.273 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.610+0000 7f5926000700 1 -- 172.21.15.117:0/2131400896 --> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f592007e550 con 0x7f5904038ed0 2024-03-19T03:22:12.273 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.612+0000 7f591cc00700 1 -- 172.21.15.117:0/2131400896 <== mgr.14120 v2:172.21.15.117:6800/2788067603 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f592007e550 con 0x7f5904038ed0 2024-03-19T03:22:12.273 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.614+0000 7f5926000700 1 -- 172.21.15.117:0/2131400896 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f5904038ed0 msgr2=0x7f590403b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:12.273 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.614+0000 7f5926000700 1 --2- 172.21.15.117:0/2131400896 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f5904038ed0 0x7f590403b390 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5908006fd0 tx=0x7f5908006e40 comp rx=0 tx=0).stop 2024-03-19T03:22:12.274 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.614+0000 7f5926000700 1 -- 172.21.15.117:0/2131400896 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5920081d00 msgr2=0x7f59201a2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:12.274 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.614+0000 7f5926000700 1 --2- 172.21.15.117:0/2131400896 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5920081d00 0x7f59201a2280 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5910011670 tx=0x7f5910011b60 comp rx=0 tx=0).stop 2024-03-19T03:22:12.274 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.615+0000 7f5926000700 1 -- 172.21.15.117:0/2131400896 shutdown_connections 2024-03-19T03:22:12.274 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.615+0000 7f5926000700 1 --2- 172.21.15.117:0/2131400896 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f5904038ed0 0x7f590403b390 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:12.274 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.615+0000 7f5926000700 1 --2- 172.21.15.117:0/2131400896 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5920081d00 0x7f59201a2280 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:12.274 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.615+0000 7f5926000700 1 -- 172.21.15.117:0/2131400896 >> 172.21.15.117:0/2131400896 conn(0x7f592007d310 msgr2=0x7f592007de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:12.274 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.615+0000 7f5926000700 1 -- 172.21.15.117:0/2131400896 shutdown_connections 2024-03-19T03:22:12.274 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:11.615+0000 7f5926000700 1 -- 172.21.15.117:0/2131400896 wait complete. 2024-03-19T03:22:12.275 INFO:teuthology.orchestra.run.smithi117.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-19T03:22:12.275 INFO:teuthology.orchestra.run.smithi117.stdout:Adding key to root@localhost authorized_keys... 2024-03-19T03:22:12.275 INFO:teuthology.orchestra.run.smithi117.stdout:Adding host smithi117... 2024-03-19T03:22:12.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:12 smithi117 ceph-mon[31838]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:22:13.391 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:13 smithi117 ceph-mon[31838]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi117", "addr": "172.21.15.117", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:22:15.460 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:15 smithi117 ceph-mon[31838]: Deploying cephadm binary to smithi117 2024-03-19T03:22:20.091 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout Added host 'smithi117' with addr '172.21.15.117' 2024-03-19T03:22:20.092 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.848+0000 7f716dc00700 1 Processor -- start 2024-03-19T03:22:20.092 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.848+0000 7f716dc00700 1 -- start start 2024-03-19T03:22:20.092 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.849+0000 7f716dc00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f716810b430 0x7f716810b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:20.092 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.849+0000 7f716dc00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f716810bd90 con 0x7f716810b430 2024-03-19T03:22:20.092 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.849+0000 7f7166a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f716810b430 0x7f716810b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:20.092 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.849+0000 7f7166a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f716810b430 0x7f716810b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:45456/0 (socket says 172.21.15.117:45456) 2024-03-19T03:22:20.092 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.849+0000 7f7166a00700 1 -- 172.21.15.117:0/2610109513 learned_addr learned my addr 172.21.15.117:0/2610109513 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:20.092 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.850+0000 7f7166a00700 1 -- 172.21.15.117:0/2610109513 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7168109d00 con 0x7f716810b430 2024-03-19T03:22:20.092 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.850+0000 7f7166a00700 1 --2- 172.21.15.117:0/2610109513 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f716810b430 0x7f716810b850 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f71580098a0 tx=0x7f7158009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17f6dd0a40789ec5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:20.092 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.850+0000 7f7165600700 1 -- 172.21.15.117:0/2610109513 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7158004160 con 0x7f716810b430 2024-03-19T03:22:20.092 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.850+0000 7f7165600700 1 -- 172.21.15.117:0/2610109513 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7158004300 con 0x7f716810b430 2024-03-19T03:22:20.092 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.851+0000 7f7165600700 1 -- 172.21.15.117:0/2610109513 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7158011b50 con 0x7f716810b430 2024-03-19T03:22:20.093 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.851+0000 7f716dc00700 1 -- 172.21.15.117:0/2610109513 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f716810b430 msgr2=0x7f716810b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:20.093 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.851+0000 7f716dc00700 1 --2- 172.21.15.117:0/2610109513 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f716810b430 0x7f716810b850 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f71580098a0 tx=0x7f7158009bb0 comp rx=0 tx=0).stop 2024-03-19T03:22:20.093 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.851+0000 7f716dc00700 1 -- 172.21.15.117:0/2610109513 shutdown_connections 2024-03-19T03:22:20.093 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.851+0000 7f716dc00700 1 --2- 172.21.15.117:0/2610109513 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f716810b430 0x7f716810b850 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:20.093 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.851+0000 7f716dc00700 1 -- 172.21.15.117:0/2610109513 >> 172.21.15.117:0/2610109513 conn(0x7f716807d3c0 msgr2=0x7f716807d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:20.093 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.851+0000 7f716dc00700 1 -- 172.21.15.117:0/2610109513 shutdown_connections 2024-03-19T03:22:20.093 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.851+0000 7f716dc00700 1 -- 172.21.15.117:0/2610109513 wait complete. 2024-03-19T03:22:20.093 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.852+0000 7f716dc00700 1 Processor -- start 2024-03-19T03:22:20.093 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.853+0000 7f716dc00700 1 -- start start 2024-03-19T03:22:20.093 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.853+0000 7f716dc00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f716810b430 0x7f716819de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:20.093 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.853+0000 7f716dc00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f716810e4a0 con 0x7f716810b430 2024-03-19T03:22:20.093 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.853+0000 7f7166a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f716810b430 0x7f716819de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:20.093 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.853+0000 7f7166a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f716810b430 0x7f716819de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:45464/0 (socket says 172.21.15.117:45464) 2024-03-19T03:22:20.093 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.853+0000 7f7166a00700 1 -- 172.21.15.117:0/527097747 learned_addr learned my addr 172.21.15.117:0/527097747 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:20.094 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.853+0000 7f7166a00700 1 -- 172.21.15.117:0/527097747 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f716819e350 con 0x7f716810b430 2024-03-19T03:22:20.094 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.854+0000 7f7166a00700 1 --2- 172.21.15.117:0/527097747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f716810b430 0x7f716819de10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7158011b60 tx=0x7f7158011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:20.094 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.854+0000 7f715f400700 1 -- 172.21.15.117:0/527097747 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7158003b20 con 0x7f716810b430 2024-03-19T03:22:20.094 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.854+0000 7f715f400700 1 -- 172.21.15.117:0/527097747 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7158003cc0 con 0x7f716810b430 2024-03-19T03:22:20.094 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.854+0000 7f715f400700 1 -- 172.21.15.117:0/527097747 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7158024840 con 0x7f716810b430 2024-03-19T03:22:20.094 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.854+0000 7f716dc00700 1 -- 172.21.15.117:0/527097747 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f716819e5e0 con 0x7f716810b430 2024-03-19T03:22:20.094 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.854+0000 7f716dc00700 1 -- 172.21.15.117:0/527097747 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71681a12d0 con 0x7f716810b430 2024-03-19T03:22:20.094 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.855+0000 7f716dc00700 1 -- 172.21.15.117:0/527097747 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7168063940 con 0x7f716810b430 2024-03-19T03:22:20.094 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.855+0000 7f715f400700 1 -- 172.21.15.117:0/527097747 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f7158022020 con 0x7f716810b430 2024-03-19T03:22:20.094 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.856+0000 7f715f400700 1 --2- 172.21.15.117:0/527097747 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f714c038f70 0x7f714c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:20.094 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.856+0000 7f715f400700 1 -- 172.21.15.117:0/527097747 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7158011e00 con 0x7f716810b430 2024-03-19T03:22:20.094 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.856+0000 7f7166000700 1 --2- 172.21.15.117:0/527097747 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f714c038f70 0x7f714c03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:20.094 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.857+0000 7f7166000700 1 --2- 172.21.15.117:0/527097747 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f714c038f70 0x7f714c03b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7154006fd0 tx=0x7f7154006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:20.095 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:12.861+0000 7f715f400700 1 -- 172.21.15.117:0/527097747 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f715800b3e0 con 0x7f716810b430 2024-03-19T03:22:20.095 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:13.084+0000 7f716dc00700 1 -- 172.21.15.117:0/527097747 --> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi117", "addr": "172.21.15.117", "target": ["mon-mgr", ""]}) -- 0x7f7168108370 con 0x7f714c038f70 2024-03-19T03:22:20.095 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:19.370+0000 7f715f400700 1 -- 172.21.15.117:0/527097747 <== mgr.14120 v2:172.21.15.117:6800/2788067603 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f7168108370 con 0x7f714c038f70 2024-03-19T03:22:20.095 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:19.374+0000 7f716dc00700 1 -- 172.21.15.117:0/527097747 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f714c038f70 msgr2=0x7f714c03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:20.095 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:19.374+0000 7f716dc00700 1 --2- 172.21.15.117:0/527097747 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f714c038f70 0x7f714c03b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7154006fd0 tx=0x7f7154006e40 comp rx=0 tx=0).stop 2024-03-19T03:22:20.095 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:19.374+0000 7f716dc00700 1 -- 172.21.15.117:0/527097747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f716810b430 msgr2=0x7f716819de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:20.095 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:19.374+0000 7f716dc00700 1 --2- 172.21.15.117:0/527097747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f716810b430 0x7f716819de10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7158011b60 tx=0x7f7158011c40 comp rx=0 tx=0).stop 2024-03-19T03:22:20.095 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:19.374+0000 7f716dc00700 1 -- 172.21.15.117:0/527097747 shutdown_connections 2024-03-19T03:22:20.095 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:19.374+0000 7f716dc00700 1 --2- 172.21.15.117:0/527097747 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f714c038f70 0x7f714c03b430 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:20.095 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:19.374+0000 7f716dc00700 1 --2- 172.21.15.117:0/527097747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f716810b430 0x7f716819de10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:20.095 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:19.374+0000 7f716dc00700 1 -- 172.21.15.117:0/527097747 >> 172.21.15.117:0/527097747 conn(0x7f716807d3c0 msgr2=0x7f7168107c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:20.095 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:19.375+0000 7f716dc00700 1 -- 172.21.15.117:0/527097747 shutdown_connections 2024-03-19T03:22:20.095 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:19.375+0000 7f716dc00700 1 -- 172.21.15.117:0/527097747 wait complete. 2024-03-19T03:22:20.095 INFO:teuthology.orchestra.run.smithi117.stdout:Deploying unmanaged mon service... 2024-03-19T03:22:20.710 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:20 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' 2024-03-19T03:22:20.710 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:20 smithi117 ceph-mon[31838]: Added host smithi117 2024-03-19T03:22:20.710 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:20 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:22:22.338 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-19T03:22:22.339 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.878+0000 7f0141a00700 1 Processor -- start 2024-03-19T03:22:22.339 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.878+0000 7f0141a00700 1 -- start start 2024-03-19T03:22:22.339 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.879+0000 7f0141a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f013c10a0e0 0x7f013c10a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:22.339 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.879+0000 7f0141a00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f013c10aa40 con 0x7f013c10a0e0 2024-03-19T03:22:22.339 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.879+0000 7f013aa00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f013c10a0e0 0x7f013c10a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:22.339 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.879+0000 7f013aa00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f013c10a0e0 0x7f013c10a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:55016/0 (socket says 172.21.15.117:55016) 2024-03-19T03:22:22.340 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.880+0000 7f013aa00700 1 -- 172.21.15.117:0/572119531 learned_addr learned my addr 172.21.15.117:0/572119531 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:22.340 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.880+0000 7f013aa00700 1 -- 172.21.15.117:0/572119531 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f013c10b080 con 0x7f013c10a0e0 2024-03-19T03:22:22.340 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.880+0000 7f013aa00700 1 --2- 172.21.15.117:0/572119531 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f013c10a0e0 0x7f013c10a500 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f012c0098a0 tx=0x7f012c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aae70ddba2e41071 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:22.340 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.881+0000 7f0139600700 1 -- 172.21.15.117:0/572119531 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f012c004160 con 0x7f013c10a0e0 2024-03-19T03:22:22.340 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.881+0000 7f0139600700 1 -- 172.21.15.117:0/572119531 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f012c004300 con 0x7f013c10a0e0 2024-03-19T03:22:22.340 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.881+0000 7f0139600700 1 -- 172.21.15.117:0/572119531 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f012c011b50 con 0x7f013c10a0e0 2024-03-19T03:22:22.340 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.881+0000 7f0141a00700 1 -- 172.21.15.117:0/572119531 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f013c10a0e0 msgr2=0x7f013c10a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:22.341 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.881+0000 7f0141a00700 1 --2- 172.21.15.117:0/572119531 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f013c10a0e0 0x7f013c10a500 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f012c0098a0 tx=0x7f012c009bb0 comp rx=0 tx=0).stop 2024-03-19T03:22:22.341 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.881+0000 7f0141a00700 1 -- 172.21.15.117:0/572119531 shutdown_connections 2024-03-19T03:22:22.341 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.881+0000 7f0141a00700 1 --2- 172.21.15.117:0/572119531 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f013c10a0e0 0x7f013c10a500 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:22.341 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.881+0000 7f0141a00700 1 -- 172.21.15.117:0/572119531 >> 172.21.15.117:0/572119531 conn(0x7f013c07d310 msgr2=0x7f013c07d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:22.341 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.882+0000 7f0141a00700 1 -- 172.21.15.117:0/572119531 shutdown_connections 2024-03-19T03:22:22.341 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.882+0000 7f0141a00700 1 -- 172.21.15.117:0/572119531 wait complete. 2024-03-19T03:22:22.341 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.883+0000 7f0141a00700 1 Processor -- start 2024-03-19T03:22:22.342 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.883+0000 7f0141a00700 1 -- start start 2024-03-19T03:22:22.342 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.883+0000 7f0141a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f013c10a0e0 0x7f013c19de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:22.342 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.883+0000 7f0141a00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f013c10e340 con 0x7f013c10a0e0 2024-03-19T03:22:22.342 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.883+0000 7f013aa00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f013c10a0e0 0x7f013c19de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:22.342 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.883+0000 7f013aa00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f013c10a0e0 0x7f013c19de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:55028/0 (socket says 172.21.15.117:55028) 2024-03-19T03:22:22.342 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.883+0000 7f013aa00700 1 -- 172.21.15.117:0/2242221305 learned_addr learned my addr 172.21.15.117:0/2242221305 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:22.342 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.884+0000 7f013aa00700 1 -- 172.21.15.117:0/2242221305 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f013c19e390 con 0x7f013c10a0e0 2024-03-19T03:22:22.342 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.884+0000 7f013aa00700 1 --2- 172.21.15.117:0/2242221305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f013c10a0e0 0x7f013c19de50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f012c011b60 tx=0x7f012c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:22.342 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.884+0000 7f0133400700 1 -- 172.21.15.117:0/2242221305 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f012c003b20 con 0x7f013c10a0e0 2024-03-19T03:22:22.342 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.884+0000 7f0133400700 1 -- 172.21.15.117:0/2242221305 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f012c003cc0 con 0x7f013c10a0e0 2024-03-19T03:22:22.342 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.884+0000 7f0141a00700 1 -- 172.21.15.117:0/2242221305 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f013c19e620 con 0x7f013c10a0e0 2024-03-19T03:22:22.342 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.884+0000 7f0133400700 1 -- 172.21.15.117:0/2242221305 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f012c024840 con 0x7f013c10a0e0 2024-03-19T03:22:22.343 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.884+0000 7f0141a00700 1 -- 172.21.15.117:0/2242221305 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f013c1a1310 con 0x7f013c10a0e0 2024-03-19T03:22:22.343 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.885+0000 7f0133400700 1 -- 172.21.15.117:0/2242221305 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f012c022020 con 0x7f013c10a0e0 2024-03-19T03:22:22.343 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.886+0000 7f0141a00700 1 -- 172.21.15.117:0/2242221305 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f013c10e4c0 con 0x7f013c10a0e0 2024-03-19T03:22:22.343 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.886+0000 7f0133400700 1 --2- 172.21.15.117:0/2242221305 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f0120038f70 0x7f012003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:22.343 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.886+0000 7f0133400700 1 -- 172.21.15.117:0/2242221305 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f012c011e00 con 0x7f013c10a0e0 2024-03-19T03:22:22.343 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.886+0000 7f013a000700 1 --2- 172.21.15.117:0/2242221305 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f0120038f70 0x7f012003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:22.343 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.887+0000 7f013a000700 1 --2- 172.21.15.117:0/2242221305 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f0120038f70 0x7f012003b430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0128006fd0 tx=0x7f0128006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:22.343 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:20.892+0000 7f0133400700 1 -- 172.21.15.117:0/2242221305 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f012c00b3e0 con 0x7f013c10a0e0 2024-03-19T03:22:22.343 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:21.130+0000 7f0141a00700 1 -- 172.21.15.117:0/2242221305 --> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f013c1083e0 con 0x7f0120038f70 2024-03-19T03:22:22.343 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:21.137+0000 7f0133400700 1 -- 172.21.15.117:0/2242221305 <== mgr.14120 v2:172.21.15.117:6800/2788067603 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f013c1083e0 con 0x7f0120038f70 2024-03-19T03:22:22.343 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:21.139+0000 7f0141a00700 1 -- 172.21.15.117:0/2242221305 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f0120038f70 msgr2=0x7f012003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:22.343 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:21.139+0000 7f0141a00700 1 --2- 172.21.15.117:0/2242221305 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f0120038f70 0x7f012003b430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0128006fd0 tx=0x7f0128006e40 comp rx=0 tx=0).stop 2024-03-19T03:22:22.343 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:21.139+0000 7f0141a00700 1 -- 172.21.15.117:0/2242221305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f013c10a0e0 msgr2=0x7f013c19de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:22.344 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:21.139+0000 7f0141a00700 1 --2- 172.21.15.117:0/2242221305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f013c10a0e0 0x7f013c19de50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f012c011b60 tx=0x7f012c011c40 comp rx=0 tx=0).stop 2024-03-19T03:22:22.344 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:21.140+0000 7f0141a00700 1 -- 172.21.15.117:0/2242221305 shutdown_connections 2024-03-19T03:22:22.344 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:21.140+0000 7f0141a00700 1 --2- 172.21.15.117:0/2242221305 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f0120038f70 0x7f012003b430 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:22.344 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:21.140+0000 7f0141a00700 1 --2- 172.21.15.117:0/2242221305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f013c10a0e0 0x7f013c19de50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:22.344 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:21.140+0000 7f0141a00700 1 -- 172.21.15.117:0/2242221305 >> 172.21.15.117:0/2242221305 conn(0x7f013c07d310 msgr2=0x7f013c107cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:22.344 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:21.140+0000 7f0141a00700 1 -- 172.21.15.117:0/2242221305 shutdown_connections 2024-03-19T03:22:22.344 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:21.140+0000 7f0141a00700 1 -- 172.21.15.117:0/2242221305 wait complete. 2024-03-19T03:22:22.344 INFO:teuthology.orchestra.run.smithi117.stdout:Deploying unmanaged mgr service... 2024-03-19T03:22:22.479 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:22 smithi117 ceph-mon[31838]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:22:22.479 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:22 smithi117 ceph-mon[31838]: Saving service mon spec with placement count:5 2024-03-19T03:22:22.480 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:22 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' 2024-03-19T03:22:24.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:23 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' 2024-03-19T03:22:24.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:23 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' 2024-03-19T03:22:24.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:23 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' 2024-03-19T03:22:24.308 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-19T03:22:24.309 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.350+0000 7fe62c200700 1 Processor -- start 2024-03-19T03:22:24.309 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.351+0000 7fe62c200700 1 -- start start 2024-03-19T03:22:24.309 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.351+0000 7fe62c200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62407dd90 0x7fe62407c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:24.309 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.351+0000 7fe62c200700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe62407c7d0 con 0x7fe62407dd90 2024-03-19T03:22:24.309 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.352+0000 7fe629800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62407dd90 0x7fe62407c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:24.309 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.352+0000 7fe629800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62407dd90 0x7fe62407c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:55030/0 (socket says 172.21.15.117:55030) 2024-03-19T03:22:24.310 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.352+0000 7fe629800700 1 -- 172.21.15.117:0/1320223882 learned_addr learned my addr 172.21.15.117:0/1320223882 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:24.310 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.352+0000 7fe629800700 1 -- 172.21.15.117:0/1320223882 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe62407c950 con 0x7fe62407dd90 2024-03-19T03:22:24.310 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.353+0000 7fe629800700 1 --2- 172.21.15.117:0/1320223882 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62407dd90 0x7fe62407c290 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe6180098a0 tx=0x7fe618009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d60abbbddd2feaf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:24.312 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.353+0000 7fe623e00700 1 -- 172.21.15.117:0/1320223882 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe618004160 con 0x7fe62407dd90 2024-03-19T03:22:24.312 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.353+0000 7fe623e00700 1 -- 172.21.15.117:0/1320223882 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe618004300 con 0x7fe62407dd90 2024-03-19T03:22:24.312 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.353+0000 7fe623e00700 1 -- 172.21.15.117:0/1320223882 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe618011b20 con 0x7fe62407dd90 2024-03-19T03:22:24.312 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.354+0000 7fe62c200700 1 -- 172.21.15.117:0/1320223882 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62407dd90 msgr2=0x7fe62407c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:24.312 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.354+0000 7fe62c200700 1 --2- 172.21.15.117:0/1320223882 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62407dd90 0x7fe62407c290 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe6180098a0 tx=0x7fe618009bb0 comp rx=0 tx=0).stop 2024-03-19T03:22:24.312 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.354+0000 7fe62c200700 1 -- 172.21.15.117:0/1320223882 shutdown_connections 2024-03-19T03:22:24.312 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.354+0000 7fe62c200700 1 --2- 172.21.15.117:0/1320223882 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62407dd90 0x7fe62407c290 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:24.312 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.354+0000 7fe62c200700 1 -- 172.21.15.117:0/1320223882 >> 172.21.15.117:0/1320223882 conn(0x7fe624103310 msgr2=0x7fe624105750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:24.312 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.354+0000 7fe62c200700 1 -- 172.21.15.117:0/1320223882 shutdown_connections 2024-03-19T03:22:24.312 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.354+0000 7fe62c200700 1 -- 172.21.15.117:0/1320223882 wait complete. 2024-03-19T03:22:24.312 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.355+0000 7fe62c200700 1 Processor -- start 2024-03-19T03:22:24.313 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.355+0000 7fe62c200700 1 -- start start 2024-03-19T03:22:24.313 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.356+0000 7fe62c200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62407dd90 0x7fe62419ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:24.313 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.356+0000 7fe62c200700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe62410e310 con 0x7fe62407dd90 2024-03-19T03:22:24.313 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.356+0000 7fe629800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62407dd90 0x7fe62419ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:24.313 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.356+0000 7fe629800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62407dd90 0x7fe62419ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:55034/0 (socket says 172.21.15.117:55034) 2024-03-19T03:22:24.313 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.356+0000 7fe629800700 1 -- 172.21.15.117:0/2468989907 learned_addr learned my addr 172.21.15.117:0/2468989907 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:24.313 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.356+0000 7fe629800700 1 -- 172.21.15.117:0/2468989907 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe62419e2f0 con 0x7fe62407dd90 2024-03-19T03:22:24.313 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.357+0000 7fe629800700 1 --2- 172.21.15.117:0/2468989907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62407dd90 0x7fe62419ddb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe6180117e0 tx=0x7fe618011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:24.313 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.357+0000 7fe622000700 1 -- 172.21.15.117:0/2468989907 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe618003ba0 con 0x7fe62407dd90 2024-03-19T03:22:24.313 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.357+0000 7fe622000700 1 -- 172.21.15.117:0/2468989907 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe61801a910 con 0x7fe62407dd90 2024-03-19T03:22:24.313 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.357+0000 7fe622000700 1 -- 172.21.15.117:0/2468989907 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6180248f0 con 0x7fe62407dd90 2024-03-19T03:22:24.313 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.357+0000 7fe62c200700 1 -- 172.21.15.117:0/2468989907 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe62419e580 con 0x7fe62407dd90 2024-03-19T03:22:24.313 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.357+0000 7fe62c200700 1 -- 172.21.15.117:0/2468989907 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe62419e9e0 con 0x7fe62407dd90 2024-03-19T03:22:24.314 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.358+0000 7fe622000700 1 -- 172.21.15.117:0/2468989907 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fe618022020 con 0x7fe62407dd90 2024-03-19T03:22:24.314 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.359+0000 7fe62c200700 1 -- 172.21.15.117:0/2468989907 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe624063940 con 0x7fe62407dd90 2024-03-19T03:22:24.314 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.359+0000 7fe622000700 1 --2- 172.21.15.117:0/2468989907 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7fe60c038f20 0x7fe60c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:24.314 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.359+0000 7fe622000700 1 -- 172.21.15.117:0/2468989907 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe61804d3c0 con 0x7fe62407dd90 2024-03-19T03:22:24.314 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.359+0000 7fe628e00700 1 --2- 172.21.15.117:0/2468989907 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7fe60c038f20 0x7fe60c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:24.314 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.360+0000 7fe628e00700 1 --2- 172.21.15.117:0/2468989907 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7fe60c038f20 0x7fe60c03b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe610006fd0 tx=0x7fe610006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:24.314 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.365+0000 7fe622000700 1 -- 172.21.15.117:0/2468989907 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe61801ab70 con 0x7fe62407dd90 2024-03-19T03:22:24.314 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.605+0000 7fe62c200700 1 -- 172.21.15.117:0/2468989907 --> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fe6241045a0 con 0x7fe60c038f20 2024-03-19T03:22:24.314 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.613+0000 7fe622000700 1 -- 172.21.15.117:0/2468989907 <== mgr.14120 v2:172.21.15.117:6800/2788067603 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fe6241045a0 con 0x7fe60c038f20 2024-03-19T03:22:24.314 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.615+0000 7fe62c200700 1 -- 172.21.15.117:0/2468989907 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7fe60c038f20 msgr2=0x7fe60c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:24.314 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.615+0000 7fe62c200700 1 --2- 172.21.15.117:0/2468989907 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7fe60c038f20 0x7fe60c03b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe610006fd0 tx=0x7fe610006e40 comp rx=0 tx=0).stop 2024-03-19T03:22:24.314 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.615+0000 7fe62c200700 1 -- 172.21.15.117:0/2468989907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62407dd90 msgr2=0x7fe62419ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:24.315 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.615+0000 7fe62c200700 1 --2- 172.21.15.117:0/2468989907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62407dd90 0x7fe62419ddb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe6180117e0 tx=0x7fe618011ba0 comp rx=0 tx=0).stop 2024-03-19T03:22:24.315 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.616+0000 7fe62c200700 1 -- 172.21.15.117:0/2468989907 shutdown_connections 2024-03-19T03:22:24.315 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.616+0000 7fe62c200700 1 --2- 172.21.15.117:0/2468989907 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7fe60c038f20 0x7fe60c03b3e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:24.315 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.616+0000 7fe62c200700 1 --2- 172.21.15.117:0/2468989907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62407dd90 0x7fe62419ddb0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:24.315 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.616+0000 7fe62c200700 1 -- 172.21.15.117:0/2468989907 >> 172.21.15.117:0/2468989907 conn(0x7fe624103310 msgr2=0x7fe624103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:24.315 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.616+0000 7fe62c200700 1 -- 172.21.15.117:0/2468989907 shutdown_connections 2024-03-19T03:22:24.315 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:23.616+0000 7fe62c200700 1 -- 172.21.15.117:0/2468989907 wait complete. 2024-03-19T03:22:25.188 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:24 smithi117 ceph-mon[31838]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:22:25.189 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:24 smithi117 ceph-mon[31838]: Saving service mgr spec with placement count:2 2024-03-19T03:22:25.189 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:24 smithi117 ceph-mon[31838]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:22:25.867 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.924+0000 7fd9b4a00700 1 Processor -- start 2024-03-19T03:22:25.867 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.925+0000 7fd9b4a00700 1 -- start start 2024-03-19T03:22:25.867 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.925+0000 7fd9b4a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b010b430 0x7fd9b010b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:25.867 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.925+0000 7fd9b4a00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b010bd90 con 0x7fd9b010b430 2024-03-19T03:22:25.867 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.925+0000 7fd9ade00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b010b430 0x7fd9b010b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:25.867 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.925+0000 7fd9ade00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b010b430 0x7fd9b010b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:55042/0 (socket says 172.21.15.117:55042) 2024-03-19T03:22:25.867 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.925+0000 7fd9ade00700 1 -- 172.21.15.117:0/576586302 learned_addr learned my addr 172.21.15.117:0/576586302 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:25.868 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.926+0000 7fd9ade00700 1 -- 172.21.15.117:0/576586302 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9b0109d00 con 0x7fd9b010b430 2024-03-19T03:22:25.871 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.926+0000 7fd9ade00700 1 --2- 172.21.15.117:0/576586302 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b010b430 0x7fd9b010b850 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd9a00098a0 tx=0x7fd9a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffaeca605a1ac07 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:25.871 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.926+0000 7fd9aca00700 1 -- 172.21.15.117:0/576586302 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9a0004160 con 0x7fd9b010b430 2024-03-19T03:22:25.871 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.926+0000 7fd9aca00700 1 -- 172.21.15.117:0/576586302 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9a0004300 con 0x7fd9b010b430 2024-03-19T03:22:25.871 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.926+0000 7fd9aca00700 1 -- 172.21.15.117:0/576586302 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9a0011b20 con 0x7fd9b010b430 2024-03-19T03:22:25.871 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.927+0000 7fd9b4a00700 1 -- 172.21.15.117:0/576586302 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b010b430 msgr2=0x7fd9b010b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:25.871 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.927+0000 7fd9b4a00700 1 --2- 172.21.15.117:0/576586302 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b010b430 0x7fd9b010b850 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd9a00098a0 tx=0x7fd9a0009bb0 comp rx=0 tx=0).stop 2024-03-19T03:22:25.871 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.928+0000 7fd9b4a00700 1 -- 172.21.15.117:0/576586302 shutdown_connections 2024-03-19T03:22:25.871 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.928+0000 7fd9b4a00700 1 --2- 172.21.15.117:0/576586302 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b010b430 0x7fd9b010b850 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:25.871 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.928+0000 7fd9b4a00700 1 -- 172.21.15.117:0/576586302 >> 172.21.15.117:0/576586302 conn(0x7fd9b007d3c0 msgr2=0x7fd9b007d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:25.871 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.928+0000 7fd9b4a00700 1 -- 172.21.15.117:0/576586302 shutdown_connections 2024-03-19T03:22:25.872 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.928+0000 7fd9b4a00700 1 -- 172.21.15.117:0/576586302 wait complete. 2024-03-19T03:22:25.872 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.929+0000 7fd9b4a00700 1 Processor -- start 2024-03-19T03:22:25.872 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.929+0000 7fd9b4a00700 1 -- start start 2024-03-19T03:22:25.872 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.929+0000 7fd9b4a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b010b430 0x7fd9b019ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:25.872 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.929+0000 7fd9b4a00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b010e4a0 con 0x7fd9b010b430 2024-03-19T03:22:25.872 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.930+0000 7fd9ade00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b010b430 0x7fd9b019ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:25.872 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.930+0000 7fd9ade00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b010b430 0x7fd9b019ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:55050/0 (socket says 172.21.15.117:55050) 2024-03-19T03:22:25.872 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.930+0000 7fd9ade00700 1 -- 172.21.15.117:0/4262201179 learned_addr learned my addr 172.21.15.117:0/4262201179 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:25.872 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.930+0000 7fd9ade00700 1 -- 172.21.15.117:0/4262201179 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9b019e2f0 con 0x7fd9b010b430 2024-03-19T03:22:25.872 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.930+0000 7fd9ade00700 1 --2- 172.21.15.117:0/4262201179 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b010b430 0x7fd9b019ddb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd9a00117e0 tx=0x7fd9a0011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:25.872 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.930+0000 7fd9a6a00700 1 -- 172.21.15.117:0/4262201179 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9a001a430 con 0x7fd9b010b430 2024-03-19T03:22:25.872 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.931+0000 7fd9a6a00700 1 -- 172.21.15.117:0/4262201179 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9a001aab0 con 0x7fd9b010b430 2024-03-19T03:22:25.872 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.931+0000 7fd9a6a00700 1 -- 172.21.15.117:0/4262201179 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9a0024a60 con 0x7fd9b010b430 2024-03-19T03:22:25.872 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.931+0000 7fd9b4a00700 1 -- 172.21.15.117:0/4262201179 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9b019e580 con 0x7fd9b010b430 2024-03-19T03:22:25.873 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.931+0000 7fd9b4a00700 1 -- 172.21.15.117:0/4262201179 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9b01a1230 con 0x7fd9b010b430 2024-03-19T03:22:25.873 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.932+0000 7fd9b4a00700 1 -- 172.21.15.117:0/4262201179 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9b0063940 con 0x7fd9b010b430 2024-03-19T03:22:25.873 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.932+0000 7fd9a6a00700 1 -- 172.21.15.117:0/4262201179 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fd9a0022020 con 0x7fd9b010b430 2024-03-19T03:22:25.873 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.933+0000 7fd9a6a00700 1 --2- 172.21.15.117:0/4262201179 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7fd994038f70 0x7fd99403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:25.874 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.933+0000 7fd9a6a00700 1 -- 172.21.15.117:0/4262201179 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd9a004d630 con 0x7fd9b010b430 2024-03-19T03:22:25.875 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.934+0000 7fd9ad400700 1 --2- 172.21.15.117:0/4262201179 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7fd994038f70 0x7fd99403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:25.875 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.934+0000 7fd9ad400700 1 --2- 172.21.15.117:0/4262201179 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7fd994038f70 0x7fd99403b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd99c006fd0 tx=0x7fd99c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:25.875 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:24.938+0000 7fd9a6a00700 1 -- 172.21.15.117:0/4262201179 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd9a00042a0 con 0x7fd9b010b430 2024-03-19T03:22:25.875 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:25.187+0000 7fd9b4a00700 1 -- 172.21.15.117:0/4262201179 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fd9b019e9e0 con 0x7fd9b010b430 2024-03-19T03:22:25.875 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:25.193+0000 7fd9a6a00700 1 -- 172.21.15.117:0/4262201179 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7fd9a002ddb0 con 0x7fd9b010b430 2024-03-19T03:22:25.875 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:25.196+0000 7fd9b4a00700 1 -- 172.21.15.117:0/4262201179 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7fd994038f70 msgr2=0x7fd99403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:25.875 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:25.196+0000 7fd9b4a00700 1 --2- 172.21.15.117:0/4262201179 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7fd994038f70 0x7fd99403b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd99c006fd0 tx=0x7fd99c006e40 comp rx=0 tx=0).stop 2024-03-19T03:22:25.875 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:25.196+0000 7fd9b4a00700 1 -- 172.21.15.117:0/4262201179 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b010b430 msgr2=0x7fd9b019ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:25.875 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:25.196+0000 7fd9b4a00700 1 --2- 172.21.15.117:0/4262201179 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b010b430 0x7fd9b019ddb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd9a00117e0 tx=0x7fd9a0011ce0 comp rx=0 tx=0).stop 2024-03-19T03:22:25.875 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:25.196+0000 7fd9b4a00700 1 -- 172.21.15.117:0/4262201179 shutdown_connections 2024-03-19T03:22:25.875 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:25.196+0000 7fd9b4a00700 1 --2- 172.21.15.117:0/4262201179 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7fd994038f70 0x7fd99403b430 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:25.875 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:25.196+0000 7fd9b4a00700 1 --2- 172.21.15.117:0/4262201179 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b010b430 0x7fd9b019ddb0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:25.875 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:25.196+0000 7fd9b4a00700 1 -- 172.21.15.117:0/4262201179 >> 172.21.15.117:0/4262201179 conn(0x7fd9b007d3c0 msgr2=0x7fd9b0107c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:25.875 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:25.196+0000 7fd9b4a00700 1 -- 172.21.15.117:0/4262201179 shutdown_connections 2024-03-19T03:22:25.875 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:25.196+0000 7fd9b4a00700 1 -- 172.21.15.117:0/4262201179 wait complete. 2024-03-19T03:22:26.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:26 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/4262201179' entity='client.admin' 2024-03-19T03:22:27.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:27 smithi117 ceph-mon[31838]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:22:27.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:27 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' 2024-03-19T03:22:27.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:27 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/4111542820' entity='client.admin' 2024-03-19T03:22:27.638 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.737+0000 7f79e0a00700 1 Processor -- start 2024-03-19T03:22:27.639 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.737+0000 7f79e0a00700 1 -- start start 2024-03-19T03:22:27.639 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.737+0000 7f79e0a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79dc07e700 0x7f79dc07cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:27.639 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.737+0000 7f79e0a00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f79dc07d130 con 0x7f79dc07e700 2024-03-19T03:22:27.639 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.738+0000 7f79d9e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79dc07e700 0x7f79dc07cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:27.639 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.738+0000 7f79d9e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79dc07e700 0x7f79dc07cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60840/0 (socket says 172.21.15.117:60840) 2024-03-19T03:22:27.639 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.738+0000 7f79d9e00700 1 -- 172.21.15.117:0/367061673 learned_addr learned my addr 172.21.15.117:0/367061673 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:27.639 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.738+0000 7f79d9e00700 1 -- 172.21.15.117:0/367061673 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79dc07d2b0 con 0x7f79dc07e700 2024-03-19T03:22:27.639 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.738+0000 7f79d9e00700 1 --2- 172.21.15.117:0/367061673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79dc07e700 0x7f79dc07cb60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f79cc0098a0 tx=0x7f79cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e8ccca5a16cdd72 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:27.639 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.739+0000 7f79d8a00700 1 -- 172.21.15.117:0/367061673 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79cc004160 con 0x7f79dc07e700 2024-03-19T03:22:27.639 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.739+0000 7f79d8a00700 1 -- 172.21.15.117:0/367061673 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79cc004300 con 0x7f79dc07e700 2024-03-19T03:22:27.639 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.739+0000 7f79d8a00700 1 -- 172.21.15.117:0/367061673 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79cc011b20 con 0x7f79dc07e700 2024-03-19T03:22:27.639 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.740+0000 7f79e0a00700 1 -- 172.21.15.117:0/367061673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79dc07e700 msgr2=0x7f79dc07cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:27.639 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.740+0000 7f79e0a00700 1 --2- 172.21.15.117:0/367061673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79dc07e700 0x7f79dc07cb60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f79cc0098a0 tx=0x7f79cc009bb0 comp rx=0 tx=0).stop 2024-03-19T03:22:27.639 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.740+0000 7f79e0a00700 1 -- 172.21.15.117:0/367061673 shutdown_connections 2024-03-19T03:22:27.640 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.740+0000 7f79e0a00700 1 --2- 172.21.15.117:0/367061673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79dc07e700 0x7f79dc07cb60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:27.640 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.740+0000 7f79e0a00700 1 -- 172.21.15.117:0/367061673 >> 172.21.15.117:0/367061673 conn(0x7f79dc103300 msgr2=0x7f79dc105720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:27.640 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.740+0000 7f79e0a00700 1 -- 172.21.15.117:0/367061673 shutdown_connections 2024-03-19T03:22:27.640 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.740+0000 7f79e0a00700 1 -- 172.21.15.117:0/367061673 wait complete. 2024-03-19T03:22:27.640 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.741+0000 7f79e0a00700 1 Processor -- start 2024-03-19T03:22:27.640 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.741+0000 7f79e0a00700 1 -- start start 2024-03-19T03:22:27.640 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.742+0000 7f79e0a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79dc07e700 0x7f79dc19ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:27.640 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.742+0000 7f79e0a00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f79dc10e390 con 0x7f79dc07e700 2024-03-19T03:22:27.640 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.742+0000 7f79d9e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79dc07e700 0x7f79dc19ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:27.640 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.742+0000 7f79d9e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79dc07e700 0x7f79dc19ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60846/0 (socket says 172.21.15.117:60846) 2024-03-19T03:22:27.643 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.742+0000 7f79d9e00700 1 -- 172.21.15.117:0/4111542820 learned_addr learned my addr 172.21.15.117:0/4111542820 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:27.643 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.742+0000 7f79d9e00700 1 -- 172.21.15.117:0/4111542820 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79dc19e410 con 0x7f79dc07e700 2024-03-19T03:22:27.643 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.743+0000 7f79d9e00700 1 --2- 172.21.15.117:0/4111542820 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79dc07e700 0x7f79dc19ded0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f79cc000c00 tx=0x7f79cc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:27.643 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.743+0000 7f79d2a00700 1 -- 172.21.15.117:0/4111542820 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79cc003b60 con 0x7f79dc07e700 2024-03-19T03:22:27.644 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.743+0000 7f79d2a00700 1 -- 172.21.15.117:0/4111542820 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79cc003d00 con 0x7f79dc07e700 2024-03-19T03:22:27.644 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.743+0000 7f79d2a00700 1 -- 172.21.15.117:0/4111542820 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79cc0248a0 con 0x7f79dc07e700 2024-03-19T03:22:27.644 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.743+0000 7f79e0a00700 1 -- 172.21.15.117:0/4111542820 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79dc19e6a0 con 0x7f79dc07e700 2024-03-19T03:22:27.644 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.743+0000 7f79e0a00700 1 -- 172.21.15.117:0/4111542820 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79dc19eb00 con 0x7f79dc07e700 2024-03-19T03:22:27.644 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.745+0000 7f79d2a00700 1 -- 172.21.15.117:0/4111542820 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f79cc022020 con 0x7f79dc07e700 2024-03-19T03:22:27.644 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.745+0000 7f79e0a00700 1 -- 172.21.15.117:0/4111542820 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79dc063940 con 0x7f79dc07e700 2024-03-19T03:22:27.644 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.745+0000 7f79d2a00700 1 --2- 172.21.15.117:0/4111542820 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f79c0038f20 0x7f79c003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:27.644 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.745+0000 7f79d2a00700 1 -- 172.21.15.117:0/4111542820 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f79cc04cfa0 con 0x7f79dc07e700 2024-03-19T03:22:27.644 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.745+0000 7f79d9400700 1 --2- 172.21.15.117:0/4111542820 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f79c0038f20 0x7f79c003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:27.644 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.746+0000 7f79d9400700 1 --2- 172.21.15.117:0/4111542820 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f79c0038f20 0x7f79c003b3e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f79c8006fd0 tx=0x7f79c8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:27.644 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.751+0000 7f79d2a00700 1 -- 172.21.15.117:0/4111542820 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f79cc04c930 con 0x7f79dc07e700 2024-03-19T03:22:27.644 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.972+0000 7f79e0a00700 1 -- 172.21.15.117:0/4111542820 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f79dc1a14d0 con 0x7f79dc07e700 2024-03-19T03:22:27.644 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.975+0000 7f79d2a00700 1 -- 172.21.15.117:0/4111542820 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f79cc01acc0 con 0x7f79dc07e700 2024-03-19T03:22:27.644 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.979+0000 7f79e0a00700 1 -- 172.21.15.117:0/4111542820 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f79c0038f20 msgr2=0x7f79c003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:27.644 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.979+0000 7f79e0a00700 1 --2- 172.21.15.117:0/4111542820 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f79c0038f20 0x7f79c003b3e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f79c8006fd0 tx=0x7f79c8006e40 comp rx=0 tx=0).stop 2024-03-19T03:22:27.645 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.979+0000 7f79e0a00700 1 -- 172.21.15.117:0/4111542820 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79dc07e700 msgr2=0x7f79dc19ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:27.645 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.979+0000 7f79e0a00700 1 --2- 172.21.15.117:0/4111542820 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79dc07e700 0x7f79dc19ded0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f79cc000c00 tx=0x7f79cc011c10 comp rx=0 tx=0).stop 2024-03-19T03:22:27.645 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.979+0000 7f79e0a00700 1 -- 172.21.15.117:0/4111542820 shutdown_connections 2024-03-19T03:22:27.645 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.979+0000 7f79e0a00700 1 --2- 172.21.15.117:0/4111542820 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f79c0038f20 0x7f79c003b3e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:27.645 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.979+0000 7f79e0a00700 1 --2- 172.21.15.117:0/4111542820 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79dc07e700 0x7f79dc19ded0 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f79cc000c00 tx=0x7f79cc011c10 comp rx=0 tx=0).stop 2024-03-19T03:22:27.645 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.979+0000 7f79e0a00700 1 -- 172.21.15.117:0/4111542820 >> 172.21.15.117:0/4111542820 conn(0x7f79dc103300 msgr2=0x7f79dc103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:27.645 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.980+0000 7f79e0a00700 1 -- 172.21.15.117:0/4111542820 shutdown_connections 2024-03-19T03:22:27.645 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:26.980+0000 7f79e0a00700 1 -- 172.21.15.117:0/4111542820 wait complete. 2024-03-19T03:22:27.645 INFO:teuthology.orchestra.run.smithi117.stdout:Enabling the dashboard module... 2024-03-19T03:22:29.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:29 smithi117 ceph-mon[31838]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:22:29.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:29 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/741965868' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-19T03:22:29.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:29 smithi117 ceph-mon[31838]: from='mgr.14120 172.21.15.117:0/1301064823' entity='mgr.x' 2024-03-19T03:22:29.542 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:29 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: ignoring --setuser ceph since I am not root 2024-03-19T03:22:29.542 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:29 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: ignoring --setgroup ceph since I am not root 2024-03-19T03:22:29.896 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.233+0000 7f2d40800700 1 Processor -- start 2024-03-19T03:22:29.896 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.233+0000 7f2d40800700 1 -- start start 2024-03-19T03:22:29.896 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.233+0000 7f2d40800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d381082a0 0x7f2d381086c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:29.896 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.233+0000 7f2d40800700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2d38108c90 con 0x7f2d381082a0 2024-03-19T03:22:29.896 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.234+0000 7f2d3de00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d381082a0 0x7f2d381086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:29.896 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.234+0000 7f2d3de00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d381082a0 0x7f2d381086c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60858/0 (socket says 172.21.15.117:60858) 2024-03-19T03:22:29.896 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.234+0000 7f2d3de00700 1 -- 172.21.15.117:0/2363365240 learned_addr learned my addr 172.21.15.117:0/2363365240 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:29.896 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.234+0000 7f2d3de00700 1 -- 172.21.15.117:0/2363365240 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d38105da0 con 0x7f2d381082a0 2024-03-19T03:22:29.896 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.235+0000 7f2d3de00700 1 --2- 172.21.15.117:0/2363365240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d381082a0 0x7f2d381086c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c0098a0 tx=0x7f2d2c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44313752e58a81e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:29.897 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.235+0000 7f2d3ca00700 1 -- 172.21.15.117:0/2363365240 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d2c004160 con 0x7f2d381082a0 2024-03-19T03:22:29.897 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.235+0000 7f2d3ca00700 1 -- 172.21.15.117:0/2363365240 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d2c004300 con 0x7f2d381082a0 2024-03-19T03:22:29.897 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.235+0000 7f2d3ca00700 1 -- 172.21.15.117:0/2363365240 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d2c011b20 con 0x7f2d381082a0 2024-03-19T03:22:29.897 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.236+0000 7f2d40800700 1 -- 172.21.15.117:0/2363365240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d381082a0 msgr2=0x7f2d381086c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:29.897 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.236+0000 7f2d40800700 1 --2- 172.21.15.117:0/2363365240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d381082a0 0x7f2d381086c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c0098a0 tx=0x7f2d2c009bb0 comp rx=0 tx=0).stop 2024-03-19T03:22:29.897 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.236+0000 7f2d40800700 1 -- 172.21.15.117:0/2363365240 shutdown_connections 2024-03-19T03:22:29.897 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.236+0000 7f2d40800700 1 --2- 172.21.15.117:0/2363365240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d381082a0 0x7f2d381086c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:29.897 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.236+0000 7f2d40800700 1 -- 172.21.15.117:0/2363365240 >> 172.21.15.117:0/2363365240 conn(0x7f2d38101940 msgr2=0x7f2d38103d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:29.897 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.236+0000 7f2d40800700 1 -- 172.21.15.117:0/2363365240 shutdown_connections 2024-03-19T03:22:29.897 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.236+0000 7f2d40800700 1 -- 172.21.15.117:0/2363365240 wait complete. 2024-03-19T03:22:29.897 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.237+0000 7f2d40800700 1 Processor -- start 2024-03-19T03:22:29.897 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.237+0000 7f2d40800700 1 -- start start 2024-03-19T03:22:29.897 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.238+0000 7f2d40800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d381082a0 0x7f2d3819de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:29.897 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.238+0000 7f2d40800700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2d38106e60 con 0x7f2d381082a0 2024-03-19T03:22:29.898 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.238+0000 7f2d3de00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d381082a0 0x7f2d3819de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:29.898 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.238+0000 7f2d3de00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d381082a0 0x7f2d3819de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60874/0 (socket says 172.21.15.117:60874) 2024-03-19T03:22:29.898 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.238+0000 7f2d3de00700 1 -- 172.21.15.117:0/741965868 learned_addr learned my addr 172.21.15.117:0/741965868 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:29.898 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.239+0000 7f2d3de00700 1 -- 172.21.15.117:0/741965868 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d3819e370 con 0x7f2d381082a0 2024-03-19T03:22:29.898 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.239+0000 7f2d3de00700 1 --2- 172.21.15.117:0/741965868 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d381082a0 0x7f2d3819de30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c0117e0 tx=0x7f2d2c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:29.898 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.239+0000 7f2d32a00700 1 -- 172.21.15.117:0/741965868 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d2c003ba0 con 0x7f2d381082a0 2024-03-19T03:22:29.898 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.239+0000 7f2d32a00700 1 -- 172.21.15.117:0/741965868 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d2c01a910 con 0x7f2d381082a0 2024-03-19T03:22:29.898 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.239+0000 7f2d32a00700 1 -- 172.21.15.117:0/741965868 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d2c0248f0 con 0x7f2d381082a0 2024-03-19T03:22:29.898 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.239+0000 7f2d40800700 1 -- 172.21.15.117:0/741965868 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d3819e600 con 0x7f2d381082a0 2024-03-19T03:22:29.898 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.239+0000 7f2d40800700 1 -- 172.21.15.117:0/741965868 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d3819ea60 con 0x7f2d381082a0 2024-03-19T03:22:29.898 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.241+0000 7f2d40800700 1 -- 172.21.15.117:0/741965868 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d38063940 con 0x7f2d381082a0 2024-03-19T03:22:29.898 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.241+0000 7f2d32a00700 1 -- 172.21.15.117:0/741965868 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f2d2c022020 con 0x7f2d381082a0 2024-03-19T03:22:29.898 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.241+0000 7f2d32a00700 1 --2- 172.21.15.117:0/741965868 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f2d20038f70 0x7f2d2003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:29.898 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.241+0000 7f2d32a00700 1 -- 172.21.15.117:0/741965868 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2d2c04d320 con 0x7f2d381082a0 2024-03-19T03:22:29.899 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.242+0000 7f2d3d400700 1 --2- 172.21.15.117:0/741965868 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f2d20038f70 0x7f2d2003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:29.899 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.242+0000 7f2d3d400700 1 --2- 172.21.15.117:0/741965868 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f2d20038f70 0x7f2d2003b430 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2d28006fd0 tx=0x7f2d28006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:29.899 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.247+0000 7f2d32a00700 1 -- 172.21.15.117:0/741965868 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2d2c01ab70 con 0x7f2d381082a0 2024-03-19T03:22:29.899 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:28.526+0000 7f2d40800700 1 -- 172.21.15.117:0/741965868 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f2d38197430 con 0x7f2d381082a0 2024-03-19T03:22:29.899 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:29.211+0000 7f2d32a00700 1 -- 172.21.15.117:0/741965868 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f2d38197430 con 0x7f2d381082a0 2024-03-19T03:22:29.899 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:29.214+0000 7f2d32a00700 1 -- 172.21.15.117:0/741965868 <== mon.0 v2:172.21.15.117:3300/0 8 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f2d2c02d3f0 con 0x7f2d381082a0 2024-03-19T03:22:29.899 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:29.216+0000 7f2d40800700 1 -- 172.21.15.117:0/741965868 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f2d20038f70 msgr2=0x7f2d2003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:29.899 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:29.216+0000 7f2d40800700 1 --2- 172.21.15.117:0/741965868 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f2d20038f70 0x7f2d2003b430 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2d28006fd0 tx=0x7f2d28006e40 comp rx=0 tx=0).stop 2024-03-19T03:22:29.899 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:29.216+0000 7f2d40800700 1 -- 172.21.15.117:0/741965868 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d381082a0 msgr2=0x7f2d3819de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:29.899 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:29.216+0000 7f2d40800700 1 --2- 172.21.15.117:0/741965868 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d381082a0 0x7f2d3819de30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c0117e0 tx=0x7f2d2c011ba0 comp rx=0 tx=0).stop 2024-03-19T03:22:29.899 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:29.217+0000 7f2d40800700 1 -- 172.21.15.117:0/741965868 shutdown_connections 2024-03-19T03:22:29.899 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:29.217+0000 7f2d40800700 1 --2- 172.21.15.117:0/741965868 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f2d20038f70 0x7f2d2003b430 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:29.899 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:29.217+0000 7f2d40800700 1 --2- 172.21.15.117:0/741965868 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d381082a0 0x7f2d3819de30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:29.900 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:29.217+0000 7f2d40800700 1 -- 172.21.15.117:0/741965868 >> 172.21.15.117:0/741965868 conn(0x7f2d38101940 msgr2=0x7f2d381944f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:29.900 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:29.217+0000 7f2d40800700 1 -- 172.21.15.117:0/741965868 shutdown_connections 2024-03-19T03:22:29.900 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:29.217+0000 7f2d40800700 1 -- 172.21.15.117:0/741965868 wait complete. 2024-03-19T03:22:30.156 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:30 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:30.035+0000 7f25ba9fa200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T03:22:30.156 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:30 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:30.155+0000 7f25ba9fa200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T03:22:30.430 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:30 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/741965868' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-19T03:22:30.430 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:30 smithi117 ceph-mon[31838]: mgrmap e9: x(active, since 25s) 2024-03-19T03:22:31.041 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:31 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:31.000+0000 7f25ba9fa200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T03:22:31.432 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:31 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/4183803382' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-19T03:22:31.432 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:31 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:31.125+0000 7f25ba9fa200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T03:22:31.432 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:31 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:31.303+0000 7f25ba9fa200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T03:22:31.432 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:31 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:31.431+0000 7f25ba9fa200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T03:22:31.791 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:31 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:31.542+0000 7f25ba9fa200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T03:22:31.850 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout { 2024-03-19T03:22:31.851 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-19T03:22:31.851 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-19T03:22:31.851 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-19T03:22:31.852 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-19T03:22:31.852 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout } 2024-03-19T03:22:31.852 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.660+0000 7f28ece00700 1 Processor -- start 2024-03-19T03:22:31.852 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.660+0000 7f28ece00700 1 -- start start 2024-03-19T03:22:31.852 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.661+0000 7f28ece00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28e8106580 0x7f28e81089b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:31.852 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.661+0000 7f28ece00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f28e807be40 con 0x7f28e8106580 2024-03-19T03:22:31.858 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.661+0000 7f28e5e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28e8106580 0x7f28e81089b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:31.858 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.661+0000 7f28e5e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28e8106580 0x7f28e81089b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60884/0 (socket says 172.21.15.117:60884) 2024-03-19T03:22:31.858 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.661+0000 7f28e5e00700 1 -- 172.21.15.117:0/3886089053 learned_addr learned my addr 172.21.15.117:0/3886089053 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:31.858 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.662+0000 7f28e5e00700 1 -- 172.21.15.117:0/3886089053 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28e8108ef0 con 0x7f28e8106580 2024-03-19T03:22:31.858 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.662+0000 7f28e5e00700 1 --2- 172.21.15.117:0/3886089053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28e8106580 0x7f28e81089b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f28d80098a0 tx=0x7f28d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6bbf60e61410243 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:31.859 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.662+0000 7f28e4a00700 1 -- 172.21.15.117:0/3886089053 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28d8004160 con 0x7f28e8106580 2024-03-19T03:22:31.859 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.662+0000 7f28e4a00700 1 -- 172.21.15.117:0/3886089053 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28d8004300 con 0x7f28e8106580 2024-03-19T03:22:31.859 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.662+0000 7f28e4a00700 1 -- 172.21.15.117:0/3886089053 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28d8011b20 con 0x7f28e8106580 2024-03-19T03:22:31.859 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.663+0000 7f28ece00700 1 -- 172.21.15.117:0/3886089053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28e8106580 msgr2=0x7f28e81089b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:31.859 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.663+0000 7f28ece00700 1 --2- 172.21.15.117:0/3886089053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28e8106580 0x7f28e81089b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f28d80098a0 tx=0x7f28d8009bb0 comp rx=0 tx=0).stop 2024-03-19T03:22:31.859 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.663+0000 7f28ece00700 1 -- 172.21.15.117:0/3886089053 shutdown_connections 2024-03-19T03:22:31.859 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.663+0000 7f28ece00700 1 --2- 172.21.15.117:0/3886089053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28e8106580 0x7f28e81089b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:31.859 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.663+0000 7f28ece00700 1 -- 172.21.15.117:0/3886089053 >> 172.21.15.117:0/3886089053 conn(0x7f28e81021c0 msgr2=0x7f28e8104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:31.859 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.663+0000 7f28ece00700 1 -- 172.21.15.117:0/3886089053 shutdown_connections 2024-03-19T03:22:31.859 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.663+0000 7f28ece00700 1 -- 172.21.15.117:0/3886089053 wait complete. 2024-03-19T03:22:31.859 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.664+0000 7f28ece00700 1 Processor -- start 2024-03-19T03:22:31.859 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.664+0000 7f28ece00700 1 -- start start 2024-03-19T03:22:31.859 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.665+0000 7f28ece00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28e8106580 0x7f28e81a21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:31.860 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.665+0000 7f28ece00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f28e8109fc0 con 0x7f28e8106580 2024-03-19T03:22:31.860 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.665+0000 7f28e5e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28e8106580 0x7f28e81a21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:31.860 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.665+0000 7f28e5e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28e8106580 0x7f28e81a21f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60894/0 (socket says 172.21.15.117:60894) 2024-03-19T03:22:31.860 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.665+0000 7f28e5e00700 1 -- 172.21.15.117:0/4183803382 learned_addr learned my addr 172.21.15.117:0/4183803382 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:31.860 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.665+0000 7f28e5e00700 1 -- 172.21.15.117:0/4183803382 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28e81a2730 con 0x7f28e8106580 2024-03-19T03:22:31.860 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.666+0000 7f28e5e00700 1 --2- 172.21.15.117:0/4183803382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28e8106580 0x7f28e81a21f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f28d80117e0 tx=0x7f28d8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:31.860 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.666+0000 7f28dea00700 1 -- 172.21.15.117:0/4183803382 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28d8003ba0 con 0x7f28e8106580 2024-03-19T03:22:31.860 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.666+0000 7f28dea00700 1 -- 172.21.15.117:0/4183803382 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28d801a910 con 0x7f28e8106580 2024-03-19T03:22:31.860 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.666+0000 7f28dea00700 1 -- 172.21.15.117:0/4183803382 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28d80248f0 con 0x7f28e8106580 2024-03-19T03:22:31.860 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.666+0000 7f28ece00700 1 -- 172.21.15.117:0/4183803382 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28e81a29c0 con 0x7f28e8106580 2024-03-19T03:22:31.860 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.666+0000 7f28ece00700 1 -- 172.21.15.117:0/4183803382 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28e81a2e20 con 0x7f28e8106580 2024-03-19T03:22:31.860 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.667+0000 7f28ece00700 1 -- 172.21.15.117:0/4183803382 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28e8063940 con 0x7f28e8106580 2024-03-19T03:22:31.860 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.667+0000 7f28dea00700 1 -- 172.21.15.117:0/4183803382 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f28d8022020 con 0x7f28e8106580 2024-03-19T03:22:31.860 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.668+0000 7f28dea00700 1 --2- 172.21.15.117:0/4183803382 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f28cc038fd0 0x7f28cc03b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:31.861 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.668+0000 7f28dea00700 1 -- 172.21.15.117:0/4183803382 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f28d804d3d0 con 0x7f28e8106580 2024-03-19T03:22:31.861 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.668+0000 7f28e5400700 1 -- 172.21.15.117:0/4183803382 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f28cc038fd0 msgr2=0x7f28cc03b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:6800/2788067603 2024-03-19T03:22:31.861 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.668+0000 7f28e5400700 1 --2- 172.21.15.117:0/4183803382 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f28cc038fd0 0x7f28cc03b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T03:22:31.861 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.673+0000 7f28dea00700 1 -- 172.21.15.117:0/4183803382 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f28d800b3e0 con 0x7f28e8106580 2024-03-19T03:22:31.861 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.869+0000 7f28e5400700 1 -- 172.21.15.117:0/4183803382 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f28cc038fd0 msgr2=0x7f28cc03b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:6800/2788067603 2024-03-19T03:22:31.861 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.869+0000 7f28e5400700 1 --2- 172.21.15.117:0/4183803382 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f28cc038fd0 0x7f28cc03b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T03:22:31.861 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.951+0000 7f28ece00700 1 -- 172.21.15.117:0/4183803382 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f28e8062f20 con 0x7f28e8106580 2024-03-19T03:22:31.861 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.952+0000 7f28dea00700 1 -- 172.21.15.117:0/4183803382 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f28d801ab70 con 0x7f28e8106580 2024-03-19T03:22:31.861 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.955+0000 7f28ece00700 1 -- 172.21.15.117:0/4183803382 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f28cc038fd0 msgr2=0x7f28cc03b490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T03:22:31.861 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.955+0000 7f28ece00700 1 --2- 172.21.15.117:0/4183803382 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f28cc038fd0 0x7f28cc03b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:31.861 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.955+0000 7f28ece00700 1 -- 172.21.15.117:0/4183803382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28e8106580 msgr2=0x7f28e81a21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:31.861 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.955+0000 7f28ece00700 1 --2- 172.21.15.117:0/4183803382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28e8106580 0x7f28e81a21f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f28d80117e0 tx=0x7f28d8011ba0 comp rx=0 tx=0).stop 2024-03-19T03:22:31.861 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.955+0000 7f28ece00700 1 -- 172.21.15.117:0/4183803382 shutdown_connections 2024-03-19T03:22:31.862 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.955+0000 7f28ece00700 1 --2- 172.21.15.117:0/4183803382 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f28cc038fd0 0x7f28cc03b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:31.862 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.955+0000 7f28ece00700 1 --2- 172.21.15.117:0/4183803382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28e8106580 0x7f28e81a21f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:31.862 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.955+0000 7f28ece00700 1 -- 172.21.15.117:0/4183803382 >> 172.21.15.117:0/4183803382 conn(0x7f28e81021c0 msgr2=0x7f28e8104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:31.862 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.955+0000 7f28ece00700 1 -- 172.21.15.117:0/4183803382 shutdown_connections 2024-03-19T03:22:31.862 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:30.955+0000 7f28ece00700 1 -- 172.21.15.117:0/4183803382 wait complete. 2024-03-19T03:22:31.862 INFO:teuthology.orchestra.run.smithi117.stdout:Waiting for the mgr to restart... 2024-03-19T03:22:31.862 INFO:teuthology.orchestra.run.smithi117.stdout:Waiting for mgr epoch 9... 2024-03-19T03:22:32.143 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:31 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:31.881+0000 7f25ba9fa200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T03:22:32.143 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:32 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:32.029+0000 7f25ba9fa200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T03:22:32.143 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:32 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:32.142+0000 7f25ba9fa200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T03:22:32.481 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:32 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:32.269+0000 7f25ba9fa200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T03:22:32.770 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:32 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:32.496+0000 7f25ba9fa200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T03:22:34.041 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:33 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:33.762+0000 7f25ba9fa200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T03:22:34.042 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:33 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:33.931+0000 7f25ba9fa200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T03:22:34.462 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:34 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:34.345+0000 7f25ba9fa200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T03:22:34.717 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:34 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:34.462+0000 7f25ba9fa200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T03:22:34.717 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:34 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:34.717+0000 7f25ba9fa200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T03:22:35.041 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:34 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:34.958+0000 7f25ba9fa200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T03:22:35.644 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:35 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:35.329+0000 7f25ba9fa200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T03:22:35.947 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:35 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:35.643+0000 7f25ba9fa200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T03:22:36.291 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:35 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:35.946+0000 7f25ba9fa200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T03:22:37.291 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:37 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:37.011+0000 7f25ba9fa200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T03:22:38.041 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:37 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:37.722+0000 7f25ba9fa200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T03:22:38.042 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:37 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:37.850+0000 7f25ba9fa200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T03:22:38.541 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:22:38 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:22:38.131+0000 7f25ba9fa200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T03:22:38.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:38 smithi117 ceph-mon[31838]: Active manager daemon x restarted 2024-03-19T03:22:38.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:38 smithi117 ceph-mon[31838]: Activating manager daemon x 2024-03-19T03:22:38.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:38 smithi117 ceph-mon[31838]: osdmap e3: 0 total, 0 up, 0 in 2024-03-19T03:22:38.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:38 smithi117 ceph-mon[31838]: mgrmap e10: x(active, starting, since 0.01093s) 2024-03-19T03:22:38.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:38 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T03:22:38.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:38 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-19T03:22:38.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:38 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-19T03:22:38.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:38 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-19T03:22:38.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:38 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-19T03:22:38.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:38 smithi117 ceph-mon[31838]: Manager daemon x is now available 2024-03-19T03:22:39.489 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:22:39.490 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:22:39.490 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-19T03:22:39.490 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-19T03:22:39.490 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:39 smithi117 ceph-mon[31838]: mgrmap e11: x(active, since 1.01149s) 2024-03-19T03:22:39.787 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout { 2024-03-19T03:22:39.788 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-19T03:22:39.788 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-19T03:22:39.788 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout } 2024-03-19T03:22:39.788 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.769+0000 7f42de800700 1 Processor -- start 2024-03-19T03:22:39.789 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.770+0000 7f42de800700 1 -- start start 2024-03-19T03:22:39.790 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.770+0000 7f42de800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42d8107d70 0x7f42d8108190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:39.790 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.770+0000 7f42de800700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f42d8108760 con 0x7f42d8107d70 2024-03-19T03:22:39.790 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.770+0000 7f42d7400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42d8107d70 0x7f42d8108190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:39.790 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.770+0000 7f42d7400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42d8107d70 0x7f42d8108190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60908/0 (socket says 172.21.15.117:60908) 2024-03-19T03:22:39.790 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.770+0000 7f42d7400700 1 -- 172.21.15.117:0/1997009427 learned_addr learned my addr 172.21.15.117:0/1997009427 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:39.790 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.771+0000 7f42d7400700 1 -- 172.21.15.117:0/1997009427 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42d8108ed0 con 0x7f42d8107d70 2024-03-19T03:22:39.792 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.771+0000 7f42d7400700 1 --2- 172.21.15.117:0/1997009427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42d8107d70 0x7f42d8108190 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f42c80098a0 tx=0x7f42c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab8d6264f03f7b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:39.792 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.771+0000 7f42d6000700 1 -- 172.21.15.117:0/1997009427 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f42c8004160 con 0x7f42d8107d70 2024-03-19T03:22:39.792 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.771+0000 7f42d6000700 1 -- 172.21.15.117:0/1997009427 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42c8004300 con 0x7f42d8107d70 2024-03-19T03:22:39.792 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.771+0000 7f42d6000700 1 -- 172.21.15.117:0/1997009427 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f42c8011b20 con 0x7f42d8107d70 2024-03-19T03:22:39.792 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.772+0000 7f42de800700 1 -- 172.21.15.117:0/1997009427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42d8107d70 msgr2=0x7f42d8108190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:39.792 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.772+0000 7f42de800700 1 --2- 172.21.15.117:0/1997009427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42d8107d70 0x7f42d8108190 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f42c80098a0 tx=0x7f42c8009bb0 comp rx=0 tx=0).stop 2024-03-19T03:22:39.792 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.772+0000 7f42de800700 1 -- 172.21.15.117:0/1997009427 shutdown_connections 2024-03-19T03:22:39.792 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.772+0000 7f42de800700 1 --2- 172.21.15.117:0/1997009427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42d8107d70 0x7f42d8108190 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:39.793 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.772+0000 7f42de800700 1 -- 172.21.15.117:0/1997009427 >> 172.21.15.117:0/1997009427 conn(0x7f42d8103310 msgr2=0x7f42d8105750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:39.793 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.772+0000 7f42de800700 1 -- 172.21.15.117:0/1997009427 shutdown_connections 2024-03-19T03:22:39.793 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.772+0000 7f42de800700 1 -- 172.21.15.117:0/1997009427 wait complete. 2024-03-19T03:22:39.793 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.773+0000 7f42de800700 1 Processor -- start 2024-03-19T03:22:39.793 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.773+0000 7f42de800700 1 -- start start 2024-03-19T03:22:39.793 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.774+0000 7f42de800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42d8107d70 0x7f42d807db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:39.793 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.774+0000 7f42de800700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f42d810c280 con 0x7f42d8107d70 2024-03-19T03:22:39.793 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.774+0000 7f42d7400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42d8107d70 0x7f42d807db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:39.793 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.774+0000 7f42d7400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42d8107d70 0x7f42d807db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60912/0 (socket says 172.21.15.117:60912) 2024-03-19T03:22:39.796 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.774+0000 7f42d7400700 1 -- 172.21.15.117:0/4206491971 learned_addr learned my addr 172.21.15.117:0/4206491971 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:39.796 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.774+0000 7f42d7400700 1 -- 172.21.15.117:0/4206491971 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42d807e040 con 0x7f42d8107d70 2024-03-19T03:22:39.796 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.775+0000 7f42d7400700 1 --2- 172.21.15.117:0/4206491971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42d8107d70 0x7f42d807db00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f42c80117e0 tx=0x7f42c8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:39.796 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.775+0000 7f42cfe00700 1 -- 172.21.15.117:0/4206491971 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f42c8003bf0 con 0x7f42d8107d70 2024-03-19T03:22:39.796 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.775+0000 7f42cfe00700 1 -- 172.21.15.117:0/4206491971 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42c801a910 con 0x7f42d8107d70 2024-03-19T03:22:39.797 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.775+0000 7f42cfe00700 1 -- 172.21.15.117:0/4206491971 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f42c8024970 con 0x7f42d8107d70 2024-03-19T03:22:39.797 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.775+0000 7f42de800700 1 -- 172.21.15.117:0/4206491971 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42d807c230 con 0x7f42d8107d70 2024-03-19T03:22:39.797 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.775+0000 7f42de800700 1 -- 172.21.15.117:0/4206491971 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42d807c6f0 con 0x7f42d8107d70 2024-03-19T03:22:39.797 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.776+0000 7f42cfe00700 1 -- 172.21.15.117:0/4206491971 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f42c8022020 con 0x7f42d8107d70 2024-03-19T03:22:39.797 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.777+0000 7f42cfe00700 1 --2- 172.21.15.117:0/4206491971 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f42bc038fd0 0x7f42bc03b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:39.797 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.777+0000 7f42cfe00700 1 -- 172.21.15.117:0/4206491971 --> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f42bc03bba0 con 0x7f42bc038fd0 2024-03-19T03:22:39.797 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.777+0000 7f42cfe00700 1 -- 172.21.15.117:0/4206491971 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f42c804d230 con 0x7f42d8107d70 2024-03-19T03:22:39.797 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.777+0000 7f42d6a00700 1 -- 172.21.15.117:0/4206491971 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f42bc038fd0 msgr2=0x7f42bc03b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:6800/2788067603 2024-03-19T03:22:39.797 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.777+0000 7f42d6a00700 1 --2- 172.21.15.117:0/4206491971 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f42bc038fd0 0x7f42bc03b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T03:22:39.797 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.978+0000 7f42d6a00700 1 -- 172.21.15.117:0/4206491971 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f42bc038fd0 msgr2=0x7f42bc03b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:6800/2788067603 2024-03-19T03:22:39.797 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:32.978+0000 7f42d6a00700 1 --2- 172.21.15.117:0/4206491971 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f42bc038fd0 0x7f42bc03b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T03:22:39.797 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:33.379+0000 7f42d6a00700 1 -- 172.21.15.117:0/4206491971 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f42bc038fd0 msgr2=0x7f42bc03b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:6800/2788067603 2024-03-19T03:22:39.797 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:33.379+0000 7f42d6a00700 1 --2- 172.21.15.117:0/4206491971 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f42bc038fd0 0x7f42bc03b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-19T03:22:39.797 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:34.180+0000 7f42d6a00700 1 -- 172.21.15.117:0/4206491971 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f42bc038fd0 msgr2=0x7f42bc03b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:6800/2788067603 2024-03-19T03:22:39.797 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:34.180+0000 7f42d6a00700 1 --2- 172.21.15.117:0/4206491971 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f42bc038fd0 0x7f42bc03b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-19T03:22:39.797 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:35.782+0000 7f42d6a00700 1 -- 172.21.15.117:0/4206491971 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f42bc038fd0 msgr2=0x7f42bc03b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.117:6800/2788067603 2024-03-19T03:22:39.798 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:35.782+0000 7f42d6a00700 1 --2- 172.21.15.117:0/4206491971 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f42bc038fd0 0x7f42bc03b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-19T03:22:39.798 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:38.141+0000 7f42cfe00700 1 -- 172.21.15.117:0/4206491971 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mgrmap(e 10) ==== 45886+0+0 (secure 0 0 0) 0x7f42c802d3f0 con 0x7f42d8107d70 2024-03-19T03:22:39.798 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:38.141+0000 7f42cfe00700 1 -- 172.21.15.117:0/4206491971 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f42bc038fd0 msgr2=0x7f42bc03b490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T03:22:39.798 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:38.141+0000 7f42cfe00700 1 --2- 172.21.15.117:0/4206491971 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f42bc038fd0 0x7f42bc03b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:39.798 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:39.143+0000 7f42cfe00700 1 -- 172.21.15.117:0/4206491971 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7f42c8019d40 con 0x7f42d8107d70 2024-03-19T03:22:39.798 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:39.143+0000 7f42cfe00700 1 --2- 172.21.15.117:0/4206491971 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f42bc03ca80 0x7f42bc03ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:39.798 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:39.144+0000 7f42cfe00700 1 -- 172.21.15.117:0/4206491971 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f42bc03bba0 con 0x7f42bc03ca80 2024-03-19T03:22:39.798 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:39.144+0000 7f42d6a00700 1 --2- 172.21.15.117:0/4206491971 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f42bc03ca80 0x7f42bc03ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:39.800 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:39.146+0000 7f42d6a00700 1 --2- 172.21.15.117:0/4206491971 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f42bc03ca80 0x7f42bc03ee70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f42c4003a10 tx=0x7f42c4009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:39.800 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:39.147+0000 7f42cfe00700 1 -- 172.21.15.117:0/4206491971 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f42bc03bba0 con 0x7f42bc03ca80 2024-03-19T03:22:39.800 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:39.155+0000 7f42de800700 1 -- 172.21.15.117:0/4206491971 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f42d8063940 con 0x7f42bc03ca80 2024-03-19T03:22:39.800 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:39.156+0000 7f42cfe00700 1 -- 172.21.15.117:0/4206491971 <== mgr.14152 v2:172.21.15.117:6800/3144614841 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f42d8063940 con 0x7f42bc03ca80 2024-03-19T03:22:39.800 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:39.156+0000 7f42de800700 1 -- 172.21.15.117:0/4206491971 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f42bc03ca80 msgr2=0x7f42bc03ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:39.800 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:39.156+0000 7f42de800700 1 --2- 172.21.15.117:0/4206491971 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f42bc03ca80 0x7f42bc03ee70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f42c4003a10 tx=0x7f42c4009270 comp rx=0 tx=0).stop 2024-03-19T03:22:39.800 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:39.156+0000 7f42de800700 1 -- 172.21.15.117:0/4206491971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42d8107d70 msgr2=0x7f42d807db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:39.800 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:39.156+0000 7f42de800700 1 --2- 172.21.15.117:0/4206491971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42d8107d70 0x7f42d807db00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f42c80117e0 tx=0x7f42c8011ba0 comp rx=0 tx=0).stop 2024-03-19T03:22:39.801 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:39.156+0000 7f42de800700 1 -- 172.21.15.117:0/4206491971 shutdown_connections 2024-03-19T03:22:39.801 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:39.156+0000 7f42de800700 1 --2- 172.21.15.117:0/4206491971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42d8107d70 0x7f42d807db00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:39.801 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:39.156+0000 7f42de800700 1 --2- 172.21.15.117:0/4206491971 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f42bc03ca80 0x7f42bc03ee70 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:39.801 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:39.156+0000 7f42de800700 1 --2- 172.21.15.117:0/4206491971 >> [v2:172.21.15.117:6800/2788067603,v1:172.21.15.117:6801/2788067603] conn(0x7f42bc038fd0 0x7f42bc03b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:39.801 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:39.156+0000 7f42de800700 1 -- 172.21.15.117:0/4206491971 >> 172.21.15.117:0/4206491971 conn(0x7f42d8103310 msgr2=0x7f42d8103dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:39.801 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:39.157+0000 7f42de800700 1 -- 172.21.15.117:0/4206491971 shutdown_connections 2024-03-19T03:22:39.801 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:39.157+0000 7f42de800700 1 -- 172.21.15.117:0/4206491971 wait complete. 2024-03-19T03:22:39.801 INFO:teuthology.orchestra.run.smithi117.stdout:mgr epoch 9 is available 2024-03-19T03:22:39.801 INFO:teuthology.orchestra.run.smithi117.stdout:Generating a dashboard self-signed certificate... 2024-03-19T03:22:40.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:40 smithi117 ceph-mon[31838]: [19/Mar/2024:03:22:38] ENGINE Bus STARTING 2024-03-19T03:22:40.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:40 smithi117 ceph-mon[31838]: [19/Mar/2024:03:22:38] ENGINE Serving on https://172.21.15.117:7150 2024-03-19T03:22:40.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:40 smithi117 ceph-mon[31838]: [19/Mar/2024:03:22:38] ENGINE Serving on http://172.21.15.117:8765 2024-03-19T03:22:40.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:40 smithi117 ceph-mon[31838]: [19/Mar/2024:03:22:38] ENGINE Bus STARTED 2024-03-19T03:22:40.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:40 smithi117 ceph-mon[31838]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-19T03:22:40.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:40 smithi117 ceph-mon[31838]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-19T03:22:41.461 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:41 smithi117 ceph-mon[31838]: mgrmap e12: x(active, since 2s) 2024-03-19T03:22:41.461 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:22:42.156 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-19T03:22:42.156 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.983+0000 7fd7ee600700 1 Processor -- start 2024-03-19T03:22:42.156 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.983+0000 7fd7ee600700 1 -- start start 2024-03-19T03:22:42.156 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.983+0000 7fd7ee600700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7e8106c20 0x7fd7e8107040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:42.156 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.983+0000 7fd7ee600700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e8107610 con 0x7fd7e8106c20 2024-03-19T03:22:42.156 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.983+0000 7fd7e7400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7e8106c20 0x7fd7e8107040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:42.156 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.984+0000 7fd7e7400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7e8106c20 0x7fd7e8107040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:52992/0 (socket says 172.21.15.117:52992) 2024-03-19T03:22:42.157 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.984+0000 7fd7e7400700 1 -- 172.21.15.117:0/3468247045 learned_addr learned my addr 172.21.15.117:0/3468247045 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:42.157 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.984+0000 7fd7e7400700 1 -- 172.21.15.117:0/3468247045 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e8107e20 con 0x7fd7e8106c20 2024-03-19T03:22:42.157 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.984+0000 7fd7e7400700 1 --2- 172.21.15.117:0/3468247045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7e8106c20 0x7fd7e8107040 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd7d80098a0 tx=0x7fd7d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1b670883a562b08 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:42.157 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.984+0000 7fd7e6a00700 1 -- 172.21.15.117:0/3468247045 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7d8004160 con 0x7fd7e8106c20 2024-03-19T03:22:42.157 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.985+0000 7fd7e6a00700 1 -- 172.21.15.117:0/3468247045 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7d8004300 con 0x7fd7e8106c20 2024-03-19T03:22:42.157 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.985+0000 7fd7e6a00700 1 -- 172.21.15.117:0/3468247045 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7d8011b20 con 0x7fd7e8106c20 2024-03-19T03:22:42.157 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.985+0000 7fd7ee600700 1 -- 172.21.15.117:0/3468247045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7e8106c20 msgr2=0x7fd7e8107040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:42.157 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.985+0000 7fd7ee600700 1 --2- 172.21.15.117:0/3468247045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7e8106c20 0x7fd7e8107040 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd7d80098a0 tx=0x7fd7d8009bb0 comp rx=0 tx=0).stop 2024-03-19T03:22:42.157 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.985+0000 7fd7ee600700 1 -- 172.21.15.117:0/3468247045 shutdown_connections 2024-03-19T03:22:42.157 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.985+0000 7fd7ee600700 1 --2- 172.21.15.117:0/3468247045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7e8106c20 0x7fd7e8107040 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:42.157 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.985+0000 7fd7ee600700 1 -- 172.21.15.117:0/3468247045 >> 172.21.15.117:0/3468247045 conn(0x7fd7e81021c0 msgr2=0x7fd7e8104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:42.157 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.985+0000 7fd7ee600700 1 -- 172.21.15.117:0/3468247045 shutdown_connections 2024-03-19T03:22:42.157 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.986+0000 7fd7ee600700 1 -- 172.21.15.117:0/3468247045 wait complete. 2024-03-19T03:22:42.158 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.987+0000 7fd7ee600700 1 Processor -- start 2024-03-19T03:22:42.158 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.987+0000 7fd7ee600700 1 -- start start 2024-03-19T03:22:42.158 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.987+0000 7fd7ee600700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7e8106c20 0x7fd7e819de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:42.158 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.987+0000 7fd7ee600700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e807af60 con 0x7fd7e8106c20 2024-03-19T03:22:42.158 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.987+0000 7fd7e7400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7e8106c20 0x7fd7e819de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:42.158 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.987+0000 7fd7e7400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7e8106c20 0x7fd7e819de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:52998/0 (socket says 172.21.15.117:52998) 2024-03-19T03:22:42.158 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.987+0000 7fd7e7400700 1 -- 172.21.15.117:0/1288403540 learned_addr learned my addr 172.21.15.117:0/1288403540 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:42.158 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.988+0000 7fd7e7400700 1 -- 172.21.15.117:0/1288403540 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e819e3d0 con 0x7fd7e8106c20 2024-03-19T03:22:42.158 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.988+0000 7fd7e7400700 1 --2- 172.21.15.117:0/1288403540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7e8106c20 0x7fd7e819de90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd7d80117e0 tx=0x7fd7d8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:42.158 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.988+0000 7fd7e4c00700 1 -- 172.21.15.117:0/1288403540 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7d8003bf0 con 0x7fd7e8106c20 2024-03-19T03:22:42.158 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.988+0000 7fd7e4c00700 1 -- 172.21.15.117:0/1288403540 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7d801a910 con 0x7fd7e8106c20 2024-03-19T03:22:42.158 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.988+0000 7fd7e4c00700 1 -- 172.21.15.117:0/1288403540 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7d8024970 con 0x7fd7e8106c20 2024-03-19T03:22:42.158 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.988+0000 7fd7ee600700 1 -- 172.21.15.117:0/1288403540 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7e819e660 con 0x7fd7e8106c20 2024-03-19T03:22:42.158 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.988+0000 7fd7ee600700 1 -- 172.21.15.117:0/1288403540 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7e819eac0 con 0x7fd7e8106c20 2024-03-19T03:22:42.161 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.989+0000 7fd7e4c00700 1 -- 172.21.15.117:0/1288403540 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7fd7d8022020 con 0x7fd7e8106c20 2024-03-19T03:22:42.161 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.990+0000 7fd7e4c00700 1 --2- 172.21.15.117:0/1288403540 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd7cc038f70 0x7fd7cc03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:42.161 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.990+0000 7fd7e4c00700 1 -- 172.21.15.117:0/1288403540 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd7d804d820 con 0x7fd7e8106c20 2024-03-19T03:22:42.161 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.990+0000 7fd7ee600700 1 -- 172.21.15.117:0/1288403540 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7e8063940 con 0x7fd7e8106c20 2024-03-19T03:22:42.161 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.990+0000 7fd7dfe00700 1 --2- 172.21.15.117:0/1288403540 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd7cc038f70 0x7fd7cc03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:42.162 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.991+0000 7fd7dfe00700 1 --2- 172.21.15.117:0/1288403540 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd7cc038f70 0x7fd7cc03b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0006fd0 tx=0x7fd7d0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:42.162 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:40.996+0000 7fd7e4c00700 1 -- 172.21.15.117:0/1288403540 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd7d8017ca0 con 0x7fd7e8106c20 2024-03-19T03:22:42.162 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:41.229+0000 7fd7ee600700 1 -- 172.21.15.117:0/1288403540 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fd7e8062750 con 0x7fd7cc038f70 2024-03-19T03:22:42.162 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:41.417+0000 7fd7e4c00700 1 -- 172.21.15.117:0/1288403540 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fd7e8062750 con 0x7fd7cc038f70 2024-03-19T03:22:42.162 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:41.421+0000 7fd7ee600700 1 -- 172.21.15.117:0/1288403540 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd7cc038f70 msgr2=0x7fd7cc03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:42.162 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:41.421+0000 7fd7ee600700 1 --2- 172.21.15.117:0/1288403540 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd7cc038f70 0x7fd7cc03b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0006fd0 tx=0x7fd7d0006e40 comp rx=0 tx=0).stop 2024-03-19T03:22:42.162 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:41.421+0000 7fd7ee600700 1 -- 172.21.15.117:0/1288403540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7e8106c20 msgr2=0x7fd7e819de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:42.162 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:41.421+0000 7fd7ee600700 1 --2- 172.21.15.117:0/1288403540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7e8106c20 0x7fd7e819de90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd7d80117e0 tx=0x7fd7d8011ba0 comp rx=0 tx=0).stop 2024-03-19T03:22:42.162 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:41.421+0000 7fd7ee600700 1 -- 172.21.15.117:0/1288403540 shutdown_connections 2024-03-19T03:22:42.162 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:41.421+0000 7fd7ee600700 1 --2- 172.21.15.117:0/1288403540 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd7cc038f70 0x7fd7cc03b430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:42.162 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:41.421+0000 7fd7ee600700 1 --2- 172.21.15.117:0/1288403540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7e8106c20 0x7fd7e819de90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:42.162 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:41.421+0000 7fd7ee600700 1 -- 172.21.15.117:0/1288403540 >> 172.21.15.117:0/1288403540 conn(0x7fd7e81021c0 msgr2=0x7fd7e81900a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:42.162 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:41.421+0000 7fd7ee600700 1 -- 172.21.15.117:0/1288403540 shutdown_connections 2024-03-19T03:22:42.162 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:41.421+0000 7fd7ee600700 1 -- 172.21.15.117:0/1288403540 wait complete. 2024-03-19T03:22:42.163 INFO:teuthology.orchestra.run.smithi117.stdout:Creating initial admin user... 2024-03-19T03:22:42.409 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:42 smithi117 ceph-mon[31838]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:22:42.409 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:42 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:22:42.409 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:42 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:22:43.860 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:43 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:22:43.860 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:43 smithi117 ceph-mon[31838]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:22:43.860 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:43 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:22:43.867 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$1ZN0KXH73mlL.saqQpY3wOQllovt.s2CeWBIGQjL9fkc/LXseE3Va", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710818563, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-19T03:22:43.868 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.704+0000 7f038f000700 1 Processor -- start 2024-03-19T03:22:43.868 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.704+0000 7f038f000700 1 -- start start 2024-03-19T03:22:43.869 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.705+0000 7f038f000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0388107db0 0x7f03881081d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:43.869 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.705+0000 7f038f000700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f03881087a0 con 0x7f0388107db0 2024-03-19T03:22:43.869 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.705+0000 7f0387e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0388107db0 0x7f03881081d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:43.869 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.705+0000 7f0387e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0388107db0 0x7f03881081d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:53010/0 (socket says 172.21.15.117:53010) 2024-03-19T03:22:43.870 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.705+0000 7f0387e00700 1 -- 172.21.15.117:0/1425794921 learned_addr learned my addr 172.21.15.117:0/1425794921 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:43.870 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.706+0000 7f0387e00700 1 -- 172.21.15.117:0/1425794921 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0388108f10 con 0x7f0388107db0 2024-03-19T03:22:43.870 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.706+0000 7f0387e00700 1 --2- 172.21.15.117:0/1425794921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0388107db0 0x7f03881081d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f03780098a0 tx=0x7f0378009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2162440b6f39f1b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:43.870 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.706+0000 7f0386a00700 1 -- 172.21.15.117:0/1425794921 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0378004160 con 0x7f0388107db0 2024-03-19T03:22:43.871 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.706+0000 7f0386a00700 1 -- 172.21.15.117:0/1425794921 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0378004300 con 0x7f0388107db0 2024-03-19T03:22:43.871 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.706+0000 7f0386a00700 1 -- 172.21.15.117:0/1425794921 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0378011b20 con 0x7f0388107db0 2024-03-19T03:22:43.871 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.707+0000 7f038f000700 1 -- 172.21.15.117:0/1425794921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0388107db0 msgr2=0x7f03881081d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:43.871 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.707+0000 7f038f000700 1 --2- 172.21.15.117:0/1425794921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0388107db0 0x7f03881081d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f03780098a0 tx=0x7f0378009bb0 comp rx=0 tx=0).stop 2024-03-19T03:22:43.871 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.707+0000 7f038f000700 1 -- 172.21.15.117:0/1425794921 shutdown_connections 2024-03-19T03:22:43.871 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.707+0000 7f038f000700 1 --2- 172.21.15.117:0/1425794921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0388107db0 0x7f03881081d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:43.871 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.707+0000 7f038f000700 1 -- 172.21.15.117:0/1425794921 >> 172.21.15.117:0/1425794921 conn(0x7f0388103330 msgr2=0x7f0388105790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:43.871 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.707+0000 7f038f000700 1 -- 172.21.15.117:0/1425794921 shutdown_connections 2024-03-19T03:22:43.876 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.707+0000 7f038f000700 1 -- 172.21.15.117:0/1425794921 wait complete. 2024-03-19T03:22:43.876 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.708+0000 7f038f000700 1 Processor -- start 2024-03-19T03:22:43.876 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.709+0000 7f038f000700 1 -- start start 2024-03-19T03:22:43.876 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.709+0000 7f038f000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0388107db0 0x7f038819de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:43.877 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.709+0000 7f038f000700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f038810c2c0 con 0x7f0388107db0 2024-03-19T03:22:43.877 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.709+0000 7f0387e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0388107db0 0x7f038819de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:43.877 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.709+0000 7f0387e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0388107db0 0x7f038819de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:53014/0 (socket says 172.21.15.117:53014) 2024-03-19T03:22:43.877 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.709+0000 7f0387e00700 1 -- 172.21.15.117:0/3958623602 learned_addr learned my addr 172.21.15.117:0/3958623602 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:43.877 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.710+0000 7f0387e00700 1 -- 172.21.15.117:0/3958623602 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f038819e3a0 con 0x7f0388107db0 2024-03-19T03:22:43.877 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.710+0000 7f0387e00700 1 --2- 172.21.15.117:0/3958623602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0388107db0 0x7f038819de60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f03780117e0 tx=0x7f0378011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:43.877 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.710+0000 7f0384c00700 1 -- 172.21.15.117:0/3958623602 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0378003bf0 con 0x7f0388107db0 2024-03-19T03:22:43.877 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.710+0000 7f0384c00700 1 -- 172.21.15.117:0/3958623602 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f037801a910 con 0x7f0388107db0 2024-03-19T03:22:43.877 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.710+0000 7f038f000700 1 -- 172.21.15.117:0/3958623602 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f038819e630 con 0x7f0388107db0 2024-03-19T03:22:43.878 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.710+0000 7f0384c00700 1 -- 172.21.15.117:0/3958623602 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0378024970 con 0x7f0388107db0 2024-03-19T03:22:43.878 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.710+0000 7f038f000700 1 -- 172.21.15.117:0/3958623602 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f038819ea90 con 0x7f0388107db0 2024-03-19T03:22:43.878 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.711+0000 7f0384c00700 1 -- 172.21.15.117:0/3958623602 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f0378022020 con 0x7f0388107db0 2024-03-19T03:22:43.878 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.711+0000 7f038f000700 1 -- 172.21.15.117:0/3958623602 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0388063940 con 0x7f0388107db0 2024-03-19T03:22:43.878 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.712+0000 7f0384c00700 1 --2- 172.21.15.117:0/3958623602 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f036c038fd0 0x7f036c03b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:43.878 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.712+0000 7f0384c00700 1 -- 172.21.15.117:0/3958623602 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f037804d3f0 con 0x7f0388107db0 2024-03-19T03:22:43.878 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.712+0000 7f0387400700 1 --2- 172.21.15.117:0/3958623602 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f036c038fd0 0x7f036c03b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:43.878 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.713+0000 7f0387400700 1 --2- 172.21.15.117:0/3958623602 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f036c038fd0 0x7f036c03b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0374006fd0 tx=0x7f0374006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:43.878 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.717+0000 7f0384c00700 1 -- 172.21.15.117:0/3958623602 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f037802d910 con 0x7f0388107db0 2024-03-19T03:22:43.879 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:42.961+0000 7f038f000700 1 -- 172.21.15.117:0/3958623602 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f03881a14a0 con 0x7f036c038fd0 2024-03-19T03:22:43.879 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:43.198+0000 7f0384c00700 1 -- 172.21.15.117:0/3958623602 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f03881a14a0 con 0x7f036c038fd0 2024-03-19T03:22:43.879 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:43.201+0000 7f038f000700 1 -- 172.21.15.117:0/3958623602 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f036c038fd0 msgr2=0x7f036c03b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:43.879 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:43.201+0000 7f038f000700 1 --2- 172.21.15.117:0/3958623602 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f036c038fd0 0x7f036c03b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0374006fd0 tx=0x7f0374006e40 comp rx=0 tx=0).stop 2024-03-19T03:22:43.879 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:43.201+0000 7f038f000700 1 -- 172.21.15.117:0/3958623602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0388107db0 msgr2=0x7f038819de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:43.879 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:43.201+0000 7f038f000700 1 --2- 172.21.15.117:0/3958623602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0388107db0 0x7f038819de60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f03780117e0 tx=0x7f0378011ba0 comp rx=0 tx=0).stop 2024-03-19T03:22:43.879 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:43.201+0000 7f038f000700 1 -- 172.21.15.117:0/3958623602 shutdown_connections 2024-03-19T03:22:43.879 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:43.201+0000 7f038f000700 1 --2- 172.21.15.117:0/3958623602 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f036c038fd0 0x7f036c03b490 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:43.879 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:43.201+0000 7f038f000700 1 --2- 172.21.15.117:0/3958623602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0388107db0 0x7f038819de60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:43.879 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:43.201+0000 7f038f000700 1 -- 172.21.15.117:0/3958623602 >> 172.21.15.117:0/3958623602 conn(0x7f0388103330 msgr2=0x7f0388103d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:43.880 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:43.201+0000 7f038f000700 1 -- 172.21.15.117:0/3958623602 shutdown_connections 2024-03-19T03:22:43.880 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:43.201+0000 7f038f000700 1 -- 172.21.15.117:0/3958623602 wait complete. 2024-03-19T03:22:43.880 INFO:teuthology.orchestra.run.smithi117.stdout:Fetching dashboard port number... 2024-03-19T03:22:45.619 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stdout 8443 2024-03-19T03:22:45.619 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.571+0000 7fc24d800700 1 Processor -- start 2024-03-19T03:22:45.619 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.571+0000 7fc24d800700 1 -- start start 2024-03-19T03:22:45.619 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.571+0000 7fc24d800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc248107ba0 0x7fc248107fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:45.619 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.571+0000 7fc24d800700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc248108590 con 0x7fc248107ba0 2024-03-19T03:22:45.619 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.572+0000 7fc247e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc248107ba0 0x7fc248107fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:45.619 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.572+0000 7fc247e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc248107ba0 0x7fc248107fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:53028/0 (socket says 172.21.15.117:53028) 2024-03-19T03:22:45.619 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.572+0000 7fc247e00700 1 -- 172.21.15.117:0/2746487722 learned_addr learned my addr 172.21.15.117:0/2746487722 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:45.620 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.572+0000 7fc247e00700 1 -- 172.21.15.117:0/2746487722 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc248108d00 con 0x7fc248107ba0 2024-03-19T03:22:45.620 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.573+0000 7fc247e00700 1 --2- 172.21.15.117:0/2746487722 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc248107ba0 0x7fc248107fc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc2380098a0 tx=0x7fc238009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d211d2c55805658f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:45.620 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.573+0000 7fc246a00700 1 -- 172.21.15.117:0/2746487722 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc238004160 con 0x7fc248107ba0 2024-03-19T03:22:45.620 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.573+0000 7fc246a00700 1 -- 172.21.15.117:0/2746487722 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc238004300 con 0x7fc248107ba0 2024-03-19T03:22:45.620 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.573+0000 7fc246a00700 1 -- 172.21.15.117:0/2746487722 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc238011b20 con 0x7fc248107ba0 2024-03-19T03:22:45.620 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.573+0000 7fc24d800700 1 -- 172.21.15.117:0/2746487722 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc248107ba0 msgr2=0x7fc248107fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:45.620 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.573+0000 7fc24d800700 1 --2- 172.21.15.117:0/2746487722 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc248107ba0 0x7fc248107fc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc2380098a0 tx=0x7fc238009bb0 comp rx=0 tx=0).stop 2024-03-19T03:22:45.620 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.574+0000 7fc24d800700 1 -- 172.21.15.117:0/2746487722 shutdown_connections 2024-03-19T03:22:45.620 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.574+0000 7fc24d800700 1 --2- 172.21.15.117:0/2746487722 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc248107ba0 0x7fc248107fc0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:45.620 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.574+0000 7fc24d800700 1 -- 172.21.15.117:0/2746487722 >> 172.21.15.117:0/2746487722 conn(0x7fc248103180 msgr2=0x7fc2481055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:45.620 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.574+0000 7fc24d800700 1 -- 172.21.15.117:0/2746487722 shutdown_connections 2024-03-19T03:22:45.620 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.574+0000 7fc24d800700 1 -- 172.21.15.117:0/2746487722 wait complete. 2024-03-19T03:22:45.620 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.575+0000 7fc24d800700 1 Processor -- start 2024-03-19T03:22:45.620 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.575+0000 7fc24d800700 1 -- start start 2024-03-19T03:22:45.621 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.575+0000 7fc24d800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc248107ba0 0x7fc24819ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:45.621 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.575+0000 7fc24d800700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc24810c0b0 con 0x7fc248107ba0 2024-03-19T03:22:45.621 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.575+0000 7fc247e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc248107ba0 0x7fc24819ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:45.621 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.575+0000 7fc247e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc248107ba0 0x7fc24819ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:53036/0 (socket says 172.21.15.117:53036) 2024-03-19T03:22:45.621 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.575+0000 7fc247e00700 1 -- 172.21.15.117:0/366891876 learned_addr learned my addr 172.21.15.117:0/366891876 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:45.621 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.576+0000 7fc247e00700 1 -- 172.21.15.117:0/366891876 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc24819bfb0 con 0x7fc248107ba0 2024-03-19T03:22:45.621 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.576+0000 7fc247e00700 1 --2- 172.21.15.117:0/366891876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc248107ba0 0x7fc24819ba70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc238000c00 tx=0x7fc238011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:45.621 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.576+0000 7fc244c00700 1 -- 172.21.15.117:0/366891876 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc238003b60 con 0x7fc248107ba0 2024-03-19T03:22:45.621 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.576+0000 7fc244c00700 1 -- 172.21.15.117:0/366891876 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc23801a910 con 0x7fc248107ba0 2024-03-19T03:22:45.621 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.576+0000 7fc24d800700 1 -- 172.21.15.117:0/366891876 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc24819c240 con 0x7fc248107ba0 2024-03-19T03:22:45.621 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.576+0000 7fc244c00700 1 -- 172.21.15.117:0/366891876 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc238024930 con 0x7fc248107ba0 2024-03-19T03:22:45.621 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.577+0000 7fc24d800700 1 -- 172.21.15.117:0/366891876 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc24819c6a0 con 0x7fc248107ba0 2024-03-19T03:22:45.621 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.578+0000 7fc24d800700 1 -- 172.21.15.117:0/366891876 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc248063940 con 0x7fc248107ba0 2024-03-19T03:22:45.621 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.578+0000 7fc244c00700 1 -- 172.21.15.117:0/366891876 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7fc238022020 con 0x7fc248107ba0 2024-03-19T03:22:45.622 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.578+0000 7fc244c00700 1 --2- 172.21.15.117:0/366891876 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc22c038fd0 0x7fc22c03b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:45.622 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.578+0000 7fc244c00700 1 -- 172.21.15.117:0/366891876 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc23804d120 con 0x7fc248107ba0 2024-03-19T03:22:45.622 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.579+0000 7fc247400700 1 --2- 172.21.15.117:0/366891876 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc22c038fd0 0x7fc22c03b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:45.622 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.579+0000 7fc247400700 1 --2- 172.21.15.117:0/366891876 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc22c038fd0 0x7fc22c03b490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc234006fd0 tx=0x7fc234006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:45.622 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.584+0000 7fc244c00700 1 -- 172.21.15.117:0/366891876 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc23800b3e0 con 0x7fc248107ba0 2024-03-19T03:22:45.622 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.748+0000 7fc244c00700 1 -- 172.21.15.117:0/366891876 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7fc238024ad0 con 0x7fc248107ba0 2024-03-19T03:22:45.622 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.813+0000 7fc24d800700 1 -- 172.21.15.117:0/366891876 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fc24819f130 con 0x7fc248107ba0 2024-03-19T03:22:45.622 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.814+0000 7fc244c00700 1 -- 172.21.15.117:0/366891876 <== mon.0 v2:172.21.15.117:3300/0 8 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fc238023370 con 0x7fc248107ba0 2024-03-19T03:22:45.622 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.817+0000 7fc24d800700 1 -- 172.21.15.117:0/366891876 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc22c038fd0 msgr2=0x7fc22c03b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:45.622 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.817+0000 7fc24d800700 1 --2- 172.21.15.117:0/366891876 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc22c038fd0 0x7fc22c03b490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc234006fd0 tx=0x7fc234006e40 comp rx=0 tx=0).stop 2024-03-19T03:22:45.622 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.817+0000 7fc24d800700 1 -- 172.21.15.117:0/366891876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc248107ba0 msgr2=0x7fc24819ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:45.623 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.817+0000 7fc24d800700 1 --2- 172.21.15.117:0/366891876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc248107ba0 0x7fc24819ba70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc238000c00 tx=0x7fc238011c10 comp rx=0 tx=0).stop 2024-03-19T03:22:45.623 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.817+0000 7fc24d800700 1 -- 172.21.15.117:0/366891876 shutdown_connections 2024-03-19T03:22:45.623 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.817+0000 7fc24d800700 1 --2- 172.21.15.117:0/366891876 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc22c038fd0 0x7fc22c03b490 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:45.623 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.817+0000 7fc24d800700 1 --2- 172.21.15.117:0/366891876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc248107ba0 0x7fc24819ba70 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:45.623 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.817+0000 7fc24d800700 1 -- 172.21.15.117:0/366891876 >> 172.21.15.117:0/366891876 conn(0x7fc248103180 msgr2=0x7fc248103d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:45.623 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.817+0000 7fc24d800700 1 -- 172.21.15.117:0/366891876 shutdown_connections 2024-03-19T03:22:45.623 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:44.817+0000 7fc24d800700 1 -- 172.21.15.117:0/366891876 wait complete. 2024-03-19T03:22:45.642 INFO:teuthology.orchestra.run.smithi117.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-19T03:22:45.642 INFO:teuthology.orchestra.run.smithi117.stdout:systemctl: stdout disabled 2024-03-19T03:22:45.662 INFO:teuthology.orchestra.run.smithi117.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-19T03:22:45.663 INFO:teuthology.orchestra.run.smithi117.stdout:systemctl: stdout inactive 2024-03-19T03:22:45.663 INFO:teuthology.orchestra.run.smithi117.stdout:firewalld.service is not enabled 2024-03-19T03:22:45.663 INFO:teuthology.orchestra.run.smithi117.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-19T03:22:45.670 INFO:teuthology.orchestra.run.smithi117.stdout:Ceph Dashboard is now available at: 2024-03-19T03:22:45.670 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:22:45.670 INFO:teuthology.orchestra.run.smithi117.stdout: URL: https://smithi117.front.sepia.ceph.com:8443/ 2024-03-19T03:22:45.670 INFO:teuthology.orchestra.run.smithi117.stdout: User: admin 2024-03-19T03:22:45.670 INFO:teuthology.orchestra.run.smithi117.stdout: Password: 9gbsfqf7ml 2024-03-19T03:22:45.670 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:22:45.671 INFO:teuthology.orchestra.run.smithi117.stdout:Saving cluster configuration to /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/config directory 2024-03-19T03:22:45.673 INFO:teuthology.orchestra.run.smithi117.stdout:Enabling autotune for osd_memory_target 2024-03-19T03:22:45.904 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:45 smithi117 ceph-mon[31838]: mgrmap e13: x(active, since 6s) 2024-03-19T03:22:45.904 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:45 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/366891876' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-19T03:22:47.599 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.732+0000 7fe632400700 1 Processor -- start 2024-03-19T03:22:47.599 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.732+0000 7fe632400700 1 -- start start 2024-03-19T03:22:47.599 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.732+0000 7fe632400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62c1087d0 0x7fe62c10abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:47.599 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.732+0000 7fe632400700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe62c07be00 con 0x7fe62c1087d0 2024-03-19T03:22:47.599 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.733+0000 7fe62b400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62c1087d0 0x7fe62c10abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:47.600 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.733+0000 7fe62b400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62c1087d0 0x7fe62c10abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:56830/0 (socket says 172.21.15.117:56830) 2024-03-19T03:22:47.600 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.733+0000 7fe62b400700 1 -- 172.21.15.117:0/3058970076 learned_addr learned my addr 172.21.15.117:0/3058970076 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:47.600 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.733+0000 7fe62b400700 1 -- 172.21.15.117:0/3058970076 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe62c10b100 con 0x7fe62c1087d0 2024-03-19T03:22:47.600 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.734+0000 7fe62b400700 1 --2- 172.21.15.117:0/3058970076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62c1087d0 0x7fe62c10abc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe61c0098a0 tx=0x7fe61c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba8680a6beb971cb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:47.600 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.734+0000 7fe62a000700 1 -- 172.21.15.117:0/3058970076 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe61c004160 con 0x7fe62c1087d0 2024-03-19T03:22:47.600 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.734+0000 7fe62a000700 1 -- 172.21.15.117:0/3058970076 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe61c004300 con 0x7fe62c1087d0 2024-03-19T03:22:47.600 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.734+0000 7fe62a000700 1 -- 172.21.15.117:0/3058970076 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe61c011b20 con 0x7fe62c1087d0 2024-03-19T03:22:47.600 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.734+0000 7fe632400700 1 -- 172.21.15.117:0/3058970076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62c1087d0 msgr2=0x7fe62c10abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:47.600 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.734+0000 7fe632400700 1 --2- 172.21.15.117:0/3058970076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62c1087d0 0x7fe62c10abc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe61c0098a0 tx=0x7fe61c009bb0 comp rx=0 tx=0).stop 2024-03-19T03:22:47.600 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.735+0000 7fe632400700 1 -- 172.21.15.117:0/3058970076 shutdown_connections 2024-03-19T03:22:47.600 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.735+0000 7fe632400700 1 --2- 172.21.15.117:0/3058970076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62c1087d0 0x7fe62c10abc0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:47.600 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.735+0000 7fe632400700 1 -- 172.21.15.117:0/3058970076 >> 172.21.15.117:0/3058970076 conn(0x7fe62c1021c0 msgr2=0x7fe62c104600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:47.601 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.735+0000 7fe632400700 1 -- 172.21.15.117:0/3058970076 shutdown_connections 2024-03-19T03:22:47.601 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.735+0000 7fe632400700 1 -- 172.21.15.117:0/3058970076 wait complete. 2024-03-19T03:22:47.601 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.736+0000 7fe632400700 1 Processor -- start 2024-03-19T03:22:47.601 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.736+0000 7fe632400700 1 -- start start 2024-03-19T03:22:47.601 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.736+0000 7fe632400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62c1087d0 0x7fe62c19de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:47.601 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.736+0000 7fe632400700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe62c10c160 con 0x7fe62c1087d0 2024-03-19T03:22:47.601 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.736+0000 7fe62b400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62c1087d0 0x7fe62c19de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:47.601 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.736+0000 7fe62b400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62c1087d0 0x7fe62c19de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:56836/0 (socket says 172.21.15.117:56836) 2024-03-19T03:22:47.601 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.736+0000 7fe62b400700 1 -- 172.21.15.117:0/606601359 learned_addr learned my addr 172.21.15.117:0/606601359 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:47.601 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.737+0000 7fe62b400700 1 -- 172.21.15.117:0/606601359 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe62c19e380 con 0x7fe62c1087d0 2024-03-19T03:22:47.601 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.737+0000 7fe62b400700 1 --2- 172.21.15.117:0/606601359 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62c1087d0 0x7fe62c19de40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe61c0117e0 tx=0x7fe61c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:47.602 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.737+0000 7fe623e00700 1 -- 172.21.15.117:0/606601359 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe61c003ba0 con 0x7fe62c1087d0 2024-03-19T03:22:47.602 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.737+0000 7fe623e00700 1 -- 172.21.15.117:0/606601359 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe61c01a910 con 0x7fe62c1087d0 2024-03-19T03:22:47.602 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.737+0000 7fe623e00700 1 -- 172.21.15.117:0/606601359 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe61c0248f0 con 0x7fe62c1087d0 2024-03-19T03:22:47.602 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.737+0000 7fe632400700 1 -- 172.21.15.117:0/606601359 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe62c19e610 con 0x7fe62c1087d0 2024-03-19T03:22:47.602 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.738+0000 7fe632400700 1 -- 172.21.15.117:0/606601359 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe62c19ea70 con 0x7fe62c1087d0 2024-03-19T03:22:47.602 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.739+0000 7fe632400700 1 -- 172.21.15.117:0/606601359 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe62c063940 con 0x7fe62c1087d0 2024-03-19T03:22:47.602 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.739+0000 7fe623e00700 1 -- 172.21.15.117:0/606601359 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7fe61c022020 con 0x7fe62c1087d0 2024-03-19T03:22:47.602 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.739+0000 7fe623e00700 1 --2- 172.21.15.117:0/606601359 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe610039070 0x7fe61003b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:47.602 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.739+0000 7fe623e00700 1 -- 172.21.15.117:0/606601359 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe61c04d570 con 0x7fe62c1087d0 2024-03-19T03:22:47.602 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.740+0000 7fe62aa00700 1 --2- 172.21.15.117:0/606601359 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe610039070 0x7fe61003b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:47.602 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.740+0000 7fe62aa00700 1 --2- 172.21.15.117:0/606601359 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe610039070 0x7fe61003b530 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe618006fd0 tx=0x7fe618006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:47.602 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.744+0000 7fe623e00700 1 -- 172.21.15.117:0/606601359 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe61c00b3e0 con 0x7fe62c1087d0 2024-03-19T03:22:47.602 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.964+0000 7fe632400700 1 -- 172.21.15.117:0/606601359 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fe62c1a1540 con 0x7fe62c1087d0 2024-03-19T03:22:47.602 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.965+0000 7fe623e00700 1 -- 172.21.15.117:0/606601359 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fe61c01ab70 con 0x7fe62c1087d0 2024-03-19T03:22:47.603 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.968+0000 7fe632400700 1 -- 172.21.15.117:0/606601359 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe610039070 msgr2=0x7fe61003b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:47.603 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.968+0000 7fe632400700 1 --2- 172.21.15.117:0/606601359 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe610039070 0x7fe61003b530 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe618006fd0 tx=0x7fe618006e40 comp rx=0 tx=0).stop 2024-03-19T03:22:47.603 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.969+0000 7fe632400700 1 -- 172.21.15.117:0/606601359 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62c1087d0 msgr2=0x7fe62c19de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:47.603 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.969+0000 7fe632400700 1 --2- 172.21.15.117:0/606601359 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62c1087d0 0x7fe62c19de40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe61c0117e0 tx=0x7fe61c011ba0 comp rx=0 tx=0).stop 2024-03-19T03:22:47.603 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.969+0000 7fe632400700 1 -- 172.21.15.117:0/606601359 shutdown_connections 2024-03-19T03:22:47.603 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.969+0000 7fe632400700 1 --2- 172.21.15.117:0/606601359 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe610039070 0x7fe61003b530 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:47.603 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.969+0000 7fe632400700 1 --2- 172.21.15.117:0/606601359 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe62c1087d0 0x7fe62c19de40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:47.603 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.969+0000 7fe632400700 1 -- 172.21.15.117:0/606601359 >> 172.21.15.117:0/606601359 conn(0x7fe62c1021c0 msgr2=0x7fe62c102bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:47.603 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.969+0000 7fe632400700 1 -- 172.21.15.117:0/606601359 shutdown_connections 2024-03-19T03:22:47.603 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:46.969+0000 7fe632400700 1 -- 172.21.15.117:0/606601359 wait complete. 2024-03-19T03:22:49.135 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.170+0000 7f111b200700 1 Processor -- start 2024-03-19T03:22:49.135 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.170+0000 7f111b200700 1 -- start start 2024-03-19T03:22:49.135 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.171+0000 7f111b200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f111c07ddf0 0x7f111c07c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:49.135 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.171+0000 7f111b200700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f111c07c820 con 0x7f111c07ddf0 2024-03-19T03:22:49.135 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.171+0000 7f1119e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f111c07ddf0 0x7f111c07c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:49.135 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.171+0000 7f1119e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f111c07ddf0 0x7f111c07c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:56852/0 (socket says 172.21.15.117:56852) 2024-03-19T03:22:49.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.171+0000 7f1119e00700 1 -- 172.21.15.117:0/2206941391 learned_addr learned my addr 172.21.15.117:0/2206941391 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:49.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.172+0000 7f1119e00700 1 -- 172.21.15.117:0/2206941391 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f111c07c9a0 con 0x7f111c07ddf0 2024-03-19T03:22:49.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.172+0000 7f1119e00700 1 --2- 172.21.15.117:0/2206941391 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f111c07ddf0 0x7f111c07c250 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f110c009e80 tx=0x7f110c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f213fe94352c0a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:49.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.172+0000 7f1119400700 1 -- 172.21.15.117:0/2206941391 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f110c017070 con 0x7f111c07ddf0 2024-03-19T03:22:49.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.172+0000 7f1119400700 1 -- 172.21.15.117:0/2206941391 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f110c00b8c0 con 0x7f111c07ddf0 2024-03-19T03:22:49.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.173+0000 7f1119400700 1 -- 172.21.15.117:0/2206941391 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f110c0045c0 con 0x7f111c07ddf0 2024-03-19T03:22:49.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.173+0000 7f111b200700 1 -- 172.21.15.117:0/2206941391 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f111c07ddf0 msgr2=0x7f111c07c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:49.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.173+0000 7f111b200700 1 --2- 172.21.15.117:0/2206941391 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f111c07ddf0 0x7f111c07c250 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f110c009e80 tx=0x7f110c00b330 comp rx=0 tx=0).stop 2024-03-19T03:22:49.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.174+0000 7f111b200700 1 -- 172.21.15.117:0/2206941391 shutdown_connections 2024-03-19T03:22:49.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.174+0000 7f111b200700 1 --2- 172.21.15.117:0/2206941391 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f111c07ddf0 0x7f111c07c250 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:49.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.174+0000 7f111b200700 1 -- 172.21.15.117:0/2206941391 >> 172.21.15.117:0/2206941391 conn(0x7f111c103120 msgr2=0x7f111c105540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:49.136 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.174+0000 7f111b200700 1 -- 172.21.15.117:0/2206941391 shutdown_connections 2024-03-19T03:22:49.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.174+0000 7f111b200700 1 -- 172.21.15.117:0/2206941391 wait complete. 2024-03-19T03:22:49.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.175+0000 7f111b200700 1 Processor -- start 2024-03-19T03:22:49.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.175+0000 7f111b200700 1 -- start start 2024-03-19T03:22:49.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.175+0000 7f111b200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f111c07ddf0 0x7f111c19dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:49.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.175+0000 7f111b200700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f111c10e180 con 0x7f111c07ddf0 2024-03-19T03:22:49.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.176+0000 7f1119e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f111c07ddf0 0x7f111c19dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:49.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.176+0000 7f1119e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f111c07ddf0 0x7f111c19dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:56862/0 (socket says 172.21.15.117:56862) 2024-03-19T03:22:49.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.176+0000 7f1119e00700 1 -- 172.21.15.117:0/1074710899 learned_addr learned my addr 172.21.15.117:0/1074710899 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:49.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.176+0000 7f1119e00700 1 -- 172.21.15.117:0/1074710899 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f111c19e1a0 con 0x7f111c07ddf0 2024-03-19T03:22:49.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.176+0000 7f1119e00700 1 --2- 172.21.15.117:0/1074710899 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f111c07ddf0 0x7f111c19dc60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f110c009fd0 tx=0x7f110c003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:49.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.176+0000 7f1112a00700 1 -- 172.21.15.117:0/1074710899 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f110c020020 con 0x7f111c07ddf0 2024-03-19T03:22:49.137 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.176+0000 7f1112a00700 1 -- 172.21.15.117:0/1074710899 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f110c00bba0 con 0x7f111c07ddf0 2024-03-19T03:22:49.138 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.176+0000 7f1112a00700 1 -- 172.21.15.117:0/1074710899 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f110c017040 con 0x7f111c07ddf0 2024-03-19T03:22:49.138 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.176+0000 7f111b200700 1 -- 172.21.15.117:0/1074710899 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f111c19e430 con 0x7f111c07ddf0 2024-03-19T03:22:49.138 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.177+0000 7f111b200700 1 -- 172.21.15.117:0/1074710899 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f111c1a1120 con 0x7f111c07ddf0 2024-03-19T03:22:49.138 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.178+0000 7f111b200700 1 -- 172.21.15.117:0/1074710899 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f111c063940 con 0x7f111c07ddf0 2024-03-19T03:22:49.138 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.178+0000 7f1112a00700 1 -- 172.21.15.117:0/1074710899 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f110c023020 con 0x7f111c07ddf0 2024-03-19T03:22:49.138 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.178+0000 7f1112a00700 1 --2- 172.21.15.117:0/1074710899 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1100039070 0x7f110003b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:49.138 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.178+0000 7f1112a00700 1 -- 172.21.15.117:0/1074710899 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f110c00f050 con 0x7f111c07ddf0 2024-03-19T03:22:49.138 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.179+0000 7f1113e00700 1 --2- 172.21.15.117:0/1074710899 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1100039070 0x7f110003b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:49.138 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.179+0000 7f1113e00700 1 --2- 172.21.15.117:0/1074710899 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1100039070 0x7f110003b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1104006fd0 tx=0x7f1104006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:49.138 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.184+0000 7f1112a00700 1 -- 172.21.15.117:0/1074710899 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f110c02a020 con 0x7f111c07ddf0 2024-03-19T03:22:49.138 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.519+0000 7f111b200700 1 -- 172.21.15.117:0/1074710899 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f111c1a16f0 con 0x7f111c07ddf0 2024-03-19T03:22:49.138 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.523+0000 7f1112a00700 1 -- 172.21.15.117:0/1074710899 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f110c01bbb0 con 0x7f111c07ddf0 2024-03-19T03:22:49.139 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.527+0000 7f111b200700 1 -- 172.21.15.117:0/1074710899 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1100039070 msgr2=0x7f110003b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:49.139 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.527+0000 7f111b200700 1 --2- 172.21.15.117:0/1074710899 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1100039070 0x7f110003b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1104006fd0 tx=0x7f1104006e40 comp rx=0 tx=0).stop 2024-03-19T03:22:49.139 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.527+0000 7f111b200700 1 -- 172.21.15.117:0/1074710899 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f111c07ddf0 msgr2=0x7f111c19dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:49.139 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.527+0000 7f111b200700 1 --2- 172.21.15.117:0/1074710899 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f111c07ddf0 0x7f111c19dc60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f110c009fd0 tx=0x7f110c003d00 comp rx=0 tx=0).stop 2024-03-19T03:22:49.139 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.527+0000 7f111b200700 1 -- 172.21.15.117:0/1074710899 shutdown_connections 2024-03-19T03:22:49.139 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.527+0000 7f111b200700 1 --2- 172.21.15.117:0/1074710899 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1100039070 0x7f110003b530 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:49.139 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.527+0000 7f111b200700 1 --2- 172.21.15.117:0/1074710899 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f111c07ddf0 0x7f111c19dc60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:49.139 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.527+0000 7f111b200700 1 -- 172.21.15.117:0/1074710899 >> 172.21.15.117:0/1074710899 conn(0x7f111c103120 msgr2=0x7f111c103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:49.139 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.527+0000 7f111b200700 1 -- 172.21.15.117:0/1074710899 shutdown_connections 2024-03-19T03:22:49.139 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr 2024-03-19T03:22:48.527+0000 7f111b200700 1 -- 172.21.15.117:0/1074710899 wait complete. 2024-03-19T03:22:49.139 INFO:teuthology.orchestra.run.smithi117.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-19T03:22:49.139 INFO:teuthology.orchestra.run.smithi117.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-19T03:22:49.140 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:22:49.140 INFO:teuthology.orchestra.run.smithi117.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-19T03:22:49.140 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:22:49.140 INFO:teuthology.orchestra.run.smithi117.stdout:Or, if you are only running a single cluster on this host: 2024-03-19T03:22:49.140 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:22:49.140 INFO:teuthology.orchestra.run.smithi117.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-19T03:22:49.140 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:22:49.140 INFO:teuthology.orchestra.run.smithi117.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-19T03:22:49.140 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:22:49.140 INFO:teuthology.orchestra.run.smithi117.stdout: ceph telemetry on 2024-03-19T03:22:49.140 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:22:49.140 INFO:teuthology.orchestra.run.smithi117.stdout:For more information see: 2024-03-19T03:22:49.140 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:22:49.141 INFO:teuthology.orchestra.run.smithi117.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-19T03:22:49.141 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:22:49.141 INFO:teuthology.orchestra.run.smithi117.stdout:Bootstrap complete. 2024-03-19T03:22:49.235 INFO:tasks.cephadm:Fetching config... 2024-03-19T03:22:49.236 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-19T03:22:49.236 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-19T03:22:49.286 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-19T03:22:49.286 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-19T03:22:49.287 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-19T03:22:49.367 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-19T03:22:49.367 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-19T03:22:49.367 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-19T03:22:49.492 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-19T03:22:49.492 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-19T03:22:49.492 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-19T03:22:49.586 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-19T03:22:49.586 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUgcFuirggaJ0n2PfVLR57W9HZ8bjQ6pWJJ1sctYDlL6TJkJZYjU+l1tNatnr2YBM65g1XMG0zwxmbALvvbLRgIz50e0F3JFIq6uo2aNVGc9mdF8ZcgZSumbefUKJKHFWcfLemYOYOGdokGW09zyC5nGqjgG4u00uOyeQItp2VHYz04G4EOOjOVp/w5rPRGuhvAlpk/X/WvrjWNNYxRn/xIRFnA9CNTmm+6D5aMTPfGv6um+k4MYaASXjpdY8F/v66lkBYJzkuxKzDjtMC1Stsk5v5FSmiFUVO3tYDZZFOPAZF0tSiwBHzbA7Ri7rR42oj9Fv6aLOfMUFa8HBK0t6kxGAtHTJPst2vtyr+Ewd1gydqwsTXoIBunaFLQdMMrY5GMb6l3DZWAhWF9uEF1W+qZmj7T44XF7KTVuQp6Dlvr+LnzOZBg+QnlRGmTmLtxIX7M9UjE1uGR4PIrjN/IxChq+UgoydBzU6oLRkHkEouITuKErisRQwUzR/aPhJ+aFU= ceph-a3a82456-e59f-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-19T03:22:49.670 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:49 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1074710899' entity='client.admin' 2024-03-19T03:22:49.711 INFO:teuthology.orchestra.run.smithi117.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUgcFuirggaJ0n2PfVLR57W9HZ8bjQ6pWJJ1sctYDlL6TJkJZYjU+l1tNatnr2YBM65g1XMG0zwxmbALvvbLRgIz50e0F3JFIq6uo2aNVGc9mdF8ZcgZSumbefUKJKHFWcfLemYOYOGdokGW09zyC5nGqjgG4u00uOyeQItp2VHYz04G4EOOjOVp/w5rPRGuhvAlpk/X/WvrjWNNYxRn/xIRFnA9CNTmm+6D5aMTPfGv6um+k4MYaASXjpdY8F/v66lkBYJzkuxKzDjtMC1Stsk5v5FSmiFUVO3tYDZZFOPAZF0tSiwBHzbA7Ri7rR42oj9Fv6aLOfMUFa8HBK0t6kxGAtHTJPst2vtyr+Ewd1gydqwsTXoIBunaFLQdMMrY5GMb6l3DZWAhWF9uEF1W+qZmj7T44XF7KTVuQp6Dlvr+LnzOZBg+QnlRGmTmLtxIX7M9UjE1uGR4PIrjN/IxChq+UgoydBzU6oLRkHkEouITuKErisRQwUzR/aPhJ+aFU= ceph-a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:22:49.771 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUgcFuirggaJ0n2PfVLR57W9HZ8bjQ6pWJJ1sctYDlL6TJkJZYjU+l1tNatnr2YBM65g1XMG0zwxmbALvvbLRgIz50e0F3JFIq6uo2aNVGc9mdF8ZcgZSumbefUKJKHFWcfLemYOYOGdokGW09zyC5nGqjgG4u00uOyeQItp2VHYz04G4EOOjOVp/w5rPRGuhvAlpk/X/WvrjWNNYxRn/xIRFnA9CNTmm+6D5aMTPfGv6um+k4MYaASXjpdY8F/v66lkBYJzkuxKzDjtMC1Stsk5v5FSmiFUVO3tYDZZFOPAZF0tSiwBHzbA7Ri7rR42oj9Fv6aLOfMUFa8HBK0t6kxGAtHTJPst2vtyr+Ewd1gydqwsTXoIBunaFLQdMMrY5GMb6l3DZWAhWF9uEF1W+qZmj7T44XF7KTVuQp6Dlvr+LnzOZBg+QnlRGmTmLtxIX7M9UjE1uGR4PIrjN/IxChq+UgoydBzU6oLRkHkEouITuKErisRQwUzR/aPhJ+aFU= ceph-a3a82456-e59f-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-19T03:22:49.817 INFO:teuthology.orchestra.run.smithi119.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUgcFuirggaJ0n2PfVLR57W9HZ8bjQ6pWJJ1sctYDlL6TJkJZYjU+l1tNatnr2YBM65g1XMG0zwxmbALvvbLRgIz50e0F3JFIq6uo2aNVGc9mdF8ZcgZSumbefUKJKHFWcfLemYOYOGdokGW09zyC5nGqjgG4u00uOyeQItp2VHYz04G4EOOjOVp/w5rPRGuhvAlpk/X/WvrjWNNYxRn/xIRFnA9CNTmm+6D5aMTPfGv6um+k4MYaASXjpdY8F/v66lkBYJzkuxKzDjtMC1Stsk5v5FSmiFUVO3tYDZZFOPAZF0tSiwBHzbA7Ri7rR42oj9Fv6aLOfMUFa8HBK0t6kxGAtHTJPst2vtyr+Ewd1gydqwsTXoIBunaFLQdMMrY5GMb6l3DZWAhWF9uEF1W+qZmj7T44XF7KTVuQp6Dlvr+LnzOZBg+QnlRGmTmLtxIX7M9UjE1uGR4PIrjN/IxChq+UgoydBzU6oLRkHkEouITuKErisRQwUzR/aPhJ+aFU= ceph-a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:22:49.833 DEBUG:teuthology.orchestra.run.smithi178:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUgcFuirggaJ0n2PfVLR57W9HZ8bjQ6pWJJ1sctYDlL6TJkJZYjU+l1tNatnr2YBM65g1XMG0zwxmbALvvbLRgIz50e0F3JFIq6uo2aNVGc9mdF8ZcgZSumbefUKJKHFWcfLemYOYOGdokGW09zyC5nGqjgG4u00uOyeQItp2VHYz04G4EOOjOVp/w5rPRGuhvAlpk/X/WvrjWNNYxRn/xIRFnA9CNTmm+6D5aMTPfGv6um+k4MYaASXjpdY8F/v66lkBYJzkuxKzDjtMC1Stsk5v5FSmiFUVO3tYDZZFOPAZF0tSiwBHzbA7Ri7rR42oj9Fv6aLOfMUFa8HBK0t6kxGAtHTJPst2vtyr+Ewd1gydqwsTXoIBunaFLQdMMrY5GMb6l3DZWAhWF9uEF1W+qZmj7T44XF7KTVuQp6Dlvr+LnzOZBg+QnlRGmTmLtxIX7M9UjE1uGR4PIrjN/IxChq+UgoydBzU6oLRkHkEouITuKErisRQwUzR/aPhJ+aFU= ceph-a3a82456-e59f-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-19T03:22:49.878 INFO:teuthology.orchestra.run.smithi178.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUgcFuirggaJ0n2PfVLR57W9HZ8bjQ6pWJJ1sctYDlL6TJkJZYjU+l1tNatnr2YBM65g1XMG0zwxmbALvvbLRgIz50e0F3JFIq6uo2aNVGc9mdF8ZcgZSumbefUKJKHFWcfLemYOYOGdokGW09zyC5nGqjgG4u00uOyeQItp2VHYz04G4EOOjOVp/w5rPRGuhvAlpk/X/WvrjWNNYxRn/xIRFnA9CNTmm+6D5aMTPfGv6um+k4MYaASXjpdY8F/v66lkBYJzkuxKzDjtMC1Stsk5v5FSmiFUVO3tYDZZFOPAZF0tSiwBHzbA7Ri7rR42oj9Fv6aLOfMUFa8HBK0t6kxGAtHTJPst2vtyr+Ewd1gydqwsTXoIBunaFLQdMMrY5GMb6l3DZWAhWF9uEF1W+qZmj7T44XF7KTVuQp6Dlvr+LnzOZBg+QnlRGmTmLtxIX7M9UjE1uGR4PIrjN/IxChq+UgoydBzU6oLRkHkEouITuKErisRQwUzR/aPhJ+aFU= ceph-a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:22:49.893 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-19T03:22:50.298 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:22:52.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.394+0000 7f401b200700 1 -- 172.21.15.117:0/2876383118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f401c103d60 msgr2=0x7f401c104180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:52.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.394+0000 7f401b200700 1 --2- 172.21.15.117:0/2876383118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f401c103d60 0x7f401c104180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f400c009910 tx=0x7f400c009c20 comp rx=0 tx=0).stop 2024-03-19T03:22:52.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.394+0000 7f401b200700 1 -- 172.21.15.117:0/2876383118 shutdown_connections 2024-03-19T03:22:52.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.394+0000 7f401b200700 1 --2- 172.21.15.117:0/2876383118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f401c103d60 0x7f401c104180 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:52.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.394+0000 7f401b200700 1 -- 172.21.15.117:0/2876383118 >> 172.21.15.117:0/2876383118 conn(0x7f401c0ff300 msgr2=0x7f401c101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:52.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.394+0000 7f401b200700 1 -- 172.21.15.117:0/2876383118 shutdown_connections 2024-03-19T03:22:52.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.394+0000 7f401b200700 1 -- 172.21.15.117:0/2876383118 wait complete. 2024-03-19T03:22:52.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.395+0000 7f401b200700 1 Processor -- start 2024-03-19T03:22:52.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.396+0000 7f401b200700 1 -- start start 2024-03-19T03:22:52.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.396+0000 7f401b200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f401c103d60 0x7f401c1994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:52.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.396+0000 7f401b200700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f401c109df0 con 0x7f401c103d60 2024-03-19T03:22:52.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.396+0000 7f4019e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f401c103d60 0x7f401c1994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:52.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.396+0000 7f4019e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f401c103d60 0x7f401c1994f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:56880/0 (socket says 172.21.15.117:56880) 2024-03-19T03:22:52.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.396+0000 7f4019e00700 1 -- 172.21.15.117:0/2788860670 learned_addr learned my addr 172.21.15.117:0/2788860670 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:52.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.397+0000 7f4019e00700 1 -- 172.21.15.117:0/2788860670 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f401c199a30 con 0x7f401c103d60 2024-03-19T03:22:52.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.397+0000 7f4019e00700 1 --2- 172.21.15.117:0/2788860670 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f401c103d60 0x7f401c1994f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f400c005850 tx=0x7f400c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:52.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.397+0000 7f4012a00700 1 -- 172.21.15.117:0/2788860670 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f400c026070 con 0x7f401c103d60 2024-03-19T03:22:52.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.397+0000 7f4012a00700 1 -- 172.21.15.117:0/2788860670 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f400c0044a0 con 0x7f401c103d60 2024-03-19T03:22:52.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.397+0000 7f401b200700 1 -- 172.21.15.117:0/2788860670 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f401c199cc0 con 0x7f401c103d60 2024-03-19T03:22:52.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.397+0000 7f4012a00700 1 -- 172.21.15.117:0/2788860670 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f400c003ab0 con 0x7f401c103d60 2024-03-19T03:22:52.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.398+0000 7f401b200700 1 -- 172.21.15.117:0/2788860670 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f401c19c960 con 0x7f401c103d60 2024-03-19T03:22:52.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.398+0000 7f4012a00700 1 -- 172.21.15.117:0/2788860670 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f400c029070 con 0x7f401c103d60 2024-03-19T03:22:52.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.399+0000 7f401b200700 1 -- 172.21.15.117:0/2788860670 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f401c10a570 con 0x7f401c103d60 2024-03-19T03:22:52.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.399+0000 7f4012a00700 1 --2- 172.21.15.117:0/2788860670 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4000039020 0x7f400003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:52.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.399+0000 7f4012a00700 1 -- 172.21.15.117:0/2788860670 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f400c01e070 con 0x7f401c103d60 2024-03-19T03:22:52.400 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.399+0000 7f4019400700 1 --2- 172.21.15.117:0/2788860670 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4000039020 0x7f400003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:52.401 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.400+0000 7f4019400700 1 --2- 172.21.15.117:0/2788860670 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4000039020 0x7f400003b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4008006fd0 tx=0x7f4008006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:52.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.405+0000 7f4012a00700 1 -- 172.21.15.117:0/2788860670 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f400c013070 con 0x7f401c103d60 2024-03-19T03:22:52.630 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.629+0000 7f401b200700 1 -- 172.21.15.117:0/2788860670 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f401c02dcc0 con 0x7f401c103d60 2024-03-19T03:22:52.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.632+0000 7f4012a00700 1 -- 172.21.15.117:0/2788860670 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f400c030020 con 0x7f401c103d60 2024-03-19T03:22:52.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.635+0000 7f401b200700 1 -- 172.21.15.117:0/2788860670 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4000039020 msgr2=0x7f400003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:52.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.635+0000 7f401b200700 1 --2- 172.21.15.117:0/2788860670 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4000039020 0x7f400003b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4008006fd0 tx=0x7f4008006e40 comp rx=0 tx=0).stop 2024-03-19T03:22:52.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.635+0000 7f401b200700 1 -- 172.21.15.117:0/2788860670 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f401c103d60 msgr2=0x7f401c1994f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:52.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.635+0000 7f401b200700 1 --2- 172.21.15.117:0/2788860670 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f401c103d60 0x7f401c1994f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f400c005850 tx=0x7f400c005030 comp rx=0 tx=0).stop 2024-03-19T03:22:52.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.635+0000 7f401b200700 1 -- 172.21.15.117:0/2788860670 shutdown_connections 2024-03-19T03:22:52.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.635+0000 7f401b200700 1 --2- 172.21.15.117:0/2788860670 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4000039020 0x7f400003b4e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:52.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.635+0000 7f401b200700 1 --2- 172.21.15.117:0/2788860670 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f401c103d60 0x7f401c1994f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:52.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.635+0000 7f401b200700 1 -- 172.21.15.117:0/2788860670 >> 172.21.15.117:0/2788860670 conn(0x7f401c0ff300 msgr2=0x7f401c0ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:52.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.635+0000 7f401b200700 1 -- 172.21.15.117:0/2788860670 shutdown_connections 2024-03-19T03:22:52.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:52.636+0000 7f401b200700 1 -- 172.21.15.117:0/2788860670 wait complete. 2024-03-19T03:22:53.801 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-19T03:22:53.802 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-19T03:22:53.888 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:53 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2788860670' entity='client.admin' 2024-03-19T03:22:54.221 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:22:56.738 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:56 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:22:56.738 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:56 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:22:56.738 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:56 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi117", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:22:56.738 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:56 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:22:56.738 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:56 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:22:56.738 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:56 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:22:56.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.904+0000 7fd407400700 1 -- 172.21.15.117:0/2598927105 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd400101d40 msgr2=0x7fd400102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:56.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.904+0000 7fd407400700 1 --2- 172.21.15.117:0/2598927105 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd400101d40 0x7fd400102160 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4009910 tx=0x7fd3f4009c20 comp rx=0 tx=0).stop 2024-03-19T03:22:56.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.904+0000 7fd407400700 1 -- 172.21.15.117:0/2598927105 shutdown_connections 2024-03-19T03:22:56.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.904+0000 7fd407400700 1 --2- 172.21.15.117:0/2598927105 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd400101d40 0x7fd400102160 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:56.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.904+0000 7fd407400700 1 -- 172.21.15.117:0/2598927105 >> 172.21.15.117:0/2598927105 conn(0x7fd4000fd300 msgr2=0x7fd4000ff720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:56.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.904+0000 7fd407400700 1 -- 172.21.15.117:0/2598927105 shutdown_connections 2024-03-19T03:22:56.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.904+0000 7fd407400700 1 -- 172.21.15.117:0/2598927105 wait complete. 2024-03-19T03:22:56.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.906+0000 7fd407400700 1 Processor -- start 2024-03-19T03:22:56.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.906+0000 7fd407400700 1 -- start start 2024-03-19T03:22:56.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.907+0000 7fd407400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd400101d40 0x7fd400197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:56.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.907+0000 7fd407400700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd400107dd0 con 0x7fd400101d40 2024-03-19T03:22:56.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.907+0000 7fd404a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd400101d40 0x7fd400197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:56.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.907+0000 7fd404a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd400101d40 0x7fd400197480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:52894/0 (socket says 172.21.15.117:52894) 2024-03-19T03:22:56.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.907+0000 7fd404a00700 1 -- 172.21.15.117:0/527621058 learned_addr learned my addr 172.21.15.117:0/527621058 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:22:56.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.907+0000 7fd404a00700 1 -- 172.21.15.117:0/527621058 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4001979c0 con 0x7fd400101d40 2024-03-19T03:22:56.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.908+0000 7fd404a00700 1 --2- 172.21.15.117:0/527621058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd400101d40 0x7fd400197480 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4005850 tx=0x7fd3f4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:56.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.908+0000 7fd3fd600700 1 -- 172.21.15.117:0/527621058 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd3f4026070 con 0x7fd400101d40 2024-03-19T03:22:56.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.908+0000 7fd407400700 1 -- 172.21.15.117:0/527621058 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd400197c50 con 0x7fd400101d40 2024-03-19T03:22:56.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.908+0000 7fd3fd600700 1 -- 172.21.15.117:0/527621058 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3f40044a0 con 0x7fd400101d40 2024-03-19T03:22:56.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.908+0000 7fd3fd600700 1 -- 172.21.15.117:0/527621058 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd3f4003ab0 con 0x7fd400101d40 2024-03-19T03:22:56.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.908+0000 7fd407400700 1 -- 172.21.15.117:0/527621058 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd40019a940 con 0x7fd400101d40 2024-03-19T03:22:56.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.909+0000 7fd3fd600700 1 -- 172.21.15.117:0/527621058 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7fd3f4029070 con 0x7fd400101d40 2024-03-19T03:22:56.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.909+0000 7fd407400700 1 -- 172.21.15.117:0/527621058 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd400108550 con 0x7fd400101d40 2024-03-19T03:22:56.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.910+0000 7fd3fd600700 1 --2- 172.21.15.117:0/527621058 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd3e8038fc0 0x7fd3e803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:22:56.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.910+0000 7fd3fd600700 1 -- 172.21.15.117:0/527621058 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd3f401e070 con 0x7fd400101d40 2024-03-19T03:22:56.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.910+0000 7fd3ffe00700 1 --2- 172.21.15.117:0/527621058 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd3e8038fc0 0x7fd3e803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:22:56.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.911+0000 7fd3ffe00700 1 --2- 172.21.15.117:0/527621058 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd3e8038fc0 0x7fd3e803b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec006fd0 tx=0x7fd3ec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:22:56.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:56.916+0000 7fd3fd600700 1 -- 172.21.15.117:0/527621058 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd3f4013070 con 0x7fd400101d40 2024-03-19T03:22:57.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:57.143+0000 7fd407400700 1 -- 172.21.15.117:0/527621058 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7fd40002dcc0 con 0x7fd3e8038fc0 2024-03-19T03:22:57.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:57.151+0000 7fd3fd600700 1 -- 172.21.15.117:0/527621058 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd40002dcc0 con 0x7fd3e8038fc0 2024-03-19T03:22:57.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:57.153+0000 7fd407400700 1 -- 172.21.15.117:0/527621058 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd3e8038fc0 msgr2=0x7fd3e803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:57.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:57.153+0000 7fd407400700 1 --2- 172.21.15.117:0/527621058 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd3e8038fc0 0x7fd3e803b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec006fd0 tx=0x7fd3ec006e40 comp rx=0 tx=0).stop 2024-03-19T03:22:57.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:57.153+0000 7fd407400700 1 -- 172.21.15.117:0/527621058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd400101d40 msgr2=0x7fd400197480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:22:57.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:57.153+0000 7fd407400700 1 --2- 172.21.15.117:0/527621058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd400101d40 0x7fd400197480 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4005850 tx=0x7fd3f4005030 comp rx=0 tx=0).stop 2024-03-19T03:22:57.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:57.154+0000 7fd407400700 1 -- 172.21.15.117:0/527621058 shutdown_connections 2024-03-19T03:22:57.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:57.154+0000 7fd407400700 1 --2- 172.21.15.117:0/527621058 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd3e8038fc0 0x7fd3e803b480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:57.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:57.154+0000 7fd407400700 1 --2- 172.21.15.117:0/527621058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd400101d40 0x7fd400197480 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:22:57.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:57.154+0000 7fd407400700 1 -- 172.21.15.117:0/527621058 >> 172.21.15.117:0/527621058 conn(0x7fd4000fd300 msgr2=0x7fd4000fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:22:57.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:57.154+0000 7fd407400700 1 -- 172.21.15.117:0/527621058 shutdown_connections 2024-03-19T03:22:57.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:22:57.154+0000 7fd407400700 1 -- 172.21.15.117:0/527621058 wait complete. 2024-03-19T03:22:57.896 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi119 2024-03-19T03:22:57.896 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-19T03:22:57.896 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/etc/ceph/ceph.conf 2024-03-19T03:22:57.914 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-19T03:22:57.914 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-19T03:22:57.971 INFO:tasks.cephadm:Adding host smithi119 to orchestrator... 2024-03-19T03:22:57.972 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph orch host add smithi119 2024-03-19T03:22:58.387 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:22:58.531 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:58 smithi117 ceph-mon[31838]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:22:58.531 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:58 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:22:58.531 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:58 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:22:58.531 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:58 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:22:58.531 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:58 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:22:58.531 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:58 smithi117 ceph-mon[31838]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-19T03:22:59.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:59 smithi117 ceph-mon[31838]: Updating smithi117:/var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:22:59.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:22:59 smithi117 ceph-mon[31838]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:00.253 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:00 smithi117 ceph-mon[31838]: Updating smithi117:/etc/ceph/ceph.client.admin.keyring 2024-03-19T03:23:00.253 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:00 smithi117 ceph-mon[31838]: Updating smithi117:/var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T03:23:00.253 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:00 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:00.253 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:00 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:00.253 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:00 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:00.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.416+0000 7fda19800700 1 -- 172.21.15.117:0/2524673877 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda14101f30 msgr2=0x7fda14102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:00.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.416+0000 7fda19800700 1 --2- 172.21.15.117:0/2524673877 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda14101f30 0x7fda14102310 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fda04009910 tx=0x7fda04009c20 comp rx=0 tx=0).stop 2024-03-19T03:23:00.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.416+0000 7fda19800700 1 -- 172.21.15.117:0/2524673877 shutdown_connections 2024-03-19T03:23:00.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.416+0000 7fda19800700 1 --2- 172.21.15.117:0/2524673877 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda14101f30 0x7fda14102310 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:00.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.416+0000 7fda19800700 1 -- 172.21.15.117:0/2524673877 >> 172.21.15.117:0/2524673877 conn(0x7fda140fd7e0 msgr2=0x7fda140ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:23:00.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.416+0000 7fda19800700 1 -- 172.21.15.117:0/2524673877 shutdown_connections 2024-03-19T03:23:00.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.416+0000 7fda19800700 1 -- 172.21.15.117:0/2524673877 wait complete. 2024-03-19T03:23:00.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.417+0000 7fda19800700 1 Processor -- start 2024-03-19T03:23:00.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.417+0000 7fda19800700 1 -- start start 2024-03-19T03:23:00.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.418+0000 7fda19800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda14101f30 0x7fda1419d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:00.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.418+0000 7fda19800700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fda1410c300 con 0x7fda14101f30 2024-03-19T03:23:00.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.418+0000 7fda12a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda14101f30 0x7fda1419d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:00.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.418+0000 7fda12a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda14101f30 0x7fda1419d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:52918/0 (socket says 172.21.15.117:52918) 2024-03-19T03:23:00.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.418+0000 7fda12a00700 1 -- 172.21.15.117:0/4201144057 learned_addr learned my addr 172.21.15.117:0/4201144057 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:23:00.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.418+0000 7fda12a00700 1 -- 172.21.15.117:0/4201144057 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda1419db40 con 0x7fda14101f30 2024-03-19T03:23:00.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.419+0000 7fda12a00700 1 --2- 172.21.15.117:0/4201144057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda14101f30 0x7fda1419d600 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fda04005850 tx=0x7fda04005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:23:00.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.419+0000 7fda0b400700 1 -- 172.21.15.117:0/4201144057 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fda04026070 con 0x7fda14101f30 2024-03-19T03:23:00.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.419+0000 7fda0b400700 1 -- 172.21.15.117:0/4201144057 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda0400bbd0 con 0x7fda14101f30 2024-03-19T03:23:00.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.419+0000 7fda19800700 1 -- 172.21.15.117:0/4201144057 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda1419de30 con 0x7fda14101f30 2024-03-19T03:23:00.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.419+0000 7fda0b400700 1 -- 172.21.15.117:0/4201144057 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fda04003970 con 0x7fda14101f30 2024-03-19T03:23:00.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.419+0000 7fda19800700 1 -- 172.21.15.117:0/4201144057 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda141978a0 con 0x7fda14101f30 2024-03-19T03:23:00.421 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.420+0000 7fda0b400700 1 -- 172.21.15.117:0/4201144057 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7fda04029070 con 0x7fda14101f30 2024-03-19T03:23:00.421 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.420+0000 7fda19800700 1 -- 172.21.15.117:0/4201144057 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda14103410 con 0x7fda14101f30 2024-03-19T03:23:00.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.421+0000 7fda0b400700 1 --2- 172.21.15.117:0/4201144057 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd9f8038fc0 0x7fd9f803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:00.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.421+0000 7fda0b400700 1 -- 172.21.15.117:0/4201144057 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fda0401e070 con 0x7fda14101f30 2024-03-19T03:23:00.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.421+0000 7fda12000700 1 --2- 172.21.15.117:0/4201144057 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd9f8038fc0 0x7fd9f803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:00.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.422+0000 7fda12000700 1 --2- 172.21.15.117:0/4201144057 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd9f8038fc0 0x7fd9f803b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fda00006fd0 tx=0x7fda00006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:23:00.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.427+0000 7fda0b400700 1 -- 172.21.15.117:0/4201144057 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fda04013070 con 0x7fda14101f30 2024-03-19T03:23:00.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:00.651+0000 7fda19800700 1 -- 172.21.15.117:0/4201144057 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi119", "target": ["mon-mgr", ""]}) -- 0x7fda14198c00 con 0x7fd9f8038fc0 2024-03-19T03:23:01.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:01 smithi117 ceph-mon[31838]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:02.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:02 smithi117 ceph-mon[31838]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi119", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:23:03.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:03 smithi117 ceph-mon[31838]: Deploying cephadm binary to smithi119 2024-03-19T03:23:03.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:03 smithi117 ceph-mon[31838]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:05.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:05.268+0000 7fda0b400700 1 -- 172.21.15.117:0/4201144057 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fda14198c00 con 0x7fd9f8038fc0 2024-03-19T03:23:05.269 INFO:teuthology.orchestra.run.smithi117.stdout:Added host 'smithi119' with addr '172.21.15.119' 2024-03-19T03:23:05.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:05.272+0000 7fda19800700 1 -- 172.21.15.117:0/4201144057 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd9f8038fc0 msgr2=0x7fd9f803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:05.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:05.272+0000 7fda19800700 1 --2- 172.21.15.117:0/4201144057 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd9f8038fc0 0x7fd9f803b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fda00006fd0 tx=0x7fda00006e40 comp rx=0 tx=0).stop 2024-03-19T03:23:05.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:05.272+0000 7fda19800700 1 -- 172.21.15.117:0/4201144057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda14101f30 msgr2=0x7fda1419d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:05.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:05.272+0000 7fda19800700 1 --2- 172.21.15.117:0/4201144057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda14101f30 0x7fda1419d600 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fda04005850 tx=0x7fda04005030 comp rx=0 tx=0).stop 2024-03-19T03:23:05.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:05.272+0000 7fda19800700 1 -- 172.21.15.117:0/4201144057 shutdown_connections 2024-03-19T03:23:05.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:05.272+0000 7fda19800700 1 --2- 172.21.15.117:0/4201144057 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd9f8038fc0 0x7fd9f803b480 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:05.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:05.272+0000 7fda19800700 1 --2- 172.21.15.117:0/4201144057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda14101f30 0x7fda1419d600 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:05.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:05.272+0000 7fda19800700 1 -- 172.21.15.117:0/4201144057 >> 172.21.15.117:0/4201144057 conn(0x7fda140fd7e0 msgr2=0x7fda14104d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:23:05.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:05.273+0000 7fda19800700 1 -- 172.21.15.117:0/4201144057 shutdown_connections 2024-03-19T03:23:05.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:05.273+0000 7fda19800700 1 -- 172.21.15.117:0/4201144057 wait complete. 2024-03-19T03:23:05.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:05 smithi117 ceph-mon[31838]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:06.001 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-19T03:23:06.435 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:23:06.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:06 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:06.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:06 smithi117 ceph-mon[31838]: Added host smithi119 2024-03-19T03:23:06.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:06 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:23:06.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:06 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:07.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:07 smithi117 ceph-mon[31838]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:07.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:07 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:08.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.537+0000 7f0a7a000700 1 -- 172.21.15.117:0/4162019628 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a74102eb0 msgr2=0x7f0a74103290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:08.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.537+0000 7f0a7a000700 1 --2- 172.21.15.117:0/4162019628 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a74102eb0 0x7f0a74103290 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0a64009910 tx=0x7f0a64009c20 comp rx=0 tx=0).stop 2024-03-19T03:23:08.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.538+0000 7f0a7a000700 1 -- 172.21.15.117:0/4162019628 shutdown_connections 2024-03-19T03:23:08.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.538+0000 7f0a7a000700 1 --2- 172.21.15.117:0/4162019628 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a74102eb0 0x7f0a74103290 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:08.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.538+0000 7f0a7a000700 1 -- 172.21.15.117:0/4162019628 >> 172.21.15.117:0/4162019628 conn(0x7f0a740fc720 msgr2=0x7f0a740feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:23:08.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.538+0000 7f0a7a000700 1 -- 172.21.15.117:0/4162019628 shutdown_connections 2024-03-19T03:23:08.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.538+0000 7f0a7a000700 1 -- 172.21.15.117:0/4162019628 wait complete. 2024-03-19T03:23:08.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.539+0000 7f0a7a000700 1 Processor -- start 2024-03-19T03:23:08.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.539+0000 7f0a7a000700 1 -- start start 2024-03-19T03:23:08.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.540+0000 7f0a7a000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a74102eb0 0x7f0a7419d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:08.540 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.540+0000 7f0a7a000700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7410c380 con 0x7f0a74102eb0 2024-03-19T03:23:08.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.540+0000 7f0a73400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a74102eb0 0x7f0a7419d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:08.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.540+0000 7f0a73400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a74102eb0 0x7f0a7419d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:55640/0 (socket says 172.21.15.117:55640) 2024-03-19T03:23:08.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.540+0000 7f0a73400700 1 -- 172.21.15.117:0/4144060335 learned_addr learned my addr 172.21.15.117:0/4144060335 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:23:08.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.540+0000 7f0a73400700 1 -- 172.21.15.117:0/4144060335 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a7419db50 con 0x7f0a74102eb0 2024-03-19T03:23:08.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.541+0000 7f0a73400700 1 --2- 172.21.15.117:0/4144060335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a74102eb0 0x7f0a7419d610 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0a64006010 tx=0x7f0a64005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:23:08.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.541+0000 7f0a6be00700 1 -- 172.21.15.117:0/4144060335 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a64026070 con 0x7f0a74102eb0 2024-03-19T03:23:08.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.541+0000 7f0a6be00700 1 -- 172.21.15.117:0/4144060335 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a6400bc30 con 0x7f0a74102eb0 2024-03-19T03:23:08.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.541+0000 7f0a7a000700 1 -- 172.21.15.117:0/4144060335 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a7419dde0 con 0x7f0a74102eb0 2024-03-19T03:23:08.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.541+0000 7f0a6be00700 1 -- 172.21.15.117:0/4144060335 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a640039a0 con 0x7f0a74102eb0 2024-03-19T03:23:08.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.541+0000 7f0a7a000700 1 -- 172.21.15.117:0/4144060335 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a741976e0 con 0x7f0a74102eb0 2024-03-19T03:23:08.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.542+0000 7f0a6be00700 1 -- 172.21.15.117:0/4144060335 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f0a64029070 con 0x7f0a74102eb0 2024-03-19T03:23:08.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.542+0000 7f0a7a000700 1 -- 172.21.15.117:0/4144060335 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a7406a9e0 con 0x7f0a74102eb0 2024-03-19T03:23:08.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.543+0000 7f0a6be00700 1 --2- 172.21.15.117:0/4144060335 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0a58039020 0x7f0a5803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:08.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.543+0000 7f0a6be00700 1 -- 172.21.15.117:0/4144060335 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0a6401e070 con 0x7f0a74102eb0 2024-03-19T03:23:08.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.543+0000 7f0a72a00700 1 --2- 172.21.15.117:0/4144060335 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0a58039020 0x7f0a5803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:08.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.544+0000 7f0a72a00700 1 --2- 172.21.15.117:0/4144060335 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0a58039020 0x7f0a5803b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0a60006fd0 tx=0x7f0a60006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:23:08.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.549+0000 7f0a6be00700 1 -- 172.21.15.117:0/4144060335 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0a64013070 con 0x7f0a74102eb0 2024-03-19T03:23:08.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.773+0000 7f0a7a000700 1 -- 172.21.15.117:0/4144060335 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0a74198c10 con 0x7f0a58039020 2024-03-19T03:23:08.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.776+0000 7f0a6be00700 1 -- 172.21.15.117:0/4144060335 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+161 (secure 0 0 0) 0x7f0a74198c10 con 0x7f0a58039020 2024-03-19T03:23:08.777 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:23:08.777 INFO:teuthology.orchestra.run.smithi117.stdout:[{"addr": "172.21.15.117", "hostname": "smithi117", "labels": [], "status": ""}, {"addr": "172.21.15.119", "hostname": "smithi119", "labels": [], "status": ""}] 2024-03-19T03:23:08.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.779+0000 7f0a7a000700 1 -- 172.21.15.117:0/4144060335 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0a58039020 msgr2=0x7f0a5803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:08.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.779+0000 7f0a7a000700 1 --2- 172.21.15.117:0/4144060335 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0a58039020 0x7f0a5803b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0a60006fd0 tx=0x7f0a60006e40 comp rx=0 tx=0).stop 2024-03-19T03:23:08.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.779+0000 7f0a7a000700 1 -- 172.21.15.117:0/4144060335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a74102eb0 msgr2=0x7f0a7419d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:08.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.779+0000 7f0a7a000700 1 --2- 172.21.15.117:0/4144060335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a74102eb0 0x7f0a7419d610 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0a64006010 tx=0x7f0a64005030 comp rx=0 tx=0).stop 2024-03-19T03:23:08.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.779+0000 7f0a7a000700 1 -- 172.21.15.117:0/4144060335 shutdown_connections 2024-03-19T03:23:08.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.779+0000 7f0a7a000700 1 --2- 172.21.15.117:0/4144060335 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0a58039020 0x7f0a5803b4e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:08.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.779+0000 7f0a7a000700 1 --2- 172.21.15.117:0/4144060335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a74102eb0 0x7f0a7419d610 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:08.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.779+0000 7f0a7a000700 1 -- 172.21.15.117:0/4144060335 >> 172.21.15.117:0/4144060335 conn(0x7f0a740fc720 msgr2=0x7f0a74103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:23:08.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.779+0000 7f0a7a000700 1 -- 172.21.15.117:0/4144060335 shutdown_connections 2024-03-19T03:23:08.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:08.779+0000 7f0a7a000700 1 -- 172.21.15.117:0/4144060335 wait complete. 2024-03-19T03:23:09.130 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:09.131 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:08 smithi117 ceph-mon[31838]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:09.459 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi178 2024-03-19T03:23:09.459 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-19T03:23:09.459 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/etc/ceph/ceph.conf 2024-03-19T03:23:09.478 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-19T03:23:09.478 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-19T03:23:09.535 INFO:tasks.cephadm:Adding host smithi178 to orchestrator... 2024-03-19T03:23:09.535 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph orch host add smithi178 2024-03-19T03:23:09.963 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:23:10.291 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:09 smithi117 ceph-mon[31838]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T03:23:11.139 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:10 smithi117 ceph-mon[31838]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:11.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.938+0000 7fd980c00700 1 -- 172.21.15.117:0/354846311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd97c102f90 msgr2=0x7fd97c103370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:11.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.938+0000 7fd980c00700 1 --2- 172.21.15.117:0/354846311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd97c102f90 0x7fd97c103370 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd96c009910 tx=0x7fd96c009c20 comp rx=0 tx=0).stop 2024-03-19T03:23:11.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.939+0000 7fd980c00700 1 -- 172.21.15.117:0/354846311 shutdown_connections 2024-03-19T03:23:11.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.939+0000 7fd980c00700 1 --2- 172.21.15.117:0/354846311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd97c102f90 0x7fd97c103370 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:11.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.939+0000 7fd980c00700 1 -- 172.21.15.117:0/354846311 >> 172.21.15.117:0/354846311 conn(0x7fd97c0779b0 msgr2=0x7fd97c0fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:23:11.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.939+0000 7fd980c00700 1 -- 172.21.15.117:0/354846311 shutdown_connections 2024-03-19T03:23:11.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.939+0000 7fd980c00700 1 -- 172.21.15.117:0/354846311 wait complete. 2024-03-19T03:23:11.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.941+0000 7fd980c00700 1 Processor -- start 2024-03-19T03:23:11.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.941+0000 7fd980c00700 1 -- start start 2024-03-19T03:23:11.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.942+0000 7fd980c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd97c102f90 0x7fd97c19d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:11.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.942+0000 7fd980c00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd97c10c370 con 0x7fd97c102f90 2024-03-19T03:23:11.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.942+0000 7fd979e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd97c102f90 0x7fd97c19d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:11.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.942+0000 7fd979e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd97c102f90 0x7fd97c19d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:55650/0 (socket says 172.21.15.117:55650) 2024-03-19T03:23:11.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.942+0000 7fd979e00700 1 -- 172.21.15.117:0/853127921 learned_addr learned my addr 172.21.15.117:0/853127921 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:23:11.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.942+0000 7fd979e00700 1 -- 172.21.15.117:0/853127921 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd97c19db20 con 0x7fd97c102f90 2024-03-19T03:23:11.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.943+0000 7fd979e00700 1 --2- 172.21.15.117:0/853127921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd97c102f90 0x7fd97c19d5e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd96c006010 tx=0x7fd96c005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:23:11.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.943+0000 7fd972a00700 1 -- 172.21.15.117:0/853127921 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd96c026070 con 0x7fd97c102f90 2024-03-19T03:23:11.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.943+0000 7fd972a00700 1 -- 172.21.15.117:0/853127921 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd96c00bc30 con 0x7fd97c102f90 2024-03-19T03:23:11.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.943+0000 7fd980c00700 1 -- 172.21.15.117:0/853127921 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd97c19ddb0 con 0x7fd97c102f90 2024-03-19T03:23:11.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.943+0000 7fd972a00700 1 -- 172.21.15.117:0/853127921 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd96c003ab0 con 0x7fd97c102f90 2024-03-19T03:23:11.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.943+0000 7fd980c00700 1 -- 172.21.15.117:0/853127921 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd97c1976b0 con 0x7fd97c102f90 2024-03-19T03:23:11.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.944+0000 7fd972a00700 1 -- 172.21.15.117:0/853127921 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7fd96c029070 con 0x7fd97c102f90 2024-03-19T03:23:11.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.944+0000 7fd980c00700 1 -- 172.21.15.117:0/853127921 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd97c069ee0 con 0x7fd97c102f90 2024-03-19T03:23:11.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.945+0000 7fd972a00700 1 --2- 172.21.15.117:0/853127921 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd960039020 0x7fd96003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:11.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.945+0000 7fd972a00700 1 -- 172.21.15.117:0/853127921 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd96c01e070 con 0x7fd97c102f90 2024-03-19T03:23:11.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.945+0000 7fd979400700 1 --2- 172.21.15.117:0/853127921 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd960039020 0x7fd96003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:11.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.946+0000 7fd979400700 1 --2- 172.21.15.117:0/853127921 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd960039020 0x7fd96003b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd968006fd0 tx=0x7fd968006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:23:11.951 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:11.950+0000 7fd972a00700 1 -- 172.21.15.117:0/853127921 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd96c013070 con 0x7fd97c102f90 2024-03-19T03:23:12.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:12.207+0000 7fd980c00700 1 -- 172.21.15.117:0/853127921 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi178", "target": ["mon-mgr", ""]}) -- 0x7fd97c198be0 con 0x7fd960039020 2024-03-19T03:23:12.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:12 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:12.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:12 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:12.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:12 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:12.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:12 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:12.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:12 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi119", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:23:12.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:12 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:12.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:12 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:23:12.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:12 smithi117 ceph-mon[31838]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-19T03:23:13.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:13 smithi117 ceph-mon[31838]: Updating smithi119:/var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:23:13.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:13 smithi117 ceph-mon[31838]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-03-19T03:23:13.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:13 smithi117 ceph-mon[31838]: Updating smithi119:/var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T03:23:13.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:13 smithi117 ceph-mon[31838]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:13.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:13 smithi117 ceph-mon[31838]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi178", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:23:13.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:13 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:13.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:13 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:13.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:13 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:14.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:14 smithi117 ceph-mon[31838]: Deploying cephadm binary to smithi178 2024-03-19T03:23:15.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:15 smithi117 ceph-mon[31838]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:16.836 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:16.836+0000 7fd972a00700 1 -- 172.21.15.117:0/853127921 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fd97c198be0 con 0x7fd960039020 2024-03-19T03:23:16.837 INFO:teuthology.orchestra.run.smithi117.stdout:Added host 'smithi178' with addr '172.21.15.178' 2024-03-19T03:23:16.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:16.840+0000 7fd980c00700 1 -- 172.21.15.117:0/853127921 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd960039020 msgr2=0x7fd96003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:16.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:16.840+0000 7fd980c00700 1 --2- 172.21.15.117:0/853127921 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd960039020 0x7fd96003b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd968006fd0 tx=0x7fd968006e40 comp rx=0 tx=0).stop 2024-03-19T03:23:16.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:16.840+0000 7fd980c00700 1 -- 172.21.15.117:0/853127921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd97c102f90 msgr2=0x7fd97c19d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:16.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:16.840+0000 7fd980c00700 1 --2- 172.21.15.117:0/853127921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd97c102f90 0x7fd97c19d5e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd96c006010 tx=0x7fd96c005050 comp rx=0 tx=0).stop 2024-03-19T03:23:16.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:16.840+0000 7fd980c00700 1 -- 172.21.15.117:0/853127921 shutdown_connections 2024-03-19T03:23:16.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:16.840+0000 7fd980c00700 1 --2- 172.21.15.117:0/853127921 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd960039020 0x7fd96003b4e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:16.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:16.840+0000 7fd980c00700 1 --2- 172.21.15.117:0/853127921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd97c102f90 0x7fd97c19d5e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:16.841 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:16.840+0000 7fd980c00700 1 -- 172.21.15.117:0/853127921 >> 172.21.15.117:0/853127921 conn(0x7fd97c0779b0 msgr2=0x7fd97c103d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:23:16.841 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:16.840+0000 7fd980c00700 1 -- 172.21.15.117:0/853127921 shutdown_connections 2024-03-19T03:23:16.841 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:16.840+0000 7fd980c00700 1 -- 172.21.15.117:0/853127921 wait complete. 2024-03-19T03:23:17.529 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:17 smithi117 ceph-mon[31838]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:17.529 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:17 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:17.529 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:17 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:23:17.584 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-19T03:23:18.014 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:23:18.410 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:18 smithi117 ceph-mon[31838]: Added host smithi178 2024-03-19T03:23:18.410 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:18 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:18.410 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:18 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:19.456 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:19 smithi117 ceph-mon[31838]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:19.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.904+0000 7fea9e000700 1 -- 172.21.15.117:0/3745357463 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea98108570 msgr2=0x7fea98108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:19.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.904+0000 7fea9e000700 1 --2- 172.21.15.117:0/3745357463 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea98108570 0x7fea98108950 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fea88009910 tx=0x7fea88009c20 comp rx=0 tx=0).stop 2024-03-19T03:23:19.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.904+0000 7fea9e000700 1 -- 172.21.15.117:0/3745357463 shutdown_connections 2024-03-19T03:23:19.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.904+0000 7fea9e000700 1 --2- 172.21.15.117:0/3745357463 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea98108570 0x7fea98108950 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:19.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.904+0000 7fea9e000700 1 -- 172.21.15.117:0/3745357463 >> 172.21.15.117:0/3745357463 conn(0x7fea98077b40 msgr2=0x7fea98079f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:23:19.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.904+0000 7fea9e000700 1 -- 172.21.15.117:0/3745357463 shutdown_connections 2024-03-19T03:23:19.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.904+0000 7fea9e000700 1 -- 172.21.15.117:0/3745357463 wait complete. 2024-03-19T03:23:19.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.905+0000 7fea9e000700 1 Processor -- start 2024-03-19T03:23:19.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.905+0000 7fea9e000700 1 -- start start 2024-03-19T03:23:19.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.906+0000 7fea9e000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea98108570 0x7fea9819de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:19.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.906+0000 7fea9e000700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fea98112930 con 0x7fea98108570 2024-03-19T03:23:19.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.906+0000 7fea97400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea98108570 0x7fea9819de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:19.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.906+0000 7fea97400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea98108570 0x7fea9819de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:53266/0 (socket says 172.21.15.117:53266) 2024-03-19T03:23:19.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.906+0000 7fea97400700 1 -- 172.21.15.117:0/3217839704 learned_addr learned my addr 172.21.15.117:0/3217839704 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:23:19.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.906+0000 7fea97400700 1 -- 172.21.15.117:0/3217839704 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea9819e370 con 0x7fea98108570 2024-03-19T03:23:19.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.907+0000 7fea97400700 1 --2- 172.21.15.117:0/3217839704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea98108570 0x7fea9819de30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fea88006010 tx=0x7fea88005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:23:19.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.907+0000 7fea8fe00700 1 -- 172.21.15.117:0/3217839704 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea88026070 con 0x7fea98108570 2024-03-19T03:23:19.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.907+0000 7fea9e000700 1 -- 172.21.15.117:0/3217839704 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea9819e660 con 0x7fea98108570 2024-03-19T03:23:19.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.907+0000 7fea8fe00700 1 -- 172.21.15.117:0/3217839704 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea8800bbd0 con 0x7fea98108570 2024-03-19T03:23:19.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.907+0000 7fea8fe00700 1 -- 172.21.15.117:0/3217839704 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea88003910 con 0x7fea98108570 2024-03-19T03:23:19.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.907+0000 7fea9e000700 1 -- 172.21.15.117:0/3217839704 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea981a2700 con 0x7fea98108570 2024-03-19T03:23:19.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.908+0000 7fea8fe00700 1 -- 172.21.15.117:0/3217839704 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7fea88029070 con 0x7fea98108570 2024-03-19T03:23:19.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.908+0000 7fea9e000700 1 -- 172.21.15.117:0/3217839704 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea98109a40 con 0x7fea98108570 2024-03-19T03:23:19.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.909+0000 7fea8fe00700 1 --2- 172.21.15.117:0/3217839704 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fea7c038fc0 0x7fea7c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:19.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.909+0000 7fea8fe00700 1 -- 172.21.15.117:0/3217839704 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fea8801e070 con 0x7fea98108570 2024-03-19T03:23:19.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.909+0000 7fea96a00700 1 --2- 172.21.15.117:0/3217839704 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fea7c038fc0 0x7fea7c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:19.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.910+0000 7fea96a00700 1 --2- 172.21.15.117:0/3217839704 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fea7c038fc0 0x7fea7c03b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fea84006fd0 tx=0x7fea84006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:23:19.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:19.914+0000 7fea8fe00700 1 -- 172.21.15.117:0/3217839704 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fea88013070 con 0x7fea98108570 2024-03-19T03:23:20.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:20.142+0000 7fea9e000700 1 -- 172.21.15.117:0/3217839704 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fea9819f190 con 0x7fea7c038fc0 2024-03-19T03:23:20.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:20.145+0000 7fea8fe00700 1 -- 172.21.15.117:0/3217839704 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+241 (secure 0 0 0) 0x7fea9819f190 con 0x7fea7c038fc0 2024-03-19T03:23:20.145 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:23:20.145 INFO:teuthology.orchestra.run.smithi117.stdout:[{"addr": "172.21.15.117", "hostname": "smithi117", "labels": [], "status": ""}, {"addr": "172.21.15.119", "hostname": "smithi119", "labels": [], "status": ""}, {"addr": "172.21.15.178", "hostname": "smithi178", "labels": [], "status": ""}] 2024-03-19T03:23:20.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:20.147+0000 7fea9e000700 1 -- 172.21.15.117:0/3217839704 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fea7c038fc0 msgr2=0x7fea7c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:20.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:20.147+0000 7fea9e000700 1 --2- 172.21.15.117:0/3217839704 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fea7c038fc0 0x7fea7c03b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fea84006fd0 tx=0x7fea84006e40 comp rx=0 tx=0).stop 2024-03-19T03:23:20.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:20.148+0000 7fea9e000700 1 -- 172.21.15.117:0/3217839704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea98108570 msgr2=0x7fea9819de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:20.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:20.148+0000 7fea9e000700 1 --2- 172.21.15.117:0/3217839704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea98108570 0x7fea9819de30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fea88006010 tx=0x7fea88005030 comp rx=0 tx=0).stop 2024-03-19T03:23:20.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:20.148+0000 7fea9e000700 1 -- 172.21.15.117:0/3217839704 shutdown_connections 2024-03-19T03:23:20.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:20.148+0000 7fea9e000700 1 --2- 172.21.15.117:0/3217839704 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fea7c038fc0 0x7fea7c03b480 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:20.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:20.148+0000 7fea9e000700 1 --2- 172.21.15.117:0/3217839704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea98108570 0x7fea9819de30 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:20.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:20.148+0000 7fea9e000700 1 -- 172.21.15.117:0/3217839704 >> 172.21.15.117:0/3217839704 conn(0x7fea98077b40 msgr2=0x7fea98109f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:23:20.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:20.148+0000 7fea9e000700 1 -- 172.21.15.117:0/3217839704 shutdown_connections 2024-03-19T03:23:20.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:20.148+0000 7fea9e000700 1 -- 172.21.15.117:0/3217839704 wait complete. 2024-03-19T03:23:20.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:20 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:20.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:20 smithi117 ceph-mon[31838]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T03:23:20.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:20 smithi117 ceph-mon[31838]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:20.904 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-19T03:23:20.904 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-19T03:23:21.331 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:23:23.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.275+0000 7f49c4400700 1 -- 172.21.15.117:0/3608633649 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49bc101f20 msgr2=0x7f49bc102300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:23.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.275+0000 7f49c4400700 1 --2- 172.21.15.117:0/3608633649 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49bc101f20 0x7f49bc102300 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f49b0009910 tx=0x7f49b0009c20 comp rx=0 tx=0).stop 2024-03-19T03:23:23.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.275+0000 7f49c4400700 1 -- 172.21.15.117:0/3608633649 shutdown_connections 2024-03-19T03:23:23.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.275+0000 7f49c4400700 1 --2- 172.21.15.117:0/3608633649 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49bc101f20 0x7f49bc102300 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:23.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.275+0000 7f49c4400700 1 -- 172.21.15.117:0/3608633649 >> 172.21.15.117:0/3608633649 conn(0x7f49bc0fd7f0 msgr2=0x7f49bc0ffc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:23:23.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.275+0000 7f49c4400700 1 -- 172.21.15.117:0/3608633649 shutdown_connections 2024-03-19T03:23:23.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.275+0000 7f49c4400700 1 -- 172.21.15.117:0/3608633649 wait complete. 2024-03-19T03:23:23.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.276+0000 7f49c4400700 1 Processor -- start 2024-03-19T03:23:23.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.276+0000 7f49c4400700 1 -- start start 2024-03-19T03:23:23.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.277+0000 7f49c4400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49bc101f20 0x7f49bc19d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:23.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.277+0000 7f49c4400700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f49bc10c2f0 con 0x7f49bc101f20 2024-03-19T03:23:23.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.277+0000 7f49c1a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49bc101f20 0x7f49bc19d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:23.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.277+0000 7f49c1a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49bc101f20 0x7f49bc19d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:53286/0 (socket says 172.21.15.117:53286) 2024-03-19T03:23:23.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.277+0000 7f49c1a00700 1 -- 172.21.15.117:0/3655883920 learned_addr learned my addr 172.21.15.117:0/3655883920 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:23:23.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.277+0000 7f49c1a00700 1 -- 172.21.15.117:0/3655883920 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49bc19daa0 con 0x7f49bc101f20 2024-03-19T03:23:23.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.278+0000 7f49c1a00700 1 --2- 172.21.15.117:0/3655883920 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49bc101f20 0x7f49bc19d560 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f49b0006010 tx=0x7f49b0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:23:23.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.278+0000 7f49ba000700 1 -- 172.21.15.117:0/3655883920 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49b0026070 con 0x7f49bc101f20 2024-03-19T03:23:23.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.278+0000 7f49ba000700 1 -- 172.21.15.117:0/3655883920 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49b000bbd0 con 0x7f49bc101f20 2024-03-19T03:23:23.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.278+0000 7f49c4400700 1 -- 172.21.15.117:0/3655883920 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49bc19dd90 con 0x7f49bc101f20 2024-03-19T03:23:23.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.278+0000 7f49ba000700 1 -- 172.21.15.117:0/3655883920 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49b0003910 con 0x7f49bc101f20 2024-03-19T03:23:23.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.278+0000 7f49c4400700 1 -- 172.21.15.117:0/3655883920 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49bc197800 con 0x7f49bc101f20 2024-03-19T03:23:23.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.279+0000 7f49af400700 1 -- 172.21.15.117:0/3655883920 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49bc103400 con 0x7f49bc101f20 2024-03-19T03:23:23.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.280+0000 7f49ba000700 1 -- 172.21.15.117:0/3655883920 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f49b0029070 con 0x7f49bc101f20 2024-03-19T03:23:23.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.280+0000 7f49ba000700 1 --2- 172.21.15.117:0/3655883920 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f49a4039020 0x7f49a403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:23.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.281+0000 7f49ba000700 1 -- 172.21.15.117:0/3655883920 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f49b001e070 con 0x7f49bc101f20 2024-03-19T03:23:23.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.281+0000 7f49c1000700 1 --2- 172.21.15.117:0/3655883920 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f49a4039020 0x7f49a403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:23.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.282+0000 7f49c1000700 1 --2- 172.21.15.117:0/3655883920 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f49a4039020 0x7f49a403b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f49a8006fd0 tx=0x7f49a8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:23:23.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.287+0000 7f49ba000700 1 -- 172.21.15.117:0/3655883920 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f49b0013070 con 0x7f49bc101f20 2024-03-19T03:23:23.519 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:23.519+0000 7f49af400700 1 -- 172.21.15.117:0/3655883920 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f49bc063940 con 0x7f49bc101f20 2024-03-19T03:23:23.519 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:23 smithi117 ceph-mon[31838]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:23.520 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:23 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:23.520 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:23 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:23.520 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:23 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:23.520 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:23 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:23.520 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:23 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi178", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:23:23.520 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:23 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:23.520 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:23 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:23:24.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:24.220+0000 7f49ba000700 1 -- 172.21.15.117:0/3655883920 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f49b000fe20 con 0x7f49bc101f20 2024-03-19T03:23:24.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:24.223+0000 7f49af400700 1 -- 172.21.15.117:0/3655883920 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f49a4039020 msgr2=0x7f49a403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:24.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:24.223+0000 7f49af400700 1 --2- 172.21.15.117:0/3655883920 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f49a4039020 0x7f49a403b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f49a8006fd0 tx=0x7f49a8006e40 comp rx=0 tx=0).stop 2024-03-19T03:23:24.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:24.224+0000 7f49af400700 1 -- 172.21.15.117:0/3655883920 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49bc101f20 msgr2=0x7f49bc19d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:24.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:24.224+0000 7f49af400700 1 --2- 172.21.15.117:0/3655883920 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49bc101f20 0x7f49bc19d560 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f49b0006010 tx=0x7f49b0005030 comp rx=0 tx=0).stop 2024-03-19T03:23:24.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:24.224+0000 7f49af400700 1 -- 172.21.15.117:0/3655883920 shutdown_connections 2024-03-19T03:23:24.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:24.224+0000 7f49af400700 1 --2- 172.21.15.117:0/3655883920 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f49a4039020 0x7f49a403b4e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:24.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:24.224+0000 7f49af400700 1 --2- 172.21.15.117:0/3655883920 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49bc101f20 0x7f49bc19d560 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:24.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:24.224+0000 7f49af400700 1 -- 172.21.15.117:0/3655883920 >> 172.21.15.117:0/3655883920 conn(0x7f49bc0fd7f0 msgr2=0x7f49bc1038f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:23:24.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:24.224+0000 7f49af400700 1 -- 172.21.15.117:0/3655883920 shutdown_connections 2024-03-19T03:23:24.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:24.224+0000 7f49af400700 1 -- 172.21.15.117:0/3655883920 wait complete. 2024-03-19T03:23:24.225 INFO:teuthology.orchestra.run.smithi117.stderr:adjusted tunables profile to default 2024-03-19T03:23:24.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:24 smithi117 ceph-mon[31838]: Updating smithi178:/etc/ceph/ceph.conf 2024-03-19T03:23:24.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:24 smithi117 ceph-mon[31838]: Updating smithi178:/var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:23:24.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:24 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3655883920' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-19T03:23:24.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:24 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:24.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:24 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:24.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:24 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:24.953 INFO:tasks.cephadm:Adding mon.a on smithi117 2024-03-19T03:23:24.953 INFO:tasks.cephadm:Adding mon.b on smithi119 2024-03-19T03:23:24.953 INFO:tasks.cephadm:Adding mon.c on smithi178 2024-03-19T03:23:24.954 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph orch apply mon '3;smithi117:172.21.15.117=a;smithi119:172.21.15.119=b;smithi178:172.21.15.178=c' 2024-03-19T03:23:25.190 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:23:25.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:25 smithi117 ceph-mon[31838]: Updating smithi178:/etc/ceph/ceph.client.admin.keyring 2024-03-19T03:23:25.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:25 smithi117 ceph-mon[31838]: Updating smithi178:/var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T03:23:25.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:25 smithi117 ceph-mon[31838]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:25.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:25 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3655883920' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-19T03:23:25.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:25 smithi117 ceph-mon[31838]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T03:23:26.612 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.611+0000 7f8dd4c99700 1 -- 172.21.15.178:0/3118457229 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8dd0102e90 msgr2=0x7f8dd0103270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:26.612 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.611+0000 7f8dd4c99700 1 --2- 172.21.15.178:0/3118457229 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8dd0102e90 0x7f8dd0103270 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0009910 tx=0x7f8dc0009c20 comp rx=0 tx=0).stop 2024-03-19T03:23:26.613 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.611+0000 7f8dd4c99700 1 -- 172.21.15.178:0/3118457229 shutdown_connections 2024-03-19T03:23:26.613 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.611+0000 7f8dd4c99700 1 --2- 172.21.15.178:0/3118457229 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8dd0102e90 0x7f8dd0103270 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:26.613 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.611+0000 7f8dd4c99700 1 -- 172.21.15.178:0/3118457229 >> 172.21.15.178:0/3118457229 conn(0x7f8dd00fc6c0 msgr2=0x7f8dd00feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:23:26.613 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.611+0000 7f8dd4c99700 1 -- 172.21.15.178:0/3118457229 shutdown_connections 2024-03-19T03:23:26.613 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.611+0000 7f8dd4c99700 1 -- 172.21.15.178:0/3118457229 wait complete. 2024-03-19T03:23:26.613 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.612+0000 7f8dd4c99700 1 Processor -- start 2024-03-19T03:23:26.614 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.612+0000 7f8dd4c99700 1 -- start start 2024-03-19T03:23:26.614 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.613+0000 7f8dd4c99700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8dd0102e90 0x7f8dd019d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:26.614 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.613+0000 7f8dd4c99700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd010c360 con 0x7f8dd0102e90 2024-03-19T03:23:26.614 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.613+0000 7f8dce59c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8dd0102e90 0x7f8dd019d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:26.614 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.613+0000 7f8dce59c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8dd0102e90 0x7f8dd019d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.178:47976/0 (socket says 172.21.15.178:47976) 2024-03-19T03:23:26.614 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.613+0000 7f8dce59c700 1 -- 172.21.15.178:0/3901556634 learned_addr learned my addr 172.21.15.178:0/3901556634 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-19T03:23:26.615 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.614+0000 7f8dce59c700 1 -- 172.21.15.178:0/3901556634 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dd019db10 con 0x7f8dd0102e90 2024-03-19T03:23:26.615 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.614+0000 7f8dce59c700 1 --2- 172.21.15.178:0/3901556634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8dd0102e90 0x7f8dd019d5d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0006010 tx=0x7f8dc0005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:23:26.615 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.615+0000 7f8dc7fff700 1 -- 172.21.15.178:0/3901556634 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8dc0026070 con 0x7f8dd0102e90 2024-03-19T03:23:26.615 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.615+0000 7f8dd4c99700 1 -- 172.21.15.178:0/3901556634 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dd019dda0 con 0x7f8dd0102e90 2024-03-19T03:23:26.615 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.615+0000 7f8dd4c99700 1 -- 172.21.15.178:0/3901556634 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dd01976a0 con 0x7f8dd0102e90 2024-03-19T03:23:26.615 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.615+0000 7f8dc7fff700 1 -- 172.21.15.178:0/3901556634 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8dc000bc30 con 0x7f8dd0102e90 2024-03-19T03:23:26.615 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.615+0000 7f8dc7fff700 1 -- 172.21.15.178:0/3901556634 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8dc0030060 con 0x7f8dd0102e90 2024-03-19T03:23:26.616 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.616+0000 7f8dd4c99700 1 -- 172.21.15.178:0/3901556634 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dd006aa20 con 0x7f8dd0102e90 2024-03-19T03:23:26.616 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.616+0000 7f8dc7fff700 1 -- 172.21.15.178:0/3901556634 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f8dc0029070 con 0x7f8dd0102e90 2024-03-19T03:23:26.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.616+0000 7f8dc7fff700 1 --2- 172.21.15.178:0/3901556634 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8dbc038fc0 0x7f8dbc03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:26.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.616+0000 7f8dc7fff700 1 -- 172.21.15.178:0/3901556634 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8dc001e070 con 0x7f8dd0102e90 2024-03-19T03:23:26.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.617+0000 7f8dc5d9b700 1 --2- 172.21.15.178:0/3901556634 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8dbc038fc0 0x7f8dbc03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:26.618 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.618+0000 7f8dc5d9b700 1 --2- 172.21.15.178:0/3901556634 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8dbc038fc0 0x7f8dbc03b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8db8006fd0 tx=0x7f8db8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:23:26.632 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.632+0000 7f8dc7fff700 1 -- 172.21.15.178:0/3901556634 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8dc0013070 con 0x7f8dd0102e90 2024-03-19T03:23:26.834 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.834+0000 7f8dd4c99700 1 -- 172.21.15.178:0/3901556634 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi117:172.21.15.117=a;smithi119:172.21.15.119=b;smithi178:172.21.15.178=c", "target": ["mon-mgr", ""]}) -- 0x7f8dd0198bd0 con 0x7f8dbc038fc0 2024-03-19T03:23:26.849 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.849+0000 7f8dc7fff700 1 -- 172.21.15.178:0/3901556634 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f8dd0198bd0 con 0x7f8dbc038fc0 2024-03-19T03:23:26.849 INFO:teuthology.orchestra.run.smithi178.stdout:Scheduled mon update... 2024-03-19T03:23:26.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.851+0000 7f8dd4c99700 1 -- 172.21.15.178:0/3901556634 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8dbc038fc0 msgr2=0x7f8dbc03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:26.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.851+0000 7f8dd4c99700 1 --2- 172.21.15.178:0/3901556634 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8dbc038fc0 0x7f8dbc03b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8db8006fd0 tx=0x7f8db8006e40 comp rx=0 tx=0).stop 2024-03-19T03:23:26.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.851+0000 7f8dd4c99700 1 -- 172.21.15.178:0/3901556634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8dd0102e90 msgr2=0x7f8dd019d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:26.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.851+0000 7f8dd4c99700 1 --2- 172.21.15.178:0/3901556634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8dd0102e90 0x7f8dd019d5d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0006010 tx=0x7f8dc0005050 comp rx=0 tx=0).stop 2024-03-19T03:23:26.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.851+0000 7f8dd4c99700 1 -- 172.21.15.178:0/3901556634 shutdown_connections 2024-03-19T03:23:26.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.851+0000 7f8dd4c99700 1 --2- 172.21.15.178:0/3901556634 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8dbc038fc0 0x7f8dbc03b480 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:26.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.851+0000 7f8dd4c99700 1 --2- 172.21.15.178:0/3901556634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8dd0102e90 0x7f8dd019d5d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:26.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.851+0000 7f8dd4c99700 1 -- 172.21.15.178:0/3901556634 >> 172.21.15.178:0/3901556634 conn(0x7f8dd00fc6c0 msgr2=0x7f8dd0103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:23:26.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.851+0000 7f8dd4c99700 1 -- 172.21.15.178:0/3901556634 shutdown_connections 2024-03-19T03:23:26.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:26.851+0000 7f8dd4c99700 1 -- 172.21.15.178:0/3901556634 wait complete. 2024-03-19T03:23:27.401 DEBUG:teuthology.orchestra.run.smithi119:mon.b> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mon.b.service 2024-03-19T03:23:27.403 DEBUG:teuthology.orchestra.run.smithi178:mon.c> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mon.c.service 2024-03-19T03:23:27.405 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-19T03:23:27.406 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T03:23:27.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:27 smithi117 ceph-mon[31838]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:27.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:27 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:27.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:27 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:23:27.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:27 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:27.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:27 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:23:27.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:27 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:27.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:27 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T03:23:27.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:27 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:28.235 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:23:28.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:28 smithi117 ceph-mon[31838]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi117:172.21.15.117=a;smithi119:172.21.15.119=b;smithi178:172.21.15.178=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:23:28.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:28 smithi117 ceph-mon[31838]: Saving service mon spec with placement smithi117:172.21.15.117=a;smithi119:172.21.15.119=b;smithi178:172.21.15.178=c;count:3 2024-03-19T03:23:28.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:28 smithi117 ceph-mon[31838]: Deploying daemon mon.c on smithi178 2024-03-19T03:23:29.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:29 smithi117 ceph-mon[31838]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:30.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.848+0000 7fe922de7700 1 -- 172.21.15.178:0/3612084965 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe91c100e70 msgr2=0x7fe91c101250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:30.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.848+0000 7fe922de7700 1 --2- 172.21.15.178:0/3612084965 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe91c100e70 0x7fe91c101250 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe910009910 tx=0x7fe910009c20 comp rx=0 tx=0).stop 2024-03-19T03:23:30.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.849+0000 7fe922de7700 1 -- 172.21.15.178:0/3612084965 shutdown_connections 2024-03-19T03:23:30.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.849+0000 7fe922de7700 1 --2- 172.21.15.178:0/3612084965 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe91c100e70 0x7fe91c101250 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:30.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.849+0000 7fe922de7700 1 -- 172.21.15.178:0/3612084965 >> 172.21.15.178:0/3612084965 conn(0x7fe91c0779b0 msgr2=0x7fe91c0fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:23:30.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.849+0000 7fe922de7700 1 -- 172.21.15.178:0/3612084965 shutdown_connections 2024-03-19T03:23:30.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.849+0000 7fe922de7700 1 -- 172.21.15.178:0/3612084965 wait complete. 2024-03-19T03:23:30.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.849+0000 7fe922de7700 1 Processor -- start 2024-03-19T03:23:30.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.849+0000 7fe922de7700 1 -- start start 2024-03-19T03:23:30.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.849+0000 7fe922de7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe91c100e70 0x7fe91c1966e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:30.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.849+0000 7fe922de7700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe91c10b1f0 con 0x7fe91c100e70 2024-03-19T03:23:30.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.850+0000 7fe920b83700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe91c100e70 0x7fe91c1966e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:30.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.850+0000 7fe920b83700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe91c100e70 0x7fe91c1966e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.178:47996/0 (socket says 172.21.15.178:47996) 2024-03-19T03:23:30.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.850+0000 7fe920b83700 1 -- 172.21.15.178:0/421305821 learned_addr learned my addr 172.21.15.178:0/421305821 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-19T03:23:30.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.851+0000 7fe920b83700 1 -- 172.21.15.178:0/421305821 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe91c196c20 con 0x7fe91c100e70 2024-03-19T03:23:30.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.851+0000 7fe920b83700 1 --2- 172.21.15.178:0/421305821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe91c100e70 0x7fe91c1966e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe910005850 tx=0x7fe910005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:23:30.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.851+0000 7fe919ffb700 1 -- 172.21.15.178:0/421305821 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe910026070 con 0x7fe91c100e70 2024-03-19T03:23:30.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.851+0000 7fe922de7700 1 -- 172.21.15.178:0/421305821 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe91c196f10 con 0x7fe91c100e70 2024-03-19T03:23:30.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.852+0000 7fe922de7700 1 -- 172.21.15.178:0/421305821 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe91c19afb0 con 0x7fe91c100e70 2024-03-19T03:23:30.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.852+0000 7fe919ffb700 1 -- 172.21.15.178:0/421305821 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe91000bbd0 con 0x7fe91c100e70 2024-03-19T03:23:30.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.852+0000 7fe919ffb700 1 -- 172.21.15.178:0/421305821 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe9100039b0 con 0x7fe91c100e70 2024-03-19T03:23:30.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.853+0000 7fe922de7700 1 -- 172.21.15.178:0/421305821 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe900005330 con 0x7fe91c100e70 2024-03-19T03:23:30.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.853+0000 7fe919ffb700 1 -- 172.21.15.178:0/421305821 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7fe910029070 con 0x7fe91c100e70 2024-03-19T03:23:30.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.853+0000 7fe919ffb700 1 --2- 172.21.15.178:0/421305821 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe90c038fc0 0x7fe90c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:30.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.853+0000 7fe919ffb700 1 -- 172.21.15.178:0/421305821 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe91001e070 con 0x7fe91c100e70 2024-03-19T03:23:30.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.854+0000 7fe91bfff700 1 --2- 172.21.15.178:0/421305821 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe90c038fc0 0x7fe90c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:30.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.854+0000 7fe91bfff700 1 --2- 172.21.15.178:0/421305821 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe90c038fc0 0x7fe90c03b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe908006fd0 tx=0x7fe908006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:23:30.870 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:30.869+0000 7fe919ffb700 1 -- 172.21.15.178:0/421305821 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe910013070 con 0x7fe91c100e70 2024-03-19T03:23:31.166 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:31.165+0000 7fe922de7700 1 -- 172.21.15.178:0/421305821 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fe900005620 con 0x7fe91c100e70 2024-03-19T03:23:31.167 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:31.166+0000 7fe919ffb700 1 -- 172.21.15.178:0/421305821 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+749 (secure 0 0 0) 0x7fe910029540 con 0x7fe91c100e70 2024-03-19T03:23:31.167 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-19T03:23:31.167 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"a3a82456-e59f-11ee-95c9-87774f69a715","modified":"2024-03-19T03:21:26.225562Z","created":"2024-03-19T03:21:26.225562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-19T03:23:31.170 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:31.168+0000 7fe922de7700 1 -- 172.21.15.178:0/421305821 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe90c038fc0 msgr2=0x7fe90c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:31.170 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:31.168+0000 7fe922de7700 1 --2- 172.21.15.178:0/421305821 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe90c038fc0 0x7fe90c03b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe908006fd0 tx=0x7fe908006e40 comp rx=0 tx=0).stop 2024-03-19T03:23:31.170 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:31.168+0000 7fe922de7700 1 -- 172.21.15.178:0/421305821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe91c100e70 msgr2=0x7fe91c1966e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:31.170 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:31.168+0000 7fe922de7700 1 --2- 172.21.15.178:0/421305821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe91c100e70 0x7fe91c1966e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe910005850 tx=0x7fe910005030 comp rx=0 tx=0).stop 2024-03-19T03:23:31.170 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:31.168+0000 7fe922de7700 1 -- 172.21.15.178:0/421305821 shutdown_connections 2024-03-19T03:23:31.170 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:31.168+0000 7fe922de7700 1 --2- 172.21.15.178:0/421305821 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe90c038fc0 0x7fe90c03b480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:31.170 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:31.168+0000 7fe922de7700 1 --2- 172.21.15.178:0/421305821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe91c100e70 0x7fe91c1966e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:31.170 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:31.168+0000 7fe922de7700 1 -- 172.21.15.178:0/421305821 >> 172.21.15.178:0/421305821 conn(0x7fe91c0779b0 msgr2=0x7fe91c0fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:23:31.170 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:31.168+0000 7fe922de7700 1 -- 172.21.15.178:0/421305821 shutdown_connections 2024-03-19T03:23:31.170 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:31.168+0000 7fe922de7700 1 -- 172.21.15.178:0/421305821 wait complete. 2024-03-19T03:23:31.171 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-03-19T03:23:31.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:31 smithi178 systemd[1]: Starting Ceph mon.c for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:23:31.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:31 smithi117 ceph-mon[31838]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:31.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:31 smithi117 ceph-mon[31838]: from='client.? 172.21.15.178:0/421305821' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:23:31.996 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:31 smithi178 podman[67559]: 2024-03-19 03:23:31.89237492 +0000 UTC m=+0.109615572 container create fa156234d0fc055bf2208e77b34be92c16bb3146d9a8ef5724c397b6b6ae151f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-19T03:23:31.997 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:31 smithi178 podman[67559]: 2024-03-19 03:23:31.800581277 +0000 UTC m=+0.017821939 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:23:32.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 podman[67559]: 2024-03-19 03:23:32.078195935 +0000 UTC m=+0.295436596 container init fa156234d0fc055bf2208e77b34be92c16bb3146d9a8ef5724c397b6b6ae151f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-c, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-03-19T03:23:32.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 podman[67559]: 2024-03-19 03:23:32.081002576 +0000 UTC m=+0.298243230 container start fa156234d0fc055bf2208e77b34be92c16bb3146d9a8ef5724c397b6b6ae151f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-c, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-19T03:23:32.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: set uid:gid to 167:167 (ceph:ceph) 2024-03-19T03:23:32.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev), process ceph-mon, pid 2 2024-03-19T03:23:32.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: pidfile_write: ignore empty --pid-file 2024-03-19T03:23:32.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: load: jerasure load: lrc 2024-03-19T03:23:32.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: RocksDB version: 7.9.2 2024-03-19T03:23:32.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Git sha 0 2024-03-19T03:23:32.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Compile date 2024-03-18 18:57:12 2024-03-19T03:23:32.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: DB SUMMARY 2024-03-19T03:23:32.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: DB Session ID: YTJKKL86DF10L1AV1HG2 2024-03-19T03:23:32.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: CURRENT file: CURRENT 2024-03-19T03:23:32.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: IDENTITY file: IDENTITY 2024-03-19T03:23:32.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-19T03:23:32.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-19T03:23:32.348 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-19T03:23:32.348 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.error_if_exists: 0 2024-03-19T03:23:32.348 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.create_if_missing: 0 2024-03-19T03:23:32.348 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.paranoid_checks: 1 2024-03-19T03:23:32.348 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-19T03:23:32.348 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-19T03:23:32.348 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-19T03:23:32.348 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.env: 0x55aef5816c60 2024-03-19T03:23:32.348 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.fs: PosixFileSystem 2024-03-19T03:23:32.348 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.info_log: 0x55aef6c2dae0 2024-03-19T03:23:32.348 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_file_opening_threads: 16 2024-03-19T03:23:32.348 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.statistics: (nil) 2024-03-19T03:23:32.348 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.use_fsync: 0 2024-03-19T03:23:32.348 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_log_file_size: 0 2024-03-19T03:23:32.349 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-19T03:23:32.349 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-19T03:23:32.349 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.keep_log_file_num: 1000 2024-03-19T03:23:32.349 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.recycle_log_file_num: 0 2024-03-19T03:23:32.349 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.allow_fallocate: 1 2024-03-19T03:23:32.349 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.allow_mmap_reads: 0 2024-03-19T03:23:32.349 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.allow_mmap_writes: 0 2024-03-19T03:23:32.349 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.use_direct_reads: 0 2024-03-19T03:23:32.349 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-19T03:23:32.349 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.create_missing_column_families: 0 2024-03-19T03:23:32.349 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.db_log_dir: 2024-03-19T03:23:32.349 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.wal_dir: 2024-03-19T03:23:32.349 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-19T03:23:32.349 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-19T03:23:32.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-19T03:23:32.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-19T03:23:32.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-19T03:23:32.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-19T03:23:32.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.advise_random_on_open: 1 2024-03-19T03:23:32.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.db_write_buffer_size: 0 2024-03-19T03:23:32.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.write_buffer_manager: 0x55aef6c001e0 2024-03-19T03:23:32.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-19T03:23:32.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-19T03:23:32.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-19T03:23:32.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.rate_limiter: (nil) 2024-03-19T03:23:32.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-19T03:23:32.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.wal_recovery_mode: 2 2024-03-19T03:23:32.350 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.enable_thread_tracking: 0 2024-03-19T03:23:32.351 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.enable_pipelined_write: 0 2024-03-19T03:23:32.351 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.unordered_write: 0 2024-03-19T03:23:32.351 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-19T03:23:32.351 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-19T03:23:32.351 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-19T03:23:32.351 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-19T03:23:32.351 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.row_cache: None 2024-03-19T03:23:32.351 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.wal_filter: None 2024-03-19T03:23:32.353 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-19T03:23:32.353 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.allow_ingest_behind: 0 2024-03-19T03:23:32.353 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.two_write_queues: 0 2024-03-19T03:23:32.353 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.manual_wal_flush: 0 2024-03-19T03:23:32.353 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.wal_compression: 0 2024-03-19T03:23:32.353 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.atomic_flush: 0 2024-03-19T03:23:32.353 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-19T03:23:32.353 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-19T03:23:32.353 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-19T03:23:32.353 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.log_readahead_size: 0 2024-03-19T03:23:32.353 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-19T03:23:32.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.best_efforts_recovery: 0 2024-03-19T03:23:32.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-19T03:23:32.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-19T03:23:32.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.allow_data_in_errors: 0 2024-03-19T03:23:32.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.db_host_id: __hostname__ 2024-03-19T03:23:32.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-19T03:23:32.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_background_jobs: 2 2024-03-19T03:23:32.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_background_compactions: -1 2024-03-19T03:23:32.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_subcompactions: 1 2024-03-19T03:23:32.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-19T03:23:32.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-19T03:23:32.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-19T03:23:32.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_total_wal_size: 0 2024-03-19T03:23:32.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-19T03:23:32.354 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-19T03:23:32.355 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-19T03:23:32.355 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-19T03:23:32.355 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_open_files: -1 2024-03-19T03:23:32.355 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.bytes_per_sync: 0 2024-03-19T03:23:32.355 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-19T03:23:32.355 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-19T03:23:32.355 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.compaction_readahead_size: 0 2024-03-19T03:23:32.355 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_background_flushes: -1 2024-03-19T03:23:32.355 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Compression algorithms supported: 2024-03-19T03:23:32.355 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-19T03:23:32.355 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: kZSTD supported: 0 2024-03-19T03:23:32.355 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: kXpressCompression supported: 0 2024-03-19T03:23:32.355 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: kLZ4HCCompression supported: 1 2024-03-19T03:23:32.355 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: kZlibCompression supported: 1 2024-03-19T03:23:32.355 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: kSnappyCompression supported: 1 2024-03-19T03:23:32.356 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: kLZ4Compression supported: 1 2024-03-19T03:23:32.356 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: kBZip2Compression supported: 0 2024-03-19T03:23:32.356 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-19T03:23:32.356 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-19T03:23:32.356 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-19T03:23:32.356 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-19T03:23:32.356 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-19T03:23:32.356 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.merge_operator: 2024-03-19T03:23:32.356 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.compaction_filter: None 2024-03-19T03:23:32.356 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.compaction_filter_factory: None 2024-03-19T03:23:32.356 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.sst_partitioner_factory: None 2024-03-19T03:23:32.356 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-19T03:23:32.356 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-19T03:23:32.356 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55aef6c2dbe0) 2024-03-19T03:23:32.356 INFO:journalctl@ceph.mon.c.smithi178.stdout: cache_index_and_filter_blocks: 1 2024-03-19T03:23:32.357 INFO:journalctl@ceph.mon.c.smithi178.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-19T03:23:32.357 INFO:journalctl@ceph.mon.c.smithi178.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-19T03:23:32.357 INFO:journalctl@ceph.mon.c.smithi178.stdout: pin_top_level_index_and_filter: 1 2024-03-19T03:23:32.357 INFO:journalctl@ceph.mon.c.smithi178.stdout: index_type: 0 2024-03-19T03:23:32.357 INFO:journalctl@ceph.mon.c.smithi178.stdout: data_block_index_type: 0 2024-03-19T03:23:32.357 INFO:journalctl@ceph.mon.c.smithi178.stdout: index_shortening: 1 2024-03-19T03:23:32.357 INFO:journalctl@ceph.mon.c.smithi178.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-19T03:23:32.357 INFO:journalctl@ceph.mon.c.smithi178.stdout: checksum: 4 2024-03-19T03:23:32.357 INFO:journalctl@ceph.mon.c.smithi178.stdout: no_block_cache: 0 2024-03-19T03:23:32.357 INFO:journalctl@ceph.mon.c.smithi178.stdout: block_cache: 0x55aef6c33090 2024-03-19T03:23:32.357 INFO:journalctl@ceph.mon.c.smithi178.stdout: block_cache_name: BinnedLRUCache 2024-03-19T03:23:32.357 INFO:journalctl@ceph.mon.c.smithi178.stdout: block_cache_options: 2024-03-19T03:23:32.357 INFO:journalctl@ceph.mon.c.smithi178.stdout: capacity : 536870912 2024-03-19T03:23:32.357 INFO:journalctl@ceph.mon.c.smithi178.stdout: num_shard_bits : 4 2024-03-19T03:23:32.357 INFO:journalctl@ceph.mon.c.smithi178.stdout: strict_capacity_limit : 0 2024-03-19T03:23:32.357 INFO:journalctl@ceph.mon.c.smithi178.stdout: high_pri_pool_ratio: 0.000 2024-03-19T03:23:32.358 INFO:journalctl@ceph.mon.c.smithi178.stdout: block_cache_compressed: (nil) 2024-03-19T03:23:32.358 INFO:journalctl@ceph.mon.c.smithi178.stdout: persistent_cache: (nil) 2024-03-19T03:23:32.358 INFO:journalctl@ceph.mon.c.smithi178.stdout: block_size: 4096 2024-03-19T03:23:32.358 INFO:journalctl@ceph.mon.c.smithi178.stdout: block_size_deviation: 10 2024-03-19T03:23:32.358 INFO:journalctl@ceph.mon.c.smithi178.stdout: block_restart_interval: 16 2024-03-19T03:23:32.358 INFO:journalctl@ceph.mon.c.smithi178.stdout: index_block_restart_interval: 1 2024-03-19T03:23:32.358 INFO:journalctl@ceph.mon.c.smithi178.stdout: metadata_block_size: 4096 2024-03-19T03:23:32.358 INFO:journalctl@ceph.mon.c.smithi178.stdout: partition_filters: 0 2024-03-19T03:23:32.358 INFO:journalctl@ceph.mon.c.smithi178.stdout: use_delta_encoding: 1 2024-03-19T03:23:32.358 INFO:journalctl@ceph.mon.c.smithi178.stdout: filter_policy: bloomfilter 2024-03-19T03:23:32.358 INFO:journalctl@ceph.mon.c.smithi178.stdout: whole_key_filtering: 1 2024-03-19T03:23:32.358 INFO:journalctl@ceph.mon.c.smithi178.stdout: verify_compression: 0 2024-03-19T03:23:32.358 INFO:journalctl@ceph.mon.c.smithi178.stdout: read_amp_bytes_per_bit: 0 2024-03-19T03:23:32.358 INFO:journalctl@ceph.mon.c.smithi178.stdout: format_version: 5 2024-03-19T03:23:32.358 INFO:journalctl@ceph.mon.c.smithi178.stdout: enable_index_compression: 1 2024-03-19T03:23:32.359 INFO:journalctl@ceph.mon.c.smithi178.stdout: block_align: 0 2024-03-19T03:23:32.359 INFO:journalctl@ceph.mon.c.smithi178.stdout: max_auto_readahead_size: 262144 2024-03-19T03:23:32.359 INFO:journalctl@ceph.mon.c.smithi178.stdout: prepopulate_block_cache: 0 2024-03-19T03:23:32.359 INFO:journalctl@ceph.mon.c.smithi178.stdout: initial_auto_readahead_size: 8192 2024-03-19T03:23:32.359 INFO:journalctl@ceph.mon.c.smithi178.stdout: num_file_reads_for_auto_readahead: 2 2024-03-19T03:23:32.359 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.write_buffer_size: 33554432 2024-03-19T03:23:32.359 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_write_buffer_number: 2 2024-03-19T03:23:32.359 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.compression: NoCompression 2024-03-19T03:23:32.359 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.bottommost_compression: Disabled 2024-03-19T03:23:32.359 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.prefix_extractor: nullptr 2024-03-19T03:23:32.359 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-19T03:23:32.359 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.num_levels: 7 2024-03-19T03:23:32.359 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-19T03:23:32.359 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-19T03:23:32.360 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-19T03:23:32.360 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-19T03:23:32.360 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-19T03:23:32.360 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-19T03:23:32.360 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-19T03:23:32.360 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-19T03:23:32.360 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-19T03:23:32.360 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-19T03:23:32.360 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-19T03:23:32.360 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-19T03:23:32.360 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-19T03:23:32.360 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.compression_opts.level: 32767 2024-03-19T03:23:32.360 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.compression_opts.strategy: 0 2024-03-19T03:23:32.360 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-19T03:23:32.360 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-19T03:23:32.360 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-19T03:23:32.361 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-19T03:23:32.361 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.compression_opts.enabled: false 2024-03-19T03:23:32.361 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-19T03:23:32.361 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-19T03:23:32.361 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-19T03:23:32.361 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-19T03:23:32.361 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.target_file_size_base: 67108864 2024-03-19T03:23:32.361 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-19T03:23:32.361 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-19T03:23:32.361 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-19T03:23:32.361 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-19T03:23:32.361 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-19T03:23:32.361 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-19T03:23:32.361 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-19T03:23:32.362 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-19T03:23:32.362 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-19T03:23:32.362 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-19T03:23:32.362 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-19T03:23:32.362 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-19T03:23:32.362 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-19T03:23:32.362 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-19T03:23:32.362 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.arena_block_size: 1048576 2024-03-19T03:23:32.362 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-19T03:23:32.362 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-19T03:23:32.362 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.disable_auto_compactions: 0 2024-03-19T03:23:32.362 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-19T03:23:32.362 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-19T03:23:32.362 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-19T03:23:32.363 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-19T03:23:32.363 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-19T03:23:32.363 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-19T03:23:32.363 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-19T03:23:32.363 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-19T03:23:32.363 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-19T03:23:32.363 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-19T03:23:32.363 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.table_properties_collectors: 2024-03-19T03:23:32.363 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.inplace_update_support: 0 2024-03-19T03:23:32.363 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-19T03:23:32.363 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-19T03:23:32.363 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-19T03:23:32.363 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-19T03:23:32.363 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.bloom_locality: 0 2024-03-19T03:23:32.363 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.max_successive_merges: 0 2024-03-19T03:23:32.364 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-19T03:23:32.364 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.paranoid_file_checks: 0 2024-03-19T03:23:32.364 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.force_consistency_checks: 1 2024-03-19T03:23:32.364 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.report_bg_io_stats: 0 2024-03-19T03:23:32.364 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.ttl: 2592000 2024-03-19T03:23:32.364 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-19T03:23:32.364 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-19T03:23:32.364 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-19T03:23:32.364 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.enable_blob_files: false 2024-03-19T03:23:32.364 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.min_blob_size: 0 2024-03-19T03:23:32.364 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.blob_file_size: 268435456 2024-03-19T03:23:32.364 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-19T03:23:32.364 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-19T03:23:32.364 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-19T03:23:32.365 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-19T03:23:32.365 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-19T03:23:32.365 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.blob_file_starting_level: 0 2024-03-19T03:23:32.365 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-19T03:23:32.365 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-19T03:23:32.365 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-19T03:23:32.365 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2df0fd2f-ba2a-4bcc-a70f-a2aed5675ed5 2024-03-19T03:23:32.365 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710818612110582, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-19T03:23:32.365 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-19T03:23:32.365 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710818612111004, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710818612, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2df0fd2f-ba2a-4bcc-a70f-a2aed5675ed5", "db_session_id": "YTJKKL86DF10L1AV1HG2", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-19T03:23:32.365 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710818612111083, "job": 1, "event": "recovery_finished"} 2024-03-19T03:23:32.365 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-19T03:23:32.365 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-19T03:23:32.365 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55aef6d26000 2024-03-19T03:23:32.365 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: DB pointer 0x55aef6d10000 2024-03-19T03:23:32.365 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-19T03:23:32.366 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-19T03:23:32.366 INFO:journalctl@ceph.mon.c.smithi178.stdout: ** DB Stats ** 2024-03-19T03:23:32.366 INFO:journalctl@ceph.mon.c.smithi178.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-19T03:23:32.366 INFO:journalctl@ceph.mon.c.smithi178.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-19T03:23:32.366 INFO:journalctl@ceph.mon.c.smithi178.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-19T03:23:32.366 INFO:journalctl@ceph.mon.c.smithi178.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-19T03:23:32.366 INFO:journalctl@ceph.mon.c.smithi178.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-19T03:23:32.366 INFO:journalctl@ceph.mon.c.smithi178.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-19T03:23:32.366 INFO:journalctl@ceph.mon.c.smithi178.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-19T03:23:32.366 INFO:journalctl@ceph.mon.c.smithi178.stdout: 2024-03-19T03:23:32.366 INFO:journalctl@ceph.mon.c.smithi178.stdout: ** Compaction Stats [default] ** 2024-03-19T03:23:32.366 INFO:journalctl@ceph.mon.c.smithi178.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-19T03:23:32.366 INFO:journalctl@ceph.mon.c.smithi178.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-19T03:23:32.366 INFO:journalctl@ceph.mon.c.smithi178.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-19T03:23:32.366 INFO:journalctl@ceph.mon.c.smithi178.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-19T03:23:32.367 INFO:journalctl@ceph.mon.c.smithi178.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-19T03:23:32.367 INFO:journalctl@ceph.mon.c.smithi178.stdout: 2024-03-19T03:23:32.367 INFO:journalctl@ceph.mon.c.smithi178.stdout: ** Compaction Stats [default] ** 2024-03-19T03:23:32.367 INFO:journalctl@ceph.mon.c.smithi178.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-19T03:23:32.367 INFO:journalctl@ceph.mon.c.smithi178.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-19T03:23:32.367 INFO:journalctl@ceph.mon.c.smithi178.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-19T03:23:32.367 INFO:journalctl@ceph.mon.c.smithi178.stdout: 2024-03-19T03:23:32.367 INFO:journalctl@ceph.mon.c.smithi178.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-19T03:23:32.367 INFO:journalctl@ceph.mon.c.smithi178.stdout: 2024-03-19T03:23:32.367 INFO:journalctl@ceph.mon.c.smithi178.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-19T03:23:32.367 INFO:journalctl@ceph.mon.c.smithi178.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-19T03:23:32.367 INFO:journalctl@ceph.mon.c.smithi178.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-19T03:23:32.367 INFO:journalctl@ceph.mon.c.smithi178.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-19T03:23:32.367 INFO:journalctl@ceph.mon.c.smithi178.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-19T03:23:32.367 INFO:journalctl@ceph.mon.c.smithi178.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-19T03:23:32.367 INFO:journalctl@ceph.mon.c.smithi178.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-19T03:23:32.368 INFO:journalctl@ceph.mon.c.smithi178.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-19T03:23:32.368 INFO:journalctl@ceph.mon.c.smithi178.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-19T03:23:32.368 INFO:journalctl@ceph.mon.c.smithi178.stdout: Block cache BinnedLRUCache@0x55aef6c33090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 7e-06 secs_since: 0 2024-03-19T03:23:32.368 INFO:journalctl@ceph.mon.c.smithi178.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-19T03:23:32.368 INFO:journalctl@ceph.mon.c.smithi178.stdout: 2024-03-19T03:23:32.368 INFO:journalctl@ceph.mon.c.smithi178.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-19T03:23:32.368 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-19T03:23:32.368 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: using public_addr v2:172.21.15.178:0/0 -> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-03-19T03:23:32.368 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: starting mon.c rank -1 at public addrs [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] at bind addrs [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:23:32.368 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: mon.c@-1(???) e0 preinit fsid a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:23:32.368 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: mon.c@-1(synchronizing).mds e1 new map 2024-03-19T03:23:32.368 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-19T03:23:32.368 INFO:journalctl@ceph.mon.c.smithi178.stdout: e1 2024-03-19T03:23:32.368 INFO:journalctl@ceph.mon.c.smithi178.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-19T03:23:32.368 INFO:journalctl@ceph.mon.c.smithi178.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-19T03:23:32.369 INFO:journalctl@ceph.mon.c.smithi178.stdout: legacy client fscid: -1 2024-03-19T03:23:32.369 INFO:journalctl@ceph.mon.c.smithi178.stdout: 2024-03-19T03:23:32.369 INFO:journalctl@ceph.mon.c.smithi178.stdout: No filesystems configured 2024-03-19T03:23:32.369 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-19T03:23:32.369 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-19T03:23:32.369 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-19T03:23:32.369 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-19T03:23:32.369 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-19T03:23:32.369 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-19T03:23:32.369 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-19T03:23:32.369 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T03:23:32.369 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T03:23:32.369 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T03:23:32.369 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi119", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:23:32.369 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: Deploying cephadm binary to smithi119 2024-03-19T03:23:32.370 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:32.370 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:32.370 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:32.370 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: Added host smithi119 2024-03-19T03:23:32.370 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:23:32.370 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:32.370 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:32.370 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:32.370 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:32.370 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:32.370 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T03:23:32.370 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:32.370 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:32.370 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:32.371 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:32.371 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:32.371 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi119", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:23:32.371 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:32.371 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:23:32.371 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-19T03:23:32.371 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: Updating smithi119:/var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:23:32.371 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-03-19T03:23:32.371 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: Updating smithi119:/var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T03:23:32.371 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:32.371 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi178", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:23:32.371 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:32.371 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:32.371 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:32.371 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: Deploying cephadm binary to smithi178 2024-03-19T03:23:32.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:32.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:32.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:32.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:23:32.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: Added host smithi178 2024-03-19T03:23:32.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:32.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:32.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:32.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:32.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T03:23:32.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:32.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:32.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:32.372 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:32.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:32.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:32.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi178", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:23:32.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:32.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:23:32.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: Updating smithi178:/etc/ceph/ceph.conf 2024-03-19T03:23:32.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: Updating smithi178:/var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:23:32.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3655883920' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-19T03:23:32.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:32.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:32.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:32.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: Updating smithi178:/etc/ceph/ceph.client.admin.keyring 2024-03-19T03:23:32.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: Updating smithi178:/var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T03:23:32.373 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:32.374 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3655883920' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-19T03:23:32.374 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T03:23:32.374 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:32.374 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:32.374 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:23:32.374 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:32.374 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:23:32.374 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:32.374 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T03:23:32.374 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:32.374 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi117:172.21.15.117=a;smithi119:172.21.15.119=b;smithi178:172.21.15.178=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:23:32.374 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: Saving service mon spec with placement smithi117:172.21.15.117=a;smithi119:172.21.15.119=b;smithi178:172.21.15.178=c;count:3 2024-03-19T03:23:32.374 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: Deploying daemon mon.c on smithi178 2024-03-19T03:23:32.374 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:32.374 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:32.375 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: from='client.? 172.21.15.178:0/421305821' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:23:32.375 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 ceph-mon[67578]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-19T03:23:32.375 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 bash[67559]: fa156234d0fc055bf2208e77b34be92c16bb3146d9a8ef5724c397b6b6ae151f 2024-03-19T03:23:32.375 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:32 smithi178 systemd[1]: Started Ceph mon.c for a3a82456-e59f-11ee-95c9-87774f69a715. 2024-03-19T03:23:32.758 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-19T03:23:32.758 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T03:23:32.992 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T03:23:37.276 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.273+0000 7fe26863a700 1 -- 172.21.15.178:0/575040279 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe24c0035c0 msgr2=0x7fe24c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:37.277 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.273+0000 7fe26863a700 1 --2- 172.21.15.178:0/575040279 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe24c0035c0 0x7fe24c005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe26006a5d0 tx=0x7fe25c00feb0 comp rx=0 tx=0).stop 2024-03-19T03:23:37.277 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.273+0000 7fe26863a700 1 -- 172.21.15.178:0/575040279 shutdown_connections 2024-03-19T03:23:37.277 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.273+0000 7fe26863a700 1 --2- 172.21.15.178:0/575040279 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe24c0035c0 0x7fe24c005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:37.277 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.273+0000 7fe26863a700 1 -- 172.21.15.178:0/575040279 >> 172.21.15.178:0/575040279 conn(0x7fe2600fc7b0 msgr2=0x7fe2600febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:23:37.277 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.273+0000 7fe26863a700 1 -- 172.21.15.178:0/575040279 shutdown_connections 2024-03-19T03:23:37.277 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.273+0000 7fe26863a700 1 -- 172.21.15.178:0/575040279 wait complete. 2024-03-19T03:23:37.277 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.274+0000 7fe26863a700 1 Processor -- start 2024-03-19T03:23:37.277 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.274+0000 7fe26863a700 1 -- start start 2024-03-19T03:23:37.277 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.275+0000 7fe26863a700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe24c0035c0 0x7fe26019d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:37.277 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.275+0000 7fe26863a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe26019dbf0 0x7fe2601978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:37.277 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.275+0000 7fe26863a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe260109e80 con 0x7fe26019dbf0 2024-03-19T03:23:37.277 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.275+0000 7fe26863a700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fe260109d00 con 0x7fe24c0035c0 2024-03-19T03:23:37.277 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.275+0000 7fe2663d6700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe24c0035c0 0x7fe26019d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:37.277 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.275+0000 7fe2663d6700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe24c0035c0 0x7fe26019d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:34706/0 (socket says 172.21.15.178:34706) 2024-03-19T03:23:37.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.275+0000 7fe2663d6700 1 -- 172.21.15.178:0/1525663200 learned_addr learned my addr 172.21.15.178:0/1525663200 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-19T03:23:37.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.275+0000 7fe2663d6700 1 -- 172.21.15.178:0/1525663200 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe24c0035c0 msgr2=0x7fe26019d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-19T03:23:37.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.275+0000 7fe2663d6700 1 -- 172.21.15.178:0/1525663200 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe24c0035c0 msgr2=0x7fe26019d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-19T03:23:37.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.275+0000 7fe2663d6700 1 --2- 172.21.15.178:0/1525663200 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe24c0035c0 0x7fe26019d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T03:23:37.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.275+0000 7fe2663d6700 1 --2- 172.21.15.178:0/1525663200 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe24c0035c0 0x7fe26019d6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T03:23:37.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.275+0000 7fe265bd5700 1 --2- 172.21.15.178:0/1525663200 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe26019dbf0 0x7fe2601978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:37.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.276+0000 7fe265bd5700 1 -- 172.21.15.178:0/1525663200 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe24c0035c0 msgr2=0x7fe26019d6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T03:23:37.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.276+0000 7fe265bd5700 1 --2- 172.21.15.178:0/1525663200 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe24c0035c0 0x7fe26019d6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:37.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.276+0000 7fe265bd5700 1 -- 172.21.15.178:0/1525663200 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe26019e230 con 0x7fe26019dbf0 2024-03-19T03:23:37.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.276+0000 7fe265bd5700 1 --2- 172.21.15.178:0/1525663200 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe26019dbf0 0x7fe2601978d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe25000b4d0 tx=0x7fe25000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:23:37.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.276+0000 7fe2577fe700 1 -- 172.21.15.178:0/1525663200 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fe250011630 con 0x7fe26019dbf0 2024-03-19T03:23:37.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.276+0000 7fe26863a700 1 -- 172.21.15.178:0/1525663200 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe260197e70 con 0x7fe26019dbf0 2024-03-19T03:23:37.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.276+0000 7fe2577fe700 1 -- 172.21.15.178:0/1525663200 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe250011cb0 con 0x7fe26019dbf0 2024-03-19T03:23:37.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.276+0000 7fe2577fe700 1 -- 172.21.15.178:0/1525663200 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fe250010e60 con 0x7fe26019dbf0 2024-03-19T03:23:37.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.277+0000 7fe26863a700 1 -- 172.21.15.178:0/1525663200 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2601983d0 con 0x7fe26019dbf0 2024-03-19T03:23:37.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.278+0000 7fe26863a700 1 -- 172.21.15.178:0/1525663200 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe26006add0 con 0x7fe26019dbf0 2024-03-19T03:23:37.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.278+0000 7fe2577fe700 1 -- 172.21.15.178:0/1525663200 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7fe250018020 con 0x7fe26019dbf0 2024-03-19T03:23:37.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.278+0000 7fe2577fe700 1 --2- 172.21.15.178:0/1525663200 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe244038d10 0x7fe24403b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:37.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.278+0000 7fe2577fe700 1 -- 172.21.15.178:0/1525663200 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe25004ba90 con 0x7fe26019dbf0 2024-03-19T03:23:37.281 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.279+0000 7fe2663d6700 1 --2- 172.21.15.178:0/1525663200 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe244038d10 0x7fe24403b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:37.281 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.280+0000 7fe2663d6700 1 --2- 172.21.15.178:0/1525663200 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe244038d10 0x7fe24403b1d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00fa60 tx=0x7fe25c005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:23:37.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.299+0000 7fe2577fe700 1 -- 172.21.15.178:0/1525663200 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe25001f020 con 0x7fe26019dbf0 2024-03-19T03:23:37.482 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T03:23:37.482 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: mon.a calling monitor election 2024-03-19T03:23:37.483 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:23:37.483 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:37.483 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:23:37.483 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: mon.c calling monitor election 2024-03-19T03:23:37.483 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:23:37.483 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:37.483 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:23:37.483 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:23:37.483 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:37.483 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:23:37.483 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-19T03:23:37.483 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: monmap epoch 2 2024-03-19T03:23:37.483 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: fsid a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:23:37.484 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: last_changed 2024-03-19T03:23:32.141056+0000 2024-03-19T03:23:37.484 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: created 2024-03-19T03:21:26.225562+0000 2024-03-19T03:23:37.484 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: min_mon_release 19 (squid) 2024-03-19T03:23:37.484 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: election_strategy: 1 2024-03-19T03:23:37.484 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: 0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-03-19T03:23:37.484 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: 1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.c 2024-03-19T03:23:37.484 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: fsmap 2024-03-19T03:23:37.484 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T03:23:37.484 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: mgrmap e13: x(active, since 59s) 2024-03-19T03:23:37.484 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: overall HEALTH_OK 2024-03-19T03:23:37.484 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:37.484 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:37.484 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:37.484 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T03:23:37.484 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:37.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T03:23:37.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: mon.a calling monitor election 2024-03-19T03:23:37.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:23:37.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:37.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:23:37.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: mon.c calling monitor election 2024-03-19T03:23:37.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:23:37.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:37.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:23:37.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:23:37.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:37.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:23:37.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-19T03:23:37.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: monmap epoch 2 2024-03-19T03:23:37.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: fsid a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:23:37.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: last_changed 2024-03-19T03:23:32.141056+0000 2024-03-19T03:23:37.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: created 2024-03-19T03:21:26.225562+0000 2024-03-19T03:23:37.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: min_mon_release 19 (squid) 2024-03-19T03:23:37.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: election_strategy: 1 2024-03-19T03:23:37.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: 0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-03-19T03:23:37.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: 1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.c 2024-03-19T03:23:37.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: fsmap 2024-03-19T03:23:37.545 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T03:23:37.545 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: mgrmap e13: x(active, since 59s) 2024-03-19T03:23:37.545 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: overall HEALTH_OK 2024-03-19T03:23:37.545 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:37.545 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:37.545 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:37.545 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T03:23:37.545 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:37.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.583+0000 7fe26863a700 1 -- 172.21.15.178:0/1525663200 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fe260198740 con 0x7fe26019dbf0 2024-03-19T03:23:37.586 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.585+0000 7fe2577fe700 1 -- 172.21.15.178:0/1525663200 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7fe250014170 con 0x7fe26019dbf0 2024-03-19T03:23:37.586 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-19T03:23:37.586 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":2,"fsid":"a3a82456-e59f-11ee-95c9-87774f69a715","modified":"2024-03-19T03:23:32.141056Z","created":"2024-03-19T03:21:26.225562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:3300","nonce":0},{"type":"v1","addr":"172.21.15.178:6789","nonce":0}]},"addr":"172.21.15.178:6789/0","public_addr":"172.21.15.178:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-19T03:23:37.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.587+0000 7fe26863a700 1 -- 172.21.15.178:0/1525663200 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe244038d10 msgr2=0x7fe24403b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:37.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.587+0000 7fe26863a700 1 --2- 172.21.15.178:0/1525663200 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe244038d10 0x7fe24403b1d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00fa60 tx=0x7fe25c005a90 comp rx=0 tx=0).stop 2024-03-19T03:23:37.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.587+0000 7fe26863a700 1 -- 172.21.15.178:0/1525663200 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe26019dbf0 msgr2=0x7fe2601978d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:37.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.587+0000 7fe26863a700 1 --2- 172.21.15.178:0/1525663200 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe26019dbf0 0x7fe2601978d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe25000b4d0 tx=0x7fe25000b890 comp rx=0 tx=0).stop 2024-03-19T03:23:37.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.587+0000 7fe26863a700 1 -- 172.21.15.178:0/1525663200 shutdown_connections 2024-03-19T03:23:37.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.587+0000 7fe26863a700 1 --2- 172.21.15.178:0/1525663200 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe244038d10 0x7fe24403b1d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:37.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.587+0000 7fe26863a700 1 --2- 172.21.15.178:0/1525663200 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe24c0035c0 0x7fe26019d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:37.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.587+0000 7fe26863a700 1 --2- 172.21.15.178:0/1525663200 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe26019dbf0 0x7fe2601978d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:37.590 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.587+0000 7fe26863a700 1 -- 172.21.15.178:0/1525663200 >> 172.21.15.178:0/1525663200 conn(0x7fe2600fc7b0 msgr2=0x7fe260103df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:23:37.590 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.588+0000 7fe26863a700 1 -- 172.21.15.178:0/1525663200 shutdown_connections 2024-03-19T03:23:37.590 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:37.588+0000 7fe26863a700 1 -- 172.21.15.178:0/1525663200 wait complete. 2024-03-19T03:23:37.590 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 2 2024-03-19T03:23:38.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:38 smithi117 ceph-mon[31838]: Deploying daemon mon.b on smithi119 2024-03-19T03:23:38.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:38 smithi117 ceph-mon[31838]: from='client.? 172.21.15.178:0/1525663200' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:23:38.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:38 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:23:38.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:38 smithi178 ceph-mon[67578]: Deploying daemon mon.b on smithi119 2024-03-19T03:23:38.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:38 smithi178 ceph-mon[67578]: from='client.? 172.21.15.178:0/1525663200' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:23:38.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:38 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:23:39.151 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-19T03:23:39.151 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T03:23:39.395 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T03:23:39.517 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:39 smithi178 ceph-mon[67578]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:39.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:39 smithi117 ceph-mon[31838]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:40.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.799+0000 7f1057bad700 1 -- 172.21.15.178:0/1422910475 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1050102ff0 msgr2=0x7f10501033d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:40.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.799+0000 7f1057bad700 1 --2- 172.21.15.178:0/1422910475 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1050102ff0 0x7f10501033d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f104800b0a0 tx=0x7f104800b3b0 comp rx=0 tx=0).stop 2024-03-19T03:23:40.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.799+0000 7f1057bad700 1 -- 172.21.15.178:0/1422910475 shutdown_connections 2024-03-19T03:23:40.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.799+0000 7f1057bad700 1 --2- 172.21.15.178:0/1422910475 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1050102ff0 0x7f10501033d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:40.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.799+0000 7f1057bad700 1 -- 172.21.15.178:0/1422910475 >> 172.21.15.178:0/1422910475 conn(0x7f10500fc7c0 msgr2=0x7f10500febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:23:40.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.800+0000 7f1057bad700 1 -- 172.21.15.178:0/1422910475 shutdown_connections 2024-03-19T03:23:40.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.800+0000 7f1057bad700 1 -- 172.21.15.178:0/1422910475 wait complete. 2024-03-19T03:23:40.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.800+0000 7f1057bad700 1 Processor -- start 2024-03-19T03:23:40.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.800+0000 7f1057bad700 1 -- start start 2024-03-19T03:23:40.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.801+0000 7f1057bad700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1050102ff0 0x7f105019d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:40.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.801+0000 7f1057bad700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f105019dcd0 0x7f1050197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:40.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.801+0000 7f1057bad700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f105010c470 con 0x7f1050102ff0 2024-03-19T03:23:40.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.801+0000 7f1057bad700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f105010c2f0 con 0x7f105019dcd0 2024-03-19T03:23:40.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.801+0000 7f1055148700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f105019dcd0 0x7f1050197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:40.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.801+0000 7f1055148700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f105019dcd0 0x7f1050197950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:34716/0 (socket says 172.21.15.178:34716) 2024-03-19T03:23:40.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.801+0000 7f1055148700 1 -- 172.21.15.178:0/823411457 learned_addr learned my addr 172.21.15.178:0/823411457 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-19T03:23:40.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.801+0000 7f1055148700 1 -- 172.21.15.178:0/823411457 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f105019dcd0 msgr2=0x7f1050197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-19T03:23:40.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.801+0000 7f1055148700 1 -- 172.21.15.178:0/823411457 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f105019dcd0 msgr2=0x7f1050197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-19T03:23:40.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.801+0000 7f1055148700 1 --2- 172.21.15.178:0/823411457 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f105019dcd0 0x7f1050197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T03:23:40.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.801+0000 7f1055148700 1 --2- 172.21.15.178:0/823411457 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f105019dcd0 0x7f1050197950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T03:23:40.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.801+0000 7f1055949700 1 --2- 172.21.15.178:0/823411457 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1050102ff0 0x7f105019d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:40.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.802+0000 7f1055949700 1 -- 172.21.15.178:0/823411457 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f105019dcd0 msgr2=0x7f1050197950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T03:23:40.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.802+0000 7f1055949700 1 --2- 172.21.15.178:0/823411457 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f105019dcd0 0x7f1050197950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:40.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.802+0000 7f1055949700 1 -- 172.21.15.178:0/823411457 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f105019e310 con 0x7f1050102ff0 2024-03-19T03:23:40.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.802+0000 7f1055949700 1 --2- 172.21.15.178:0/823411457 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1050102ff0 0x7f105019d790 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f104800b0a0 tx=0x7f1048008b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:23:40.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.802+0000 7f1046ffd700 1 -- 172.21.15.178:0/823411457 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f1048009420 con 0x7f1050102ff0 2024-03-19T03:23:40.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.802+0000 7f1057bad700 1 -- 172.21.15.178:0/823411457 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1050197e90 con 0x7f1050102ff0 2024-03-19T03:23:40.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.802+0000 7f1057bad700 1 -- 172.21.15.178:0/823411457 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1050198420 con 0x7f1050102ff0 2024-03-19T03:23:40.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.802+0000 7f1046ffd700 1 -- 172.21.15.178:0/823411457 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1048023070 con 0x7f1050102ff0 2024-03-19T03:23:40.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.802+0000 7f1046ffd700 1 -- 172.21.15.178:0/823411457 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f1048003ba0 con 0x7f1050102ff0 2024-03-19T03:23:40.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.803+0000 7f1057bad700 1 -- 172.21.15.178:0/823411457 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1034005330 con 0x7f1050102ff0 2024-03-19T03:23:40.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.804+0000 7f1046ffd700 1 -- 172.21.15.178:0/823411457 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f1048028c40 con 0x7f1050102ff0 2024-03-19T03:23:40.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.804+0000 7f1046ffd700 1 --2- 172.21.15.178:0/823411457 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f103c038f00 0x7f103c03b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:40.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.804+0000 7f1046ffd700 1 -- 172.21.15.178:0/823411457 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f104800f050 con 0x7f1050102ff0 2024-03-19T03:23:40.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.804+0000 7f1055148700 1 --2- 172.21.15.178:0/823411457 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f103c038f00 0x7f103c03b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:40.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.805+0000 7f1055148700 1 --2- 172.21.15.178:0/823411457 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f103c038f00 0x7f103c03b3c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1040009180 tx=0x7f1040008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:23:40.820 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:40.819+0000 7f1046ffd700 1 -- 172.21.15.178:0/823411457 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f104802d020 con 0x7f1050102ff0 2024-03-19T03:23:41.077 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:41.076+0000 7f1046ffd700 1 -- 172.21.15.178:0/823411457 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104801d7c0 con 0x7f1050102ff0 2024-03-19T03:23:41.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:41.094+0000 7f1057bad700 1 -- 172.21.15.178:0/823411457 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f10340058f0 con 0x7f1050102ff0 2024-03-19T03:23:41.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:41 smithi119 systemd[1]: Started Ceph mon.b for a3a82456-e59f-11ee-95c9-87774f69a715. 2024-03-19T03:23:46.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:46.094+0000 7f1046ffd700 1 -- 172.21.15.178:0/823411457 <== mon.0 v2:172.21.15.117:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1287 (secure 0 0 0) 0x7f104804c010 con 0x7f1050102ff0 2024-03-19T03:23:46.096 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-19T03:23:46.096 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":3,"fsid":"a3a82456-e59f-11ee-95c9-87774f69a715","modified":"2024-03-19T03:23:41.075373Z","created":"2024-03-19T03:21:26.225562Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:3300","nonce":0},{"type":"v1","addr":"172.21.15.178:6789","nonce":0}]},"addr":"172.21.15.178:6789/0","public_addr":"172.21.15.178:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-19T03:23:46.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:46.097+0000 7f1057bad700 1 -- 172.21.15.178:0/823411457 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f103c038f00 msgr2=0x7f103c03b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:46.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:46.097+0000 7f1057bad700 1 --2- 172.21.15.178:0/823411457 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f103c038f00 0x7f103c03b3c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1040009180 tx=0x7f1040008040 comp rx=0 tx=0).stop 2024-03-19T03:23:46.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:46.097+0000 7f1057bad700 1 -- 172.21.15.178:0/823411457 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1050102ff0 msgr2=0x7f105019d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:46.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:46.097+0000 7f1057bad700 1 --2- 172.21.15.178:0/823411457 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1050102ff0 0x7f105019d790 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f104800b0a0 tx=0x7f1048008b70 comp rx=0 tx=0).stop 2024-03-19T03:23:46.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:46.097+0000 7f1057bad700 1 -- 172.21.15.178:0/823411457 shutdown_connections 2024-03-19T03:23:46.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:46.097+0000 7f1057bad700 1 --2- 172.21.15.178:0/823411457 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f103c038f00 0x7f103c03b3c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:46.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:46.097+0000 7f1057bad700 1 --2- 172.21.15.178:0/823411457 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1050102ff0 0x7f105019d790 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:46.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:46.097+0000 7f1057bad700 1 --2- 172.21.15.178:0/823411457 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f105019dcd0 0x7f1050197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:46.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:46.097+0000 7f1057bad700 1 -- 172.21.15.178:0/823411457 >> 172.21.15.178:0/823411457 conn(0x7f10500fc7c0 msgr2=0x7f1050103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:23:46.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:46.098+0000 7f1057bad700 1 -- 172.21.15.178:0/823411457 shutdown_connections 2024-03-19T03:23:46.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:46.098+0000 7f1057bad700 1 -- 172.21.15.178:0/823411457 wait complete. 2024-03-19T03:23:46.100 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 3 2024-03-19T03:23:46.403 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: mon.c calling monitor election 2024-03-19T03:23:46.403 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T03:23:46.403 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: mon.a calling monitor election 2024-03-19T03:23:46.403 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:23:46.403 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:23:46.403 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: from='client.? 172.21.15.178:0/823411457' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:23:46.403 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:23:46.403 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:46.403 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:23:46.403 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:23:46.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:46.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:23:46.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:23:46.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-19T03:23:46.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: monmap epoch 3 2024-03-19T03:23:46.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: fsid a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:23:46.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: last_changed 2024-03-19T03:23:41.075373+0000 2024-03-19T03:23:46.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: created 2024-03-19T03:21:26.225562+0000 2024-03-19T03:23:46.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: min_mon_release 19 (squid) 2024-03-19T03:23:46.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: election_strategy: 1 2024-03-19T03:23:46.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: 0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-03-19T03:23:46.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: 1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.c 2024-03-19T03:23:46.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: 2: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.b 2024-03-19T03:23:46.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: fsmap 2024-03-19T03:23:46.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T03:23:46.405 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: mgrmap e13: x(active, since 67s) 2024-03-19T03:23:46.405 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-19T03:23:46.405 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-19T03:23:46.405 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-19T03:23:46.405 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: mon.b (rank 2) addr [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] is down (out of quorum) 2024-03-19T03:23:46.405 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:46.405 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:46 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:46.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: mon.c calling monitor election 2024-03-19T03:23:46.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T03:23:46.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: mon.a calling monitor election 2024-03-19T03:23:46.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:23:46.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:23:46.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: from='client.? 172.21.15.178:0/823411457' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:23:46.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:23:46.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:46.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:23:46.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:23:46.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:46.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:23:46.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:23:46.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-19T03:23:46.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: monmap epoch 3 2024-03-19T03:23:46.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: fsid a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:23:46.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: last_changed 2024-03-19T03:23:41.075373+0000 2024-03-19T03:23:46.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: created 2024-03-19T03:21:26.225562+0000 2024-03-19T03:23:46.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: min_mon_release 19 (squid) 2024-03-19T03:23:46.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: election_strategy: 1 2024-03-19T03:23:46.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: 0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-03-19T03:23:46.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: 1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.c 2024-03-19T03:23:46.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: 2: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.b 2024-03-19T03:23:46.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: fsmap 2024-03-19T03:23:46.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T03:23:46.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: mgrmap e13: x(active, since 67s) 2024-03-19T03:23:46.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-19T03:23:46.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-19T03:23:46.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-19T03:23:46.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: mon.b (rank 2) addr [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] is down (out of quorum) 2024-03-19T03:23:46.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:46.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:46 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:46.681 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-19T03:23:46.681 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-19T03:23:47.120 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:23:47.430 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:47 smithi117 ceph-mon[31838]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:47.431 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:47 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:47.431 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:47 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:47.431 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:47 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:23:47.431 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:47 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:23:47.489 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:47 smithi178 ceph-mon[67578]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:47.489 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:47 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:47.489 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:47 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:47.489 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:47 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:23:47.489 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:47 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:23:48.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: mon.c calling monitor election 2024-03-19T03:23:48.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T03:23:48.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: mon.a calling monitor election 2024-03-19T03:23:48.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:23:48.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:23:48.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: from='client.? 172.21.15.178:0/823411457' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T03:23:48.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:23:48.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:48.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:23:48.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:23:48.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:48.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:23:48.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:23:48.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-19T03:23:48.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: monmap epoch 3 2024-03-19T03:23:48.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: fsid a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:23:48.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: last_changed 2024-03-19T03:23:41.075373+0000 2024-03-19T03:23:48.486 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: created 2024-03-19T03:21:26.225562+0000 2024-03-19T03:23:48.486 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: min_mon_release 19 (squid) 2024-03-19T03:23:48.486 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: election_strategy: 1 2024-03-19T03:23:48.486 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: 0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-03-19T03:23:48.486 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: 1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.c 2024-03-19T03:23:48.486 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: 2: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.b 2024-03-19T03:23:48.486 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: fsmap 2024-03-19T03:23:48.486 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T03:23:48.486 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: mgrmap e13: x(active, since 67s) 2024-03-19T03:23:48.486 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-19T03:23:48.486 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-19T03:23:48.486 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-19T03:23:48.486 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: mon.b (rank 2) addr [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] is down (out of quorum) 2024-03-19T03:23:48.486 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:48.486 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:48.487 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:48.487 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:48.487 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:48.487 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:23:48.487 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:23:48.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: mon.b calling monitor election 2024-03-19T03:23:48.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-19T03:23:48.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-19T03:23:48.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: Updating smithi178:/etc/ceph/ceph.conf 2024-03-19T03:23:48.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: Updating smithi119:/var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:23:48.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: mon.b calling monitor election 2024-03-19T03:23:48.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: mon.c calling monitor election 2024-03-19T03:23:48.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: mon.a calling monitor election 2024-03-19T03:23:48.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: Updating smithi178:/var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:23:48.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-19T03:23:48.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: monmap epoch 3 2024-03-19T03:23:48.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: fsid a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:23:48.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: last_changed 2024-03-19T03:23:41.075373+0000 2024-03-19T03:23:48.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: created 2024-03-19T03:21:26.225562+0000 2024-03-19T03:23:48.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: min_mon_release 19 (squid) 2024-03-19T03:23:48.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: election_strategy: 1 2024-03-19T03:23:48.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: 0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-03-19T03:23:48.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: 1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.c 2024-03-19T03:23:48.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: 2: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.b 2024-03-19T03:23:48.986 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: fsmap 2024-03-19T03:23:48.986 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T03:23:48.986 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: mgrmap e13: x(active, since 69s) 2024-03-19T03:23:48.986 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-19T03:23:48.986 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: Cluster is now healthy 2024-03-19T03:23:48.986 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: overall HEALTH_OK 2024-03-19T03:23:48.986 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:48.986 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:48.986 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:48.986 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:48.986 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:48.986 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:48 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:48.987 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: mon.b calling monitor election 2024-03-19T03:23:48.987 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-19T03:23:48.987 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-19T03:23:48.987 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: Updating smithi178:/etc/ceph/ceph.conf 2024-03-19T03:23:48.987 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: Updating smithi119:/var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:23:48.987 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: mon.b calling monitor election 2024-03-19T03:23:48.987 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: mon.c calling monitor election 2024-03-19T03:23:48.987 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: mon.a calling monitor election 2024-03-19T03:23:48.987 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: Updating smithi178:/var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:23:48.987 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-19T03:23:48.987 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: monmap epoch 3 2024-03-19T03:23:48.987 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: fsid a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:23:48.988 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: last_changed 2024-03-19T03:23:41.075373+0000 2024-03-19T03:23:48.988 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: created 2024-03-19T03:21:26.225562+0000 2024-03-19T03:23:48.988 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: min_mon_release 19 (squid) 2024-03-19T03:23:48.988 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: election_strategy: 1 2024-03-19T03:23:48.988 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: 0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-03-19T03:23:48.988 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: 1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.c 2024-03-19T03:23:48.988 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: 2: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.b 2024-03-19T03:23:48.988 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: fsmap 2024-03-19T03:23:48.988 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T03:23:48.988 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: mgrmap e13: x(active, since 69s) 2024-03-19T03:23:48.988 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-19T03:23:48.988 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: Cluster is now healthy 2024-03-19T03:23:48.988 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: overall HEALTH_OK 2024-03-19T03:23:48.988 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:48.988 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:48.989 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:48.989 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:48.989 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:48.989 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:48 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:49.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: mon.b calling monitor election 2024-03-19T03:23:49.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-19T03:23:49.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: Updating smithi119:/etc/ceph/ceph.conf 2024-03-19T03:23:49.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: Updating smithi178:/etc/ceph/ceph.conf 2024-03-19T03:23:49.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: Updating smithi119:/var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:23:49.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: mon.b calling monitor election 2024-03-19T03:23:49.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: mon.c calling monitor election 2024-03-19T03:23:49.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: mon.a calling monitor election 2024-03-19T03:23:49.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: Updating smithi178:/var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:23:49.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-19T03:23:49.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: monmap epoch 3 2024-03-19T03:23:49.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: fsid a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:23:49.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: last_changed 2024-03-19T03:23:41.075373+0000 2024-03-19T03:23:49.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: created 2024-03-19T03:21:26.225562+0000 2024-03-19T03:23:49.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: min_mon_release 19 (squid) 2024-03-19T03:23:49.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: election_strategy: 1 2024-03-19T03:23:49.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: 0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-03-19T03:23:49.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: 1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.c 2024-03-19T03:23:49.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: 2: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.b 2024-03-19T03:23:49.098 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: fsmap 2024-03-19T03:23:49.098 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T03:23:49.098 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: mgrmap e13: x(active, since 69s) 2024-03-19T03:23:49.098 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-19T03:23:49.098 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: Cluster is now healthy 2024-03-19T03:23:49.098 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: overall HEALTH_OK 2024-03-19T03:23:49.098 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:49.098 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:49.098 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:49.098 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:49.098 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:49.098 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:48 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:49.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.144+0000 7f59d0e00700 1 -- 172.21.15.117:0/3438419198 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59cc102ed0 msgr2=0x7f59cc1032b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:49.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.144+0000 7f59d0e00700 1 --2- 172.21.15.117:0/3438419198 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59cc102ed0 0x7f59cc1032b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f59bc009910 tx=0x7f59bc009c20 comp rx=0 tx=0).stop 2024-03-19T03:23:49.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.144+0000 7f59d0e00700 1 -- 172.21.15.117:0/3438419198 shutdown_connections 2024-03-19T03:23:49.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.144+0000 7f59d0e00700 1 --2- 172.21.15.117:0/3438419198 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59cc102ed0 0x7f59cc1032b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:49.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.144+0000 7f59d0e00700 1 -- 172.21.15.117:0/3438419198 >> 172.21.15.117:0/3438419198 conn(0x7f59cc0779b0 msgr2=0x7f59cc0fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:23:49.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.144+0000 7f59d0e00700 1 -- 172.21.15.117:0/3438419198 shutdown_connections 2024-03-19T03:23:49.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.144+0000 7f59d0e00700 1 -- 172.21.15.117:0/3438419198 wait complete. 2024-03-19T03:23:49.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.145+0000 7f59d0e00700 1 Processor -- start 2024-03-19T03:23:49.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.146+0000 7f59d0e00700 1 -- start start 2024-03-19T03:23:49.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.146+0000 7f59d0e00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59cc102ed0 0x7f59cc19fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:49.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.146+0000 7f59d0e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59cc19ff80 0x7f59cc199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:49.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.146+0000 7f59d0e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59cc19a050 0x7f59cc19a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:49.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.146+0000 7f59d0e00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f59cc10c370 con 0x7f59cc19ff80 2024-03-19T03:23:49.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.146+0000 7f59d0e00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f59cc10c1f0 con 0x7f59cc19a050 2024-03-19T03:23:49.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.146+0000 7f59d0e00700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f59cc10c4f0 con 0x7f59cc102ed0 2024-03-19T03:23:49.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.146+0000 7f59c9400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59cc19ff80 0x7f59cc199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:49.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.146+0000 7f59c9e00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59cc102ed0 0x7f59cc19fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:49.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.146+0000 7f59c9400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59cc19ff80 0x7f59cc199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:33968/0 (socket says 172.21.15.117:33968) 2024-03-19T03:23:49.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.146+0000 7f59c9400700 1 -- 172.21.15.117:0/3606831468 learned_addr learned my addr 172.21.15.117:0/3606831468 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:23:49.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.146+0000 7f59c9e00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59cc102ed0 0x7f59cc19fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:39448/0 (socket says 172.21.15.117:39448) 2024-03-19T03:23:49.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.147+0000 7f59ca800700 1 --2- 172.21.15.117:0/3606831468 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59cc19a050 0x7f59cc19a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:49.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.147+0000 7f59c9400700 1 -- 172.21.15.117:0/3606831468 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59cc102ed0 msgr2=0x7f59cc19fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:49.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.147+0000 7f59c9400700 1 --2- 172.21.15.117:0/3606831468 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59cc102ed0 0x7f59cc19fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:49.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.147+0000 7f59c9400700 1 -- 172.21.15.117:0/3606831468 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59cc19a050 msgr2=0x7f59cc19a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:49.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.147+0000 7f59c9400700 1 --2- 172.21.15.117:0/3606831468 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59cc19a050 0x7f59cc19a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:49.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.147+0000 7f59c9400700 1 -- 172.21.15.117:0/3606831468 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59cc1a67f0 con 0x7f59cc19ff80 2024-03-19T03:23:49.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.147+0000 7f59c9400700 1 --2- 172.21.15.117:0/3606831468 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59cc19ff80 0x7f59cc199b10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f59b800b780 tx=0x7f59b800ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:23:49.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.148+0000 7f59c2a00700 1 -- 172.21.15.117:0/3606831468 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b8016070 con 0x7f59cc19ff80 2024-03-19T03:23:49.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.148+0000 7f59c2a00700 1 -- 172.21.15.117:0/3606831468 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59b80036a0 con 0x7f59cc19ff80 2024-03-19T03:23:49.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.148+0000 7f59c2a00700 1 -- 172.21.15.117:0/3606831468 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b8011af0 con 0x7f59cc19ff80 2024-03-19T03:23:49.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.148+0000 7f59d0e00700 1 -- 172.21.15.117:0/3606831468 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59cc1a6a80 con 0x7f59cc19ff80 2024-03-19T03:23:49.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.148+0000 7f59d0e00700 1 -- 172.21.15.117:0/3606831468 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59cc1a6f70 con 0x7f59cc19ff80 2024-03-19T03:23:49.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.149+0000 7f59d0e00700 1 -- 172.21.15.117:0/3606831468 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59cc069fa0 con 0x7f59cc19ff80 2024-03-19T03:23:49.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.149+0000 7f59c2a00700 1 -- 172.21.15.117:0/3606831468 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f59b8019070 con 0x7f59cc19ff80 2024-03-19T03:23:49.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.150+0000 7f59c2a00700 1 --2- 172.21.15.117:0/3606831468 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f59b0039450 0x7f59b003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:49.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.150+0000 7f59c2a00700 1 -- 172.21.15.117:0/3606831468 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f59b804ac50 con 0x7f59cc19ff80 2024-03-19T03:23:49.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.150+0000 7f59c9e00700 1 --2- 172.21.15.117:0/3606831468 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f59b0039450 0x7f59b003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:49.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.151+0000 7f59c9e00700 1 --2- 172.21.15.117:0/3606831468 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f59b0039450 0x7f59b003b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f59bc006010 tx=0x7f59bc0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:23:49.157 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.156+0000 7f59c2a00700 1 -- 172.21.15.117:0/3606831468 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f59b8010380 con 0x7f59cc19ff80 2024-03-19T03:23:49.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.406+0000 7f59d0e00700 1 -- 172.21.15.117:0/3606831468 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f59cc1a72a0 con 0x7f59cc19ff80 2024-03-19T03:23:49.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.406+0000 7f59c2a00700 1 -- 172.21.15.117:0/3606831468 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+277 (secure 0 0 0) 0x7f59b8014120 con 0x7f59cc19ff80 2024-03-19T03:23:49.407 INFO:teuthology.orchestra.run.smithi117.stdout:# minimal ceph.conf for a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:23:49.407 INFO:teuthology.orchestra.run.smithi117.stdout:[global] 2024-03-19T03:23:49.408 INFO:teuthology.orchestra.run.smithi117.stdout: fsid = a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:23:49.408 INFO:teuthology.orchestra.run.smithi117.stdout: mon_host = [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2024-03-19T03:23:49.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.409+0000 7f59d0e00700 1 -- 172.21.15.117:0/3606831468 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f59b0039450 msgr2=0x7f59b003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:49.411 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.409+0000 7f59d0e00700 1 --2- 172.21.15.117:0/3606831468 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f59b0039450 0x7f59b003b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f59bc006010 tx=0x7f59bc0058e0 comp rx=0 tx=0).stop 2024-03-19T03:23:49.411 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.409+0000 7f59d0e00700 1 -- 172.21.15.117:0/3606831468 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59cc19ff80 msgr2=0x7f59cc199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:49.411 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.409+0000 7f59d0e00700 1 --2- 172.21.15.117:0/3606831468 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59cc19ff80 0x7f59cc199b10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f59b800b780 tx=0x7f59b800ba90 comp rx=0 tx=0).stop 2024-03-19T03:23:49.411 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.410+0000 7f59d0e00700 1 -- 172.21.15.117:0/3606831468 shutdown_connections 2024-03-19T03:23:49.411 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.410+0000 7f59d0e00700 1 --2- 172.21.15.117:0/3606831468 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f59b0039450 0x7f59b003b910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:49.411 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.410+0000 7f59d0e00700 1 --2- 172.21.15.117:0/3606831468 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59cc19a050 0x7f59cc19a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:49.411 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.410+0000 7f59d0e00700 1 --2- 172.21.15.117:0/3606831468 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59cc102ed0 0x7f59cc19fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:49.411 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.410+0000 7f59d0e00700 1 --2- 172.21.15.117:0/3606831468 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59cc19ff80 0x7f59cc199b10 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:49.411 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.410+0000 7f59d0e00700 1 -- 172.21.15.117:0/3606831468 >> 172.21.15.117:0/3606831468 conn(0x7f59cc0779b0 msgr2=0x7f59cc1080f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:23:49.411 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.410+0000 7f59d0e00700 1 -- 172.21.15.117:0/3606831468 shutdown_connections 2024-03-19T03:23:49.411 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:23:49.410+0000 7f59d0e00700 1 -- 172.21.15.117:0/3606831468 wait complete. 2024-03-19T03:23:50.035 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:50 smithi117 ceph-mon[31838]: Updating smithi117:/var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:23:50.035 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:50 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:50.036 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:50 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:50.036 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:50 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:50.036 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:50 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:23:50.036 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:50 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:50.036 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:50 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:50.036 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:50 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:50.036 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:50 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:50.036 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:50 smithi117 ceph-mon[31838]: Reconfiguring mon.a (unknown last config time)... 2024-03-19T03:23:50.036 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:50 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T03:23:50.036 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:50 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T03:23:50.036 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:50 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:50.036 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:50 smithi117 ceph-mon[31838]: Reconfiguring daemon mon.a on smithi117 2024-03-19T03:23:50.037 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:50 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3606831468' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:50.176 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-19T03:23:50.176 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-19T03:23:50.176 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/ceph.conf 2024-03-19T03:23:50.261 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-19T03:23:50.262 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-19T03:23:50.291 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:23:50 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:23:50.076+0000 7f256fc00700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-19T03:23:50.329 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-19T03:23:50.329 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.conf 2024-03-19T03:23:50.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:50 smithi178 ceph-mon[67578]: Updating smithi117:/var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:23:50.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:50 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:50.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:50 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:50.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:50 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:50.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:50 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:23:50.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:50 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:50.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:50 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:50.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:50 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:50.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:50 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:50.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:50 smithi178 ceph-mon[67578]: Reconfiguring mon.a (unknown last config time)... 2024-03-19T03:23:50.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:50 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T03:23:50.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:50 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T03:23:50.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:50 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:50.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:50 smithi178 ceph-mon[67578]: Reconfiguring daemon mon.a on smithi117 2024-03-19T03:23:50.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:50 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3606831468' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:50.358 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:50 smithi119 ceph-mon[66861]: Updating smithi117:/var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T03:23:50.358 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:50 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:50.359 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:50 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:50.359 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:50 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:50.359 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:50 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:23:50.359 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:50 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:50.359 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:50 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:50.359 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:50 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:50.359 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:50 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:50.359 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:50 smithi119 ceph-mon[66861]: Reconfiguring mon.a (unknown last config time)... 2024-03-19T03:23:50.359 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:50 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T03:23:50.359 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:50 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T03:23:50.359 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:50 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:50.359 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:50 smithi119 ceph-mon[66861]: Reconfiguring daemon mon.a on smithi117 2024-03-19T03:23:50.359 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:50 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3606831468' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:50.363 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-19T03:23:50.363 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-19T03:23:50.434 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-19T03:23:50.434 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/ceph.conf 2024-03-19T03:23:50.465 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-19T03:23:50.465 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-19T03:23:50.532 INFO:tasks.cephadm:Adding mgr.x on smithi117 2024-03-19T03:23:50.532 INFO:tasks.cephadm:Adding mgr.y on smithi119 2024-03-19T03:23:50.532 INFO:tasks.cephadm:Adding mgr.z on smithi178 2024-03-19T03:23:50.532 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph orch apply mgr '3;smithi117=x;smithi119=y;smithi178=z' 2024-03-19T03:23:50.810 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T03:23:51.210 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:51 smithi178 ceph-mon[67578]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:51.291 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:51 smithi117 ceph-mon[31838]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:51.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:51 smithi119 ceph-mon[66861]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:52.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.229+0000 7f71efc67700 1 -- 172.21.15.178:0/4258574694 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71e8100d70 msgr2=0x7f71e8101190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:52.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.229+0000 7f71efc67700 1 --2- 172.21.15.178:0/4258574694 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71e8100d70 0x7f71e8101190 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f71e4009910 tx=0x7f71e4009c20 comp rx=0 tx=0).stop 2024-03-19T03:23:52.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.229+0000 7f71efc67700 1 -- 172.21.15.178:0/4258574694 shutdown_connections 2024-03-19T03:23:52.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.229+0000 7f71efc67700 1 --2- 172.21.15.178:0/4258574694 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71e8100d70 0x7f71e8101190 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:52.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.229+0000 7f71efc67700 1 -- 172.21.15.178:0/4258574694 >> 172.21.15.178:0/4258574694 conn(0x7f71e80fc2f0 msgr2=0x7f71e80fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:23:52.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.229+0000 7f71efc67700 1 -- 172.21.15.178:0/4258574694 shutdown_connections 2024-03-19T03:23:52.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.229+0000 7f71efc67700 1 -- 172.21.15.178:0/4258574694 wait complete. 2024-03-19T03:23:52.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.230+0000 7f71efc67700 1 Processor -- start 2024-03-19T03:23:52.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.230+0000 7f71efc67700 1 -- start start 2024-03-19T03:23:52.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.230+0000 7f71efc67700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71e8100d70 0x7f71e8196560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:52.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.230+0000 7f71efc67700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f71e8196aa0 0x7f71e819bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:52.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.230+0000 7f71efc67700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71e8196f20 0x7f71e819e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:52.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.230+0000 7f71efc67700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f71e8106e00 con 0x7f71e8196f20 2024-03-19T03:23:52.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.230+0000 7f71efc67700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f71e8106c80 con 0x7f71e8100d70 2024-03-19T03:23:52.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.230+0000 7f71efc67700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f71e8106f80 con 0x7f71e8196aa0 2024-03-19T03:23:52.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.230+0000 7f71eda03700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71e8100d70 0x7f71e8196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:52.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.230+0000 7f71ed202700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f71e8196aa0 0x7f71e819bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:52.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.230+0000 7f71eda03700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71e8100d70 0x7f71e8196560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.178:41902/0 (socket says 172.21.15.178:41902) 2024-03-19T03:23:52.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.230+0000 7f71eda03700 1 -- 172.21.15.178:0/3568688076 learned_addr learned my addr 172.21.15.178:0/3568688076 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-19T03:23:52.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.230+0000 7f71ee204700 1 --2- 172.21.15.178:0/3568688076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71e8196f20 0x7f71e819e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:52.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.231+0000 7f71ed202700 1 -- 172.21.15.178:0/3568688076 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71e8100d70 msgr2=0x7f71e8196560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:52.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.231+0000 7f71ed202700 1 --2- 172.21.15.178:0/3568688076 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71e8100d70 0x7f71e8196560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:52.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.231+0000 7f71ed202700 1 -- 172.21.15.178:0/3568688076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71e8196f20 msgr2=0x7f71e819e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:52.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.231+0000 7f71ed202700 1 --2- 172.21.15.178:0/3568688076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71e8196f20 0x7f71e819e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:52.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.231+0000 7f71ed202700 1 -- 172.21.15.178:0/3568688076 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71e819e970 con 0x7f71e8196aa0 2024-03-19T03:23:52.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.231+0000 7f71eda03700 1 --2- 172.21.15.178:0/3568688076 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71e8100d70 0x7f71e8196560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:23:52.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.231+0000 7f71ed202700 1 --2- 172.21.15.178:0/3568688076 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f71e8196aa0 0x7f71e819bb10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f71d800da00 tx=0x7f71d800dd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:23:52.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.231+0000 7f71deffd700 1 -- 172.21.15.178:0/3568688076 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71d800ec50 con 0x7f71e8196aa0 2024-03-19T03:23:52.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.231+0000 7f71deffd700 1 -- 172.21.15.178:0/3568688076 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71d8004510 con 0x7f71e8196aa0 2024-03-19T03:23:52.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.231+0000 7f71deffd700 1 -- 172.21.15.178:0/3568688076 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71d8016430 con 0x7f71e8196aa0 2024-03-19T03:23:52.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.231+0000 7f71efc67700 1 -- 172.21.15.178:0/3568688076 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71e806a550 con 0x7f71e8196aa0 2024-03-19T03:23:52.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.231+0000 7f71efc67700 1 -- 172.21.15.178:0/3568688076 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71e806ab10 con 0x7f71e8196aa0 2024-03-19T03:23:52.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.232+0000 7f71deffd700 1 -- 172.21.15.178:0/3568688076 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f71d80165f0 con 0x7f71e8196aa0 2024-03-19T03:23:52.235 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.232+0000 7f71deffd700 1 --2- 172.21.15.178:0/3568688076 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f71d4039450 0x7f71d403b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:23:52.235 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.232+0000 7f71efc67700 1 -- 172.21.15.178:0/3568688076 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71e8063940 con 0x7f71e8196aa0 2024-03-19T03:23:52.235 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.232+0000 7f71deffd700 1 -- 172.21.15.178:0/3568688076 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f71d804daa0 con 0x7f71e8196aa0 2024-03-19T03:23:52.235 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.233+0000 7f71eda03700 1 --2- 172.21.15.178:0/3568688076 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f71d4039450 0x7f71d403b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:23:52.235 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.233+0000 7f71eda03700 1 --2- 172.21.15.178:0/3568688076 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f71d4039450 0x7f71d403b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f71e4006010 tx=0x7f71e401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:23:52.237 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.237+0000 7f71deffd700 1 -- 172.21.15.178:0/3568688076 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f71d801c270 con 0x7f71e8196aa0 2024-03-19T03:23:52.440 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.440+0000 7f71efc67700 1 -- 172.21.15.178:0/3568688076 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi117=x;smithi119=y;smithi178=z", "target": ["mon-mgr", ""]}) -- 0x7f71e806ad70 con 0x7f71d4039450 2024-03-19T03:23:52.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.453+0000 7f71deffd700 1 -- 172.21.15.178:0/3568688076 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f71e806ad70 con 0x7f71d4039450 2024-03-19T03:23:52.454 INFO:teuthology.orchestra.run.smithi178.stdout:Scheduled mgr update... 2024-03-19T03:23:52.456 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.455+0000 7f71efc67700 1 -- 172.21.15.178:0/3568688076 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f71d4039450 msgr2=0x7f71d403b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:52.456 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.455+0000 7f71efc67700 1 --2- 172.21.15.178:0/3568688076 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f71d4039450 0x7f71d403b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f71e4006010 tx=0x7f71e401a040 comp rx=0 tx=0).stop 2024-03-19T03:23:52.456 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.455+0000 7f71efc67700 1 -- 172.21.15.178:0/3568688076 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f71e8196aa0 msgr2=0x7f71e819bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:23:52.456 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.455+0000 7f71efc67700 1 --2- 172.21.15.178:0/3568688076 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f71e8196aa0 0x7f71e819bb10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f71d800da00 tx=0x7f71d800dd10 comp rx=0 tx=0).stop 2024-03-19T03:23:52.456 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.455+0000 7f71efc67700 1 -- 172.21.15.178:0/3568688076 shutdown_connections 2024-03-19T03:23:52.456 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.455+0000 7f71efc67700 1 --2- 172.21.15.178:0/3568688076 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f71d4039450 0x7f71d403b910 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:52.457 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.455+0000 7f71efc67700 1 --2- 172.21.15.178:0/3568688076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71e8196f20 0x7f71e819e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:52.457 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.455+0000 7f71efc67700 1 --2- 172.21.15.178:0/3568688076 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71e8100d70 0x7f71e8196560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:52.457 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.456+0000 7f71efc67700 1 --2- 172.21.15.178:0/3568688076 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f71e8196aa0 0x7f71e819bb10 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:23:52.457 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.456+0000 7f71efc67700 1 -- 172.21.15.178:0/3568688076 >> 172.21.15.178:0/3568688076 conn(0x7f71e80fc2f0 msgr2=0x7f71e80fcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:23:52.457 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.456+0000 7f71efc67700 1 -- 172.21.15.178:0/3568688076 shutdown_connections 2024-03-19T03:23:52.457 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:23:52.456+0000 7f71efc67700 1 -- 172.21.15.178:0/3568688076 wait complete. 2024-03-19T03:23:52.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:52 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:52.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:52 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:52.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:52 smithi178 ceph-mon[67578]: Reconfiguring mon.b (monmap changed)... 2024-03-19T03:23:52.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:52 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T03:23:52.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:52 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T03:23:52.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:52 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:52.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:52 smithi178 ceph-mon[67578]: Reconfiguring daemon mon.b on smithi119 2024-03-19T03:23:52.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:52 smithi178 ceph-mon[67578]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:52.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:52 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:52.956 DEBUG:teuthology.orchestra.run.smithi119:mgr.y> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mgr.y.service 2024-03-19T03:23:52.958 DEBUG:teuthology.orchestra.run.smithi178:mgr.z> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mgr.z.service 2024-03-19T03:23:52.961 INFO:tasks.cephadm:Deploying OSDs... 2024-03-19T03:23:52.961 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-19T03:23:52.961 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/scratch_devs of=/dev/stdout 2024-03-19T03:23:52.980 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:52 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:52.981 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:52 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:52.981 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:52 smithi119 ceph-mon[66861]: Reconfiguring mon.b (monmap changed)... 2024-03-19T03:23:52.981 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:52 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T03:23:52.981 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:52 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T03:23:52.981 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:52 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:52.981 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:52 smithi119 ceph-mon[66861]: Reconfiguring daemon mon.b on smithi119 2024-03-19T03:23:52.981 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:52 smithi119 ceph-mon[66861]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:52.981 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:52 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:53.018 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-19T03:23:53.019 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_1 2024-03-19T03:23:53.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:52 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:53.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:52 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:53.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:52 smithi117 ceph-mon[31838]: Reconfiguring mon.b (monmap changed)... 2024-03-19T03:23:53.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:52 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T03:23:53.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:52 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T03:23:53.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:52 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:53.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:52 smithi117 ceph-mon[31838]: Reconfiguring daemon mon.b on smithi119 2024-03-19T03:23:53.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:52 smithi117 ceph-mon[31838]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:53.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:52 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:53.070 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-19T03:23:53.070 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T03:23:53.071 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 634 Links: 1 2024-03-19T03:23:53.071 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T03:23:53.071 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T03:23:53.071 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-19 03:22:54.338767878 +0000 2024-03-19T03:23:53.071 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-19 03:03:18.684727937 +0000 2024-03-19T03:23:53.071 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-19 03:03:18.684727937 +0000 2024-03-19T03:23:53.071 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-19 03:03:18.684727937 +0000 2024-03-19T03:23:53.071 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-19T03:23:53.205 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-19T03:23:53.205 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-19T03:23:53.205 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000341187 s, 1.5 MB/s 2024-03-19T03:23:53.209 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-19T03:23:53.264 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_2 2024-03-19T03:23:53.311 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-19T03:23:53.311 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T03:23:53.311 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 651 Links: 1 2024-03-19T03:23:53.311 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T03:23:53.311 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T03:23:53.311 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-19 03:22:54.385766935 +0000 2024-03-19T03:23:53.311 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-19 03:03:19.244716801 +0000 2024-03-19T03:23:53.311 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-19 03:03:19.244716801 +0000 2024-03-19T03:23:53.311 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-19 03:03:19.244716801 +0000 2024-03-19T03:23:53.311 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-19T03:23:53.392 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-19T03:23:53.392 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-19T03:23:53.392 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.00036819 s, 1.4 MB/s 2024-03-19T03:23:53.396 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-19T03:23:53.451 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_3 2024-03-19T03:23:53.499 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-19T03:23:53.499 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T03:23:53.499 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 669 Links: 1 2024-03-19T03:23:53.499 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T03:23:53.500 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T03:23:53.500 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-19 03:22:54.440765832 +0000 2024-03-19T03:23:53.500 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-19 03:03:19.791705922 +0000 2024-03-19T03:23:53.500 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-19 03:03:19.791705922 +0000 2024-03-19T03:23:53.500 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-19 03:03:19.791705922 +0000 2024-03-19T03:23:53.500 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-19T03:23:53.585 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-19T03:23:53.585 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-19T03:23:53.585 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000325638 s, 1.6 MB/s 2024-03-19T03:23:53.589 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-19T03:23:53.645 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_4 2024-03-19T03:23:53.694 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-19T03:23:53.695 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T03:23:53.695 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 691 Links: 1 2024-03-19T03:23:53.695 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T03:23:53.695 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T03:23:53.695 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-19 03:22:54.499764649 +0000 2024-03-19T03:23:53.695 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-19 03:03:20.374694328 +0000 2024-03-19T03:23:53.695 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-19 03:03:20.374694328 +0000 2024-03-19T03:23:53.695 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-19 03:03:20.374694328 +0000 2024-03-19T03:23:53.695 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-19T03:23:53.782 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-19T03:23:53.782 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-19T03:23:53.782 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000344095 s, 1.5 MB/s 2024-03-19T03:23:53.787 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-19T03:23:53.840 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-19T03:23:53.840 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/scratch_devs of=/dev/stdout 2024-03-19T03:23:53.859 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-19T03:23:53.859 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_1 2024-03-19T03:23:53.916 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-19T03:23:53.916 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T03:23:53.916 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 660 Links: 1 2024-03-19T03:23:53.916 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T03:23:53.917 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T03:23:53.917 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-03-19 03:23:40.554307986 +0000 2024-03-19T03:23:53.917 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-03-19 03:02:12.969471101 +0000 2024-03-19T03:23:53.917 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-03-19 03:02:12.969471101 +0000 2024-03-19T03:23:53.917 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-03-19 03:02:12.969471101 +0000 2024-03-19T03:23:53.917 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-19T03:23:53.986 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-03-19T03:23:53.986 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-03-19T03:23:53.986 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000223255 s, 2.3 MB/s 2024-03-19T03:23:53.987 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-19T03:23:54.044 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_2 2024-03-19T03:23:54.100 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-19T03:23:54.100 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T03:23:54.100 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-03-19T03:23:54.100 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T03:23:54.100 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T03:23:54.100 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-03-19 03:23:40.554307986 +0000 2024-03-19T03:23:54.100 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-03-19 03:02:13.328461775 +0000 2024-03-19T03:23:54.100 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-03-19 03:02:13.328461775 +0000 2024-03-19T03:23:54.101 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-03-19 03:02:13.328461775 +0000 2024-03-19T03:23:54.101 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-19T03:23:54.170 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-03-19T03:23:54.170 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-03-19T03:23:54.170 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000247805 s, 2.1 MB/s 2024-03-19T03:23:54.171 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-19T03:23:54.227 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_3 2024-03-19T03:23:54.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:54 smithi119 ceph-mon[66861]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi117=x;smithi119=y;smithi178=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:23:54.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:54 smithi119 ceph-mon[66861]: Saving service mgr spec with placement smithi117=x;smithi119=y;smithi178=z;count:3 2024-03-19T03:23:54.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:54 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:54.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:54 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:54.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:54 smithi119 ceph-mon[66861]: Reconfiguring mon.c (monmap changed)... 2024-03-19T03:23:54.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:54 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T03:23:54.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:54 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T03:23:54.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:54 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:54.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:54 smithi119 ceph-mon[66861]: Reconfiguring daemon mon.c on smithi178 2024-03-19T03:23:54.249 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-19T03:23:54.249 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T03:23:54.249 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 699 Links: 1 2024-03-19T03:23:54.249 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T03:23:54.249 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T03:23:54.249 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-03-19 03:23:40.554307986 +0000 2024-03-19T03:23:54.249 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-03-19 03:02:13.663453073 +0000 2024-03-19T03:23:54.249 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-03-19 03:02:13.663453073 +0000 2024-03-19T03:23:54.250 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-03-19 03:02:13.663453073 +0000 2024-03-19T03:23:54.250 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-19T03:23:54.316 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-03-19T03:23:54.316 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-03-19T03:23:54.316 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000321884 s, 1.6 MB/s 2024-03-19T03:23:54.317 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-19T03:23:54.327 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:54 smithi178 ceph-mon[67578]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi117=x;smithi119=y;smithi178=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:23:54.327 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:54 smithi178 ceph-mon[67578]: Saving service mgr spec with placement smithi117=x;smithi119=y;smithi178=z;count:3 2024-03-19T03:23:54.327 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:54 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:54.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:54 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:54.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:54 smithi178 ceph-mon[67578]: Reconfiguring mon.c (monmap changed)... 2024-03-19T03:23:54.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:54 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T03:23:54.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:54 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T03:23:54.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:54 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:54.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:54 smithi178 ceph-mon[67578]: Reconfiguring daemon mon.c on smithi178 2024-03-19T03:23:54.374 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_4 2024-03-19T03:23:54.429 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-19T03:23:54.429 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T03:23:54.430 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 717 Links: 1 2024-03-19T03:23:54.430 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T03:23:54.430 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T03:23:54.430 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-03-19 03:23:40.555307960 +0000 2024-03-19T03:23:54.430 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-03-19 03:02:13.993444502 +0000 2024-03-19T03:23:54.430 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-03-19 03:02:13.993444502 +0000 2024-03-19T03:23:54.430 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-03-19 03:02:13.993444502 +0000 2024-03-19T03:23:54.430 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-19T03:23:54.497 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-03-19T03:23:54.497 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-03-19T03:23:54.497 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000242165 s, 2.1 MB/s 2024-03-19T03:23:54.498 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-19T03:23:54.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:54 smithi117 ceph-mon[31838]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi117=x;smithi119=y;smithi178=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:23:54.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:54 smithi117 ceph-mon[31838]: Saving service mgr spec with placement smithi117=x;smithi119=y;smithi178=z;count:3 2024-03-19T03:23:54.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:54 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:54.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:54 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:54.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:54 smithi117 ceph-mon[31838]: Reconfiguring mon.c (monmap changed)... 2024-03-19T03:23:54.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:54 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T03:23:54.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:54 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T03:23:54.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:54 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:54.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:54 smithi117 ceph-mon[31838]: Reconfiguring daemon mon.c on smithi178 2024-03-19T03:23:54.554 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-19T03:23:54.555 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/scratch_devs of=/dev/stdout 2024-03-19T03:23:54.571 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-19T03:23:54.571 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_1 2024-03-19T03:23:54.627 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-19T03:23:54.627 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T03:23:54.627 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-03-19T03:23:54.627 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T03:23:54.627 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T03:23:54.627 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-03-19 03:23:31.194966574 +0000 2024-03-19T03:23:54.627 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-03-19 03:04:14.317539157 +0000 2024-03-19T03:23:54.627 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-03-19 03:04:14.317539157 +0000 2024-03-19T03:23:54.627 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2024-03-19 03:04:14.317539157 +0000 2024-03-19T03:23:54.628 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-19T03:23:54.694 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-03-19T03:23:54.695 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-03-19T03:23:54.695 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 8.7535e-05 s, 5.8 MB/s 2024-03-19T03:23:54.696 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-19T03:23:54.752 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_2 2024-03-19T03:23:54.807 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-19T03:23:54.807 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T03:23:54.807 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 679 Links: 1 2024-03-19T03:23:54.807 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T03:23:54.807 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T03:23:54.807 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-03-19 03:23:31.196966534 +0000 2024-03-19T03:23:54.807 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-03-19 03:04:14.645531592 +0000 2024-03-19T03:23:54.807 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-03-19 03:04:14.645531592 +0000 2024-03-19T03:23:54.807 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2024-03-19 03:04:14.645531592 +0000 2024-03-19T03:23:54.808 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-19T03:23:54.872 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-03-19T03:23:54.872 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-03-19T03:23:54.872 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 9.3402e-05 s, 5.5 MB/s 2024-03-19T03:23:54.873 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-19T03:23:54.929 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_3 2024-03-19T03:23:54.984 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-19T03:23:54.984 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T03:23:54.984 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 698 Links: 1 2024-03-19T03:23:54.984 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T03:23:54.984 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T03:23:54.984 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-03-19 03:23:31.196966534 +0000 2024-03-19T03:23:54.984 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-03-19 03:04:14.975523982 +0000 2024-03-19T03:23:54.985 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-03-19 03:04:14.975523982 +0000 2024-03-19T03:23:54.985 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2024-03-19 03:04:14.975523982 +0000 2024-03-19T03:23:54.985 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-19T03:23:55.049 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-03-19T03:23:55.050 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-03-19T03:23:55.050 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000111567 s, 4.6 MB/s 2024-03-19T03:23:55.051 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-19T03:23:55.106 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_4 2024-03-19T03:23:55.161 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-19T03:23:55.162 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T03:23:55.162 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 718 Links: 1 2024-03-19T03:23:55.162 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T03:23:55.162 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T03:23:55.162 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-03-19 03:23:31.197966513 +0000 2024-03-19T03:23:55.162 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-03-19 03:04:15.288516763 +0000 2024-03-19T03:23:55.162 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-03-19 03:04:15.288516763 +0000 2024-03-19T03:23:55.162 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2024-03-19 03:04:15.288516763 +0000 2024-03-19T03:23:55.162 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-19T03:23:55.228 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-03-19T03:23:55.228 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-03-19T03:23:55.228 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 9.8858e-05 s, 5.2 MB/s 2024-03-19T03:23:55.230 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-19T03:23:55.285 INFO:tasks.cephadm:Deploying osd.0 on smithi117 with /dev/vg_nvme/lv_4... 2024-03-19T03:23:55.286 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-19T03:23:55.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:55 smithi178 ceph-mon[67578]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:55.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:55.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:55.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:23:55.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:55.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:23:55.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:55.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T03:23:55.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T03:23:55.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T03:23:55.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:55.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:55 smithi119 ceph-mon[66861]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:55.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:55.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:55.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:23:55.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:55.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:23:55.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:55.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T03:23:55.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T03:23:55.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T03:23:55.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:55.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:55 smithi117 ceph-mon[31838]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:55.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:55.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:55.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:23:55.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:55.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:23:55.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:55.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T03:23:55.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T03:23:55.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T03:23:55.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:55.723 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:23:56.384 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:56 smithi178 ceph-mon[67578]: Deploying daemon mgr.z on smithi178 2024-03-19T03:23:56.483 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:56 smithi119 ceph-mon[66861]: Deploying daemon mgr.z on smithi178 2024-03-19T03:23:56.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:56 smithi117 ceph-mon[31838]: Deploying daemon mgr.z on smithi178 2024-03-19T03:23:56.762 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:23:56 smithi178 systemd[1]: Starting Ceph mgr.z for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:23:57.023 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:23:56 smithi178 podman[69434]: 2024-03-19 03:23:56.7623646 +0000 UTC m=+0.109662696 container create 7a6e85b68b39d5b03c427798b0afc852461803f5b1b32ddfb714e053b96b2018 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-19T03:23:57.023 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:23:56 smithi178 podman[69434]: 2024-03-19 03:23:56.670153069 +0000 UTC m=+0.017451165 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:23:57.023 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:23:56 smithi178 podman[69434]: 2024-03-19 03:23:56.947946598 +0000 UTC m=+0.295244670 container init 7a6e85b68b39d5b03c427798b0afc852461803f5b1b32ddfb714e053b96b2018 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-19T03:23:57.023 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:23:56 smithi178 podman[69434]: 2024-03-19 03:23:56.950496406 +0000 UTC m=+0.297794475 container start 7a6e85b68b39d5b03c427798b0afc852461803f5b1b32ddfb714e053b96b2018 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T03:23:57.023 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:23:56 smithi178 ceph-mgr[69453]: -- 172.21.15.178:0/89290415 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5604ead345a0 con 0x5604ead09000 2024-03-19T03:23:57.023 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:23:57 smithi178 bash[69434]: 7a6e85b68b39d5b03c427798b0afc852461803f5b1b32ddfb714e053b96b2018 2024-03-19T03:23:57.276 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:57 smithi117 ceph-mon[31838]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:57.277 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:57 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:57.277 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:57 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:57.345 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:23:57 smithi178 systemd[1]: Started Ceph mgr.z for a3a82456-e59f-11ee-95c9-87774f69a715. 2024-03-19T03:23:57.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:57 smithi178 ceph-mon[67578]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:57.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:57 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:57.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:57 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:57.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:57 smithi119 ceph-mon[66861]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:57.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:57 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:57.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:57 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:57.845 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:23:57 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z[69449]: 2024-03-19T03:23:57.488+0000 7f418afd8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T03:23:57.846 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:23:57 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z[69449]: 2024-03-19T03:23:57.579+0000 7f418afd8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T03:23:58.405 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-19T03:23:58.406 INFO:teuthology.orchestra.run.smithi117.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-19T03:23:58.406 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10+0 records in 2024-03-19T03:23:58.406 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2024-03-19T03:23:58.406 INFO:teuthology.orchestra.run.smithi117.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0369772 s, 284 MB/s 2024-03-19T03:23:58.406 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping successful for: 2024-03-19T03:23:58.446 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:58 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:58.446 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:58 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T03:23:58.446 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:58 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T03:23:58.447 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:58 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T03:23:58.447 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:58 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:58.447 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:58 smithi178 ceph-mon[67578]: Deploying daemon mgr.y on smithi119 2024-03-19T03:23:58.447 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:23:58 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z[69449]: 2024-03-19T03:23:58.215+0000 7f418afd8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T03:23:58.447 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:23:58 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z[69449]: 2024-03-19T03:23:58.305+0000 7f418afd8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T03:23:58.447 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:23:58 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z[69449]: 2024-03-19T03:23:58.445+0000 7f418afd8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T03:23:58.447 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:58 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:58.448 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:58 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T03:23:58.448 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:58 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T03:23:58.448 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:58 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T03:23:58.448 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:58 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:58.448 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:58 smithi117 ceph-mon[31838]: Deploying daemon mgr.y on smithi119 2024-03-19T03:23:58.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:58 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:23:58.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:58 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T03:23:58.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:58 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T03:23:58.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:58 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T03:23:58.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:58 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:23:58.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:58 smithi119 ceph-mon[66861]: Deploying daemon mgr.y on smithi119 2024-03-19T03:23:58.845 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:23:58 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z[69449]: 2024-03-19T03:23:58.539+0000 7f418afd8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T03:23:58.846 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:23:58 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z[69449]: 2024-03-19T03:23:58.620+0000 7f418afd8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T03:23:59.125 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi117:vg_nvme/lv_4 2024-03-19T03:23:59.162 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:23:58 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z[69449]: 2024-03-19T03:23:58.880+0000 7f418afd8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T03:23:59.162 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:23:58 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z[69449]: 2024-03-19T03:23:58.986+0000 7f418afd8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T03:23:59.162 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:23:59 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z[69449]: 2024-03-19T03:23:59.069+0000 7f418afd8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T03:23:59.162 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:23:59 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z[69449]: 2024-03-19T03:23:59.161+0000 7f418afd8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T03:23:59.374 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:23:59 smithi119 ceph-mon[66861]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:59.440 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:23:59 smithi117 ceph-mon[31838]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:59.555 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:23:59.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:23:59 smithi178 ceph-mon[67578]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:23:59.596 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:23:59 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z[69449]: 2024-03-19T03:23:59.337+0000 7f418afd8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T03:23:59.655 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:23:59 smithi119 ceph-mgr[68256]: -- 172.21.15.119:0/1450160067 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5632e65fa5a0 con 0x5632e65ce800 2024-03-19T03:23:59.655 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:23:59 smithi119 bash[68237]: ed517e272106f9bd858316f92244da5659b60bb47255272a7873365ce93a0ff1 2024-03-19T03:23:59.984 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:23:59 smithi119 systemd[1]: Started Ceph mgr.y for a3a82456-e59f-11ee-95c9-87774f69a715. 2024-03-19T03:24:00.484 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:24:00 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y[68252]: 2024-03-19T03:24:00.130+0000 7fccbed5f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T03:24:00.484 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:24:00 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y[68252]: 2024-03-19T03:24:00.227+0000 7fccbed5f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T03:24:00.595 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:24:00 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z[69449]: 2024-03-19T03:24:00.313+0000 7f418afd8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T03:24:00.596 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:24:00 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z[69449]: 2024-03-19T03:24:00.447+0000 7f418afd8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T03:24:00.780 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:00 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:00.780 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:00 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:00.780 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:00 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:00.780 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:00 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:00.780 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:00 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:24:00.780 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:00 smithi117 ceph-mon[31838]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:00.984 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:24:00 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y[68252]: 2024-03-19T03:24:00.901+0000 7fccbed5f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T03:24:00.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:00 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:00.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:00 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:00.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:00 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:00.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:00 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:00.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:00 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:24:00.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:00 smithi119 ceph-mon[66861]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:01.096 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:24:00 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z[69449]: 2024-03-19T03:24:00.806+0000 7f418afd8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T03:24:01.096 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:24:00 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z[69449]: 2024-03-19T03:24:00.897+0000 7f418afd8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T03:24:01.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:00 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:01.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:00 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:01.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:00 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:01.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:00 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:01.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:00 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:24:01.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:00 smithi178 ceph-mon[67578]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:01.329 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:24:00 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y[68252]: 2024-03-19T03:24:00.998+0000 7fccbed5f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T03:24:01.329 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:24:01 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y[68252]: 2024-03-19T03:24:01.140+0000 7fccbed5f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T03:24:01.329 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:24:01 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y[68252]: 2024-03-19T03:24:01.236+0000 7fccbed5f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T03:24:01.329 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:24:01 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y[68252]: 2024-03-19T03:24:01.327+0000 7fccbed5f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T03:24:01.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.542+0000 7f7caa400700 1 -- 172.21.15.117:0/4091266100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ca4102440 msgr2=0x7f7ca41028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:24:01.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.542+0000 7f7caa400700 1 --2- 172.21.15.117:0/4091266100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ca4102440 0x7f7ca41028a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7c900099c0 tx=0x7f7c90009cd0 comp rx=0 tx=0).stop 2024-03-19T03:24:01.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.542+0000 7f7caa400700 1 -- 172.21.15.117:0/4091266100 shutdown_connections 2024-03-19T03:24:01.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.542+0000 7f7caa400700 1 --2- 172.21.15.117:0/4091266100 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7ca4102de0 0x7f7ca4109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:24:01.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.542+0000 7f7caa400700 1 --2- 172.21.15.117:0/4091266100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ca4101240 0x7f7ca4101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:24:01.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.542+0000 7f7caa400700 1 --2- 172.21.15.117:0/4091266100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ca4102440 0x7f7ca41028a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:24:01.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.542+0000 7f7caa400700 1 -- 172.21.15.117:0/4091266100 >> 172.21.15.117:0/4091266100 conn(0x7f7ca40fc7e0 msgr2=0x7f7ca40fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:24:01.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.542+0000 7f7caa400700 1 -- 172.21.15.117:0/4091266100 shutdown_connections 2024-03-19T03:24:01.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.542+0000 7f7caa400700 1 -- 172.21.15.117:0/4091266100 wait complete. 2024-03-19T03:24:01.546 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.543+0000 7f7caa400700 1 Processor -- start 2024-03-19T03:24:01.546 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.543+0000 7f7caa400700 1 -- start start 2024-03-19T03:24:01.546 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.543+0000 7f7caa400700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7ca4101240 0x7f7ca41051d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:24:01.546 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.543+0000 7f7caa400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ca4102440 0x7f7ca4103820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:24:01.546 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.543+0000 7f7caa400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ca4102de0 0x7f7ca4103d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:24:01.546 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.543+0000 7f7caa400700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca410ce10 con 0x7f7ca4102de0 2024-03-19T03:24:01.546 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.543+0000 7f7caa400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca410cc90 con 0x7f7ca4102440 2024-03-19T03:24:01.546 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.543+0000 7f7caa400700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca410cf90 con 0x7f7ca4101240 2024-03-19T03:24:01.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.544+0000 7f7ca3400700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7ca4101240 0x7f7ca41051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:24:01.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.544+0000 7f7ca3400700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7ca4101240 0x7f7ca41051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:38636/0 (socket says 172.21.15.117:38636) 2024-03-19T03:24:01.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.544+0000 7f7ca3400700 1 -- 172.21.15.117:0/1307502220 learned_addr learned my addr 172.21.15.117:0/1307502220 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:24:01.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.544+0000 7f7ca2a00700 1 --2- 172.21.15.117:0/1307502220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ca4102440 0x7f7ca4103820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:24:01.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.544+0000 7f7ca3e00700 1 --2- 172.21.15.117:0/1307502220 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ca4102de0 0x7f7ca4103d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:24:01.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.544+0000 7f7ca3400700 1 -- 172.21.15.117:0/1307502220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ca4102440 msgr2=0x7f7ca4103820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:24:01.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.544+0000 7f7ca3400700 1 --2- 172.21.15.117:0/1307502220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ca4102440 0x7f7ca4103820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:24:01.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.544+0000 7f7ca3400700 1 -- 172.21.15.117:0/1307502220 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ca4102de0 msgr2=0x7f7ca4103d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:24:01.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.544+0000 7f7ca3400700 1 --2- 172.21.15.117:0/1307502220 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ca4102de0 0x7f7ca4103d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:24:01.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.544+0000 7f7ca3400700 1 -- 172.21.15.117:0/1307502220 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca4104610 con 0x7f7ca4101240 2024-03-19T03:24:01.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.544+0000 7f7ca2a00700 1 --2- 172.21.15.117:0/1307502220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ca4102440 0x7f7ca4103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:24:01.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.544+0000 7f7ca3e00700 1 --2- 172.21.15.117:0/1307502220 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ca4102de0 0x7f7ca4103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:24:01.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.544+0000 7f7ca3400700 1 --2- 172.21.15.117:0/1307502220 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7ca4101240 0x7f7ca41051d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7c9400ea50 tx=0x7f7c9400ee10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:24:01.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.545+0000 7f7c9be00700 1 -- 172.21.15.117:0/1307502220 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c9400cc90 con 0x7f7ca4101240 2024-03-19T03:24:01.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.545+0000 7f7caa400700 1 -- 172.21.15.117:0/1307502220 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ca41a26d0 con 0x7f7ca4101240 2024-03-19T03:24:01.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.545+0000 7f7c9be00700 1 -- 172.21.15.117:0/1307502220 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c9400ce30 con 0x7f7ca4101240 2024-03-19T03:24:01.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.545+0000 7f7caa400700 1 -- 172.21.15.117:0/1307502220 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ca41a2be0 con 0x7f7ca4101240 2024-03-19T03:24:01.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.545+0000 7f7c9be00700 1 -- 172.21.15.117:0/1307502220 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c940188b0 con 0x7f7ca4101240 2024-03-19T03:24:01.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.546+0000 7f7caa400700 1 -- 172.21.15.117:0/1307502220 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ca4068fd0 con 0x7f7ca4101240 2024-03-19T03:24:01.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.546+0000 7f7c9be00700 1 -- 172.21.15.117:0/1307502220 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f7c9401f020 con 0x7f7ca4101240 2024-03-19T03:24:01.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.546+0000 7f7c9be00700 1 --2- 172.21.15.117:0/1307502220 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7c88039200 0x7f7c8803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:24:01.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.547+0000 7f7c9be00700 1 -- 172.21.15.117:0/1307502220 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7c9404c240 con 0x7f7ca4101240 2024-03-19T03:24:01.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.547+0000 7f7ca2a00700 1 --2- 172.21.15.117:0/1307502220 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7c88039200 0x7f7c8803b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:24:01.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.547+0000 7f7ca2a00700 1 --2- 172.21.15.117:0/1307502220 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7c88039200 0x7f7c8803b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7c90000c00 tx=0x7f7c90005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:24:01.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.552+0000 7f7c9be00700 1 -- 172.21.15.117:0/1307502220 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7c94010b30 con 0x7f7ca4101240 2024-03-19T03:24:01.574 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:24:01 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z[69449]: 2024-03-19T03:24:01.099+0000 7f418afd8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T03:24:01.574 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:24:01 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z[69449]: 2024-03-19T03:24:01.292+0000 7f418afd8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T03:24:01.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:01.785+0000 7f7caa400700 1 -- 172.21.15.117:0/1307502220 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f7ca40fe9b0 con 0x7f7c88039200 2024-03-19T03:24:01.836 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:24:01 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z[69449]: 2024-03-19T03:24:01.574+0000 7f418afd8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T03:24:01.836 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:24:01 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z[69449]: 2024-03-19T03:24:01.835+0000 7f418afd8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T03:24:01.886 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:24:01 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y[68252]: 2024-03-19T03:24:01.598+0000 7fccbed5f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T03:24:01.886 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:24:01 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y[68252]: 2024-03-19T03:24:01.706+0000 7fccbed5f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T03:24:01.886 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:24:01 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y[68252]: 2024-03-19T03:24:01.791+0000 7fccbed5f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T03:24:01.887 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:24:01 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y[68252]: 2024-03-19T03:24:01.885+0000 7fccbed5f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T03:24:02.096 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:24:02 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z[69449]: 2024-03-19T03:24:02.068+0000 7f418afd8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T03:24:02.234 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:24:02 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y[68252]: 2024-03-19T03:24:02.057+0000 7fccbed5f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T03:24:02.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:02 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:02.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:02 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:02.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:02 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:02.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:02 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:02.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:02 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:02.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:02 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:24:02.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:02 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:02.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:02 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T03:24:02.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:02 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T03:24:02.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:02 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:02.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:02 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:24:02.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:02 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:24:02.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:02 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:02.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:02 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:02.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:02 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:02.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:02 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:02.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:02 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:02.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:02 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:02.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:02 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:24:02.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:02 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:02.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:02 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T03:24:02.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:02 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T03:24:02.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:02 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:02.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:02 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:24:02.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:02 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:24:02.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:02 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:02.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:02 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:02.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:02 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:02.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:02 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:02.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:02 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:02.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:02 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:02.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:02 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:24:02.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:02 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:02.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:02 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T03:24:02.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:02 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T03:24:02.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:02 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:02.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:02 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:24:02.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:02 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:24:02.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:02 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:03.278 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:24:03 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y[68252]: 2024-03-19T03:24:03.016+0000 7fccbed5f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T03:24:03.278 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:24:03 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y[68252]: 2024-03-19T03:24:03.144+0000 7fccbed5f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T03:24:03.278 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:24:02 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z[69449]: 2024-03-19T03:24:02.860+0000 7f418afd8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T03:24:03.505 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:03 smithi117 ceph-mon[31838]: Reconfiguring mgr.x (unknown last config time)... 2024-03-19T03:24:03.505 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:03 smithi117 ceph-mon[31838]: Reconfiguring daemon mgr.x on smithi117 2024-03-19T03:24:03.505 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:03 smithi117 ceph-mon[31838]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:24:03.505 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:03 smithi117 ceph-mon[31838]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:03.506 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:03 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:03.538 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:24:03 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z[69449]: 2024-03-19T03:24:03.442+0000 7f418afd8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T03:24:03.538 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:24:03 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z[69449]: 2024-03-19T03:24:03.537+0000 7f418afd8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T03:24:03.539 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:03 smithi178 ceph-mon[67578]: Reconfiguring mgr.x (unknown last config time)... 2024-03-19T03:24:03.539 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:03 smithi178 ceph-mon[67578]: Reconfiguring daemon mgr.x on smithi117 2024-03-19T03:24:03.539 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:03 smithi178 ceph-mon[67578]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:24:03.539 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:03 smithi178 ceph-mon[67578]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:03.539 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:03 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:03.565 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:24:03 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y[68252]: 2024-03-19T03:24:03.475+0000 7fccbed5f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T03:24:03.566 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:03 smithi119 ceph-mon[66861]: Reconfiguring mgr.x (unknown last config time)... 2024-03-19T03:24:03.566 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:03 smithi119 ceph-mon[66861]: Reconfiguring daemon mgr.x on smithi117 2024-03-19T03:24:03.566 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:03 smithi119 ceph-mon[66861]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:24:03.566 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:03 smithi119 ceph-mon[66861]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:03.566 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:03 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:03.846 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:24:03 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z[69449]: 2024-03-19T03:24:03.751+0000 7f418afd8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T03:24:03.943 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:24:03 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y[68252]: 2024-03-19T03:24:03.564+0000 7fccbed5f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T03:24:03.943 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:24:03 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y[68252]: 2024-03-19T03:24:03.758+0000 7fccbed5f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T03:24:03.943 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:24:03 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y[68252]: 2024-03-19T03:24:03.942+0000 7fccbed5f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T03:24:04.478 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:24:04 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y[68252]: 2024-03-19T03:24:04.222+0000 7fccbed5f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T03:24:04.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:04.695+0000 7f7c9be00700 1 -- 172.21.15.117:0/1307502220 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mgrmap(e 14) ==== 91699+0+0 (secure 0 0 0) 0x7f7c94021920 con 0x7f7ca4101240 2024-03-19T03:24:04.734 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:24:04 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y[68252]: 2024-03-19T03:24:04.478+0000 7fccbed5f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T03:24:04.734 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:24:04 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y[68252]: 2024-03-19T03:24:04.713+0000 7fccbed5f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T03:24:04.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:04 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:04.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:04 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:04.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:04 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:24:04.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:04 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:04.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:04 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:24:04.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:04 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:04.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:04 smithi119 ceph-mon[66861]: Standby manager daemon z started 2024-03-19T03:24:04.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:04 smithi119 ceph-mon[66861]: from='mgr.? 172.21.15.178:0/2183940668' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-19T03:24:04.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:04 smithi119 ceph-mon[66861]: from='mgr.? 172.21.15.178:0/2183940668' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T03:24:04.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:04 smithi119 ceph-mon[66861]: from='mgr.? 172.21.15.178:0/2183940668' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-19T03:24:04.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:04 smithi119 ceph-mon[66861]: from='mgr.? 172.21.15.178:0/2183940668' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T03:24:04.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:04 smithi119 ceph-mon[66861]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:04.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:04 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:04.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:04 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:04.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:04 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:24:04.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:04 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:04.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:04 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:24:04.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:04 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:04.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:04 smithi117 ceph-mon[31838]: Standby manager daemon z started 2024-03-19T03:24:04.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:04 smithi117 ceph-mon[31838]: from='mgr.? 172.21.15.178:0/2183940668' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-19T03:24:04.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:04 smithi117 ceph-mon[31838]: from='mgr.? 172.21.15.178:0/2183940668' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T03:24:04.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:04 smithi117 ceph-mon[31838]: from='mgr.? 172.21.15.178:0/2183940668' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-19T03:24:04.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:04 smithi117 ceph-mon[31838]: from='mgr.? 172.21.15.178:0/2183940668' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T03:24:04.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:04 smithi117 ceph-mon[31838]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:05.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:04 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:05.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:04 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:05.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:04 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:24:05.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:04 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:05.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:04 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:24:05.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:04 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:05.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:04 smithi178 ceph-mon[67578]: Standby manager daemon z started 2024-03-19T03:24:05.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:04 smithi178 ceph-mon[67578]: from='mgr.? 172.21.15.178:0/2183940668' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-19T03:24:05.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:04 smithi178 ceph-mon[67578]: from='mgr.? 172.21.15.178:0/2183940668' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T03:24:05.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:04 smithi178 ceph-mon[67578]: from='mgr.? 172.21.15.178:0/2183940668' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-19T03:24:05.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:04 smithi178 ceph-mon[67578]: from='mgr.? 172.21.15.178:0/2183940668' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T03:24:05.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:04 smithi178 ceph-mon[67578]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:05.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:05 smithi119 ceph-mon[66861]: mgrmap e14: x(active, since 86s), standbys: z 2024-03-19T03:24:05.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:05 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-19T03:24:05.984 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:24:05 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y[68252]: 2024-03-19T03:24:05.509+0000 7fccbed5f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T03:24:06.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:05 smithi117 ceph-mon[31838]: mgrmap e14: x(active, since 86s), standbys: z 2024-03-19T03:24:06.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:05 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-19T03:24:06.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:05 smithi178 ceph-mon[67578]: mgrmap e14: x(active, since 86s), standbys: z 2024-03-19T03:24:06.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:05 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-19T03:24:06.396 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:24:06 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y[68252]: 2024-03-19T03:24:06.084+0000 7fccbed5f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T03:24:06.397 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:24:06 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y[68252]: 2024-03-19T03:24:06.181+0000 7fccbed5f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T03:24:06.397 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:24:06 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y[68252]: 2024-03-19T03:24:06.395+0000 7fccbed5f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T03:24:06.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:06.712+0000 7f7c9be00700 1 -- 172.21.15.117:0/1307502220 <== mon.1 v2:172.21.15.178:3300/0 8 ==== mgrmap(e 15) ==== 137235+0+0 (secure 0 0 0) 0x7f7c94014230 con 0x7f7ca4101240 2024-03-19T03:24:06.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:06 smithi119 ceph-mon[66861]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:06.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:06 smithi119 ceph-mon[66861]: Standby manager daemon y started 2024-03-19T03:24:06.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:06 smithi119 ceph-mon[66861]: from='mgr.? 172.21.15.119:0/2667807985' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-19T03:24:06.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:06 smithi119 ceph-mon[66861]: from='mgr.? 172.21.15.119:0/2667807985' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T03:24:06.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:06 smithi119 ceph-mon[66861]: from='mgr.? 172.21.15.119:0/2667807985' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-19T03:24:06.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:06 smithi119 ceph-mon[66861]: from='mgr.? 172.21.15.119:0/2667807985' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T03:24:06.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:06 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1750748030' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b32eb0ee-9d4b-4748-87ec-eda02c907415"}]: dispatch 2024-03-19T03:24:06.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:06 smithi119 ceph-mon[66861]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b32eb0ee-9d4b-4748-87ec-eda02c907415"}]: dispatch 2024-03-19T03:24:06.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:06 smithi119 ceph-mon[66861]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b32eb0ee-9d4b-4748-87ec-eda02c907415"}]': finished 2024-03-19T03:24:06.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:06 smithi119 ceph-mon[66861]: osdmap e5: 1 total, 0 up, 1 in 2024-03-19T03:24:06.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:06 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:24:07.036 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:06 smithi117 ceph-mon[31838]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:07.037 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:06 smithi117 ceph-mon[31838]: Standby manager daemon y started 2024-03-19T03:24:07.037 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:06 smithi117 ceph-mon[31838]: from='mgr.? 172.21.15.119:0/2667807985' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-19T03:24:07.037 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:06 smithi117 ceph-mon[31838]: from='mgr.? 172.21.15.119:0/2667807985' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T03:24:07.037 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:06 smithi117 ceph-mon[31838]: from='mgr.? 172.21.15.119:0/2667807985' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-19T03:24:07.037 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:06 smithi117 ceph-mon[31838]: from='mgr.? 172.21.15.119:0/2667807985' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T03:24:07.037 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:06 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1750748030' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b32eb0ee-9d4b-4748-87ec-eda02c907415"}]: dispatch 2024-03-19T03:24:07.037 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:06 smithi117 ceph-mon[31838]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b32eb0ee-9d4b-4748-87ec-eda02c907415"}]: dispatch 2024-03-19T03:24:07.037 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:06 smithi117 ceph-mon[31838]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b32eb0ee-9d4b-4748-87ec-eda02c907415"}]': finished 2024-03-19T03:24:07.037 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:06 smithi117 ceph-mon[31838]: osdmap e5: 1 total, 0 up, 1 in 2024-03-19T03:24:07.037 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:06 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:24:07.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:06 smithi178 ceph-mon[67578]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:07.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:06 smithi178 ceph-mon[67578]: Standby manager daemon y started 2024-03-19T03:24:07.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:06 smithi178 ceph-mon[67578]: from='mgr.? 172.21.15.119:0/2667807985' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-19T03:24:07.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:06 smithi178 ceph-mon[67578]: from='mgr.? 172.21.15.119:0/2667807985' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T03:24:07.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:06 smithi178 ceph-mon[67578]: from='mgr.? 172.21.15.119:0/2667807985' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-19T03:24:07.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:06 smithi178 ceph-mon[67578]: from='mgr.? 172.21.15.119:0/2667807985' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T03:24:07.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:06 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1750748030' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b32eb0ee-9d4b-4748-87ec-eda02c907415"}]: dispatch 2024-03-19T03:24:07.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:06 smithi178 ceph-mon[67578]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b32eb0ee-9d4b-4748-87ec-eda02c907415"}]: dispatch 2024-03-19T03:24:07.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:06 smithi178 ceph-mon[67578]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b32eb0ee-9d4b-4748-87ec-eda02c907415"}]': finished 2024-03-19T03:24:07.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:06 smithi178 ceph-mon[67578]: osdmap e5: 1 total, 0 up, 1 in 2024-03-19T03:24:07.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:06 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:24:07.974 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:07 smithi117 ceph-mon[31838]: mgrmap e15: x(active, since 88s), standbys: z, y 2024-03-19T03:24:07.974 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:07 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-19T03:24:07.974 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:07 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1032348313' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:24:07.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:07 smithi119 ceph-mon[66861]: mgrmap e15: x(active, since 88s), standbys: z, y 2024-03-19T03:24:07.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:07 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-19T03:24:07.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:07 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1032348313' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:24:08.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:07 smithi178 ceph-mon[67578]: mgrmap e15: x(active, since 88s), standbys: z, y 2024-03-19T03:24:08.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:07 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-19T03:24:08.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:07 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1032348313' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:24:08.971 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:08 smithi117 ceph-mon[31838]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:08.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:08 smithi119 ceph-mon[66861]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:09.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:08 smithi178 ceph-mon[67578]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:11.291 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:11 smithi117 ceph-mon[31838]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:11.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:11 smithi119 ceph-mon[66861]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:11.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:11 smithi178 ceph-mon[67578]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:13.462 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:13 smithi117 ceph-mon[31838]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:13.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:13 smithi119 ceph-mon[66861]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:13.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:13 smithi178 ceph-mon[67578]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:15.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:15 smithi119 ceph-mon[66861]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:15.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:15 smithi117 ceph-mon[31838]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:15.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:15 smithi178 ceph-mon[67578]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:16.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:16 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-19T03:24:16.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:16 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:16.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:16 smithi117 ceph-mon[31838]: Deploying daemon osd.0 on smithi117 2024-03-19T03:24:16.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:16 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-19T03:24:16.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:16 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:16.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:16 smithi178 ceph-mon[67578]: Deploying daemon osd.0 on smithi117 2024-03-19T03:24:16.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:16 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-19T03:24:16.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:16 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:16.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:16 smithi119 ceph-mon[66861]: Deploying daemon osd.0 on smithi117 2024-03-19T03:24:17.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:17 smithi117 ceph-mon[31838]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:17.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:17 smithi178 ceph-mon[67578]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:17 smithi119 ceph-mon[66861]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:19.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:19 smithi117 ceph-mon[31838]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:19.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:19 smithi178 ceph-mon[67578]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:19.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:19 smithi119 ceph-mon[66861]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:21.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:21 smithi117 ceph-mon[31838]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:21.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:21 smithi178 ceph-mon[67578]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:21 smithi119 ceph-mon[66861]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:23.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:23 smithi117 ceph-mon[31838]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:23.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:23 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:23.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:23 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:24:23.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:23 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:23.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:23 smithi178 ceph-mon[67578]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:23.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:23 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:23.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:23 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:24:23.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:23 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:23.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:23 smithi119 ceph-mon[66861]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:23.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:23 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:23.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:23 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:24:23.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:23 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:25.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:25 smithi117 ceph-mon[31838]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:25.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:25 smithi178 ceph-mon[67578]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:25.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:25 smithi119 ceph-mon[66861]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:27.522 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:27 smithi117 ceph-mon[31838]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:27.522 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:27 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:27.522 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:27 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:27.522 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:27 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:27.522 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:27 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:24:27.522 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:27 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:27.522 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:27 smithi117 ceph-mon[31838]: from='osd.0 [v2:172.21.15.117:6802/3014934900,v1:172.21.15.117:6803/3014934900]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-19T03:24:27.536 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:27 smithi119 ceph-mon[66861]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:27.536 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:27 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:27.536 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:27 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:27.537 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:27 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:27.537 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:27 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:24:27.537 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:27 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:27.537 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:27 smithi119 ceph-mon[66861]: from='osd.0 [v2:172.21.15.117:6802/3014934900,v1:172.21.15.117:6803/3014934900]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-19T03:24:27.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:27 smithi178 ceph-mon[67578]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:27.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:27 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:27.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:27 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:27.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:27 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:27.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:27 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:24:27.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:27 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:27.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:27 smithi178 ceph-mon[67578]: from='osd.0 [v2:172.21.15.117:6802/3014934900,v1:172.21.15.117:6803/3014934900]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-19T03:24:27.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:27.625+0000 7f7c9be00700 1 -- 172.21.15.117:0/1307502220 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7ca40fe9b0 con 0x7f7c88039200 2024-03-19T03:24:27.626 INFO:teuthology.orchestra.run.smithi117.stdout:Created osd(s) 0 on host 'smithi117' 2024-03-19T03:24:27.630 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:27.629+0000 7f7caa400700 1 -- 172.21.15.117:0/1307502220 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7c88039200 msgr2=0x7f7c8803b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:24:27.630 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:27.629+0000 7f7caa400700 1 --2- 172.21.15.117:0/1307502220 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7c88039200 0x7f7c8803b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7c90000c00 tx=0x7f7c90005fd0 comp rx=0 tx=0).stop 2024-03-19T03:24:27.630 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:27.629+0000 7f7caa400700 1 -- 172.21.15.117:0/1307502220 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7ca4101240 msgr2=0x7f7ca41051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:24:27.630 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:27.629+0000 7f7caa400700 1 --2- 172.21.15.117:0/1307502220 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7ca4101240 0x7f7ca41051d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7c9400ea50 tx=0x7f7c9400ee10 comp rx=0 tx=0).stop 2024-03-19T03:24:27.631 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:27.629+0000 7f7caa400700 1 -- 172.21.15.117:0/1307502220 shutdown_connections 2024-03-19T03:24:27.631 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:27.629+0000 7f7caa400700 1 --2- 172.21.15.117:0/1307502220 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7c88039200 0x7f7c8803b6c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:24:27.631 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:27.629+0000 7f7caa400700 1 --2- 172.21.15.117:0/1307502220 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ca4102de0 0x7f7ca4103d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:24:27.631 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:27.629+0000 7f7caa400700 1 --2- 172.21.15.117:0/1307502220 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7ca4101240 0x7f7ca41051d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:24:27.631 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:27.629+0000 7f7caa400700 1 --2- 172.21.15.117:0/1307502220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ca4102440 0x7f7ca4103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:24:27.631 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:27.629+0000 7f7caa400700 1 -- 172.21.15.117:0/1307502220 >> 172.21.15.117:0/1307502220 conn(0x7f7ca40fc7e0 msgr2=0x7f7ca40fe2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:24:27.631 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:27.629+0000 7f7caa400700 1 -- 172.21.15.117:0/1307502220 shutdown_connections 2024-03-19T03:24:27.631 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:24:27.630+0000 7f7caa400700 1 -- 172.21.15.117:0/1307502220 wait complete. 2024-03-19T03:24:28.367 DEBUG:teuthology.orchestra.run.smithi117:osd.0> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.0.service 2024-03-19T03:24:28.371 INFO:tasks.cephadm:Deploying osd.1 on smithi119 with /dev/vg_nvme/lv_4... 2024-03-19T03:24:28.371 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-19T03:24:28.592 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:28 smithi117 ceph-mon[31838]: from='osd.0 [v2:172.21.15.117:6802/3014934900,v1:172.21.15.117:6803/3014934900]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-19T03:24:28.592 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:28 smithi117 ceph-mon[31838]: osdmap e6: 1 total, 0 up, 1 in 2024-03-19T03:24:28.592 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:28 smithi117 ceph-mon[31838]: from='osd.0 [v2:172.21.15.117:6802/3014934900,v1:172.21.15.117:6803/3014934900]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-19T03:24:28.592 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:28 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:24:28.592 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:28 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:28.593 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:28 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:28.593 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:28 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:24:28.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:28 smithi178 ceph-mon[67578]: from='osd.0 [v2:172.21.15.117:6802/3014934900,v1:172.21.15.117:6803/3014934900]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-19T03:24:28.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:28 smithi178 ceph-mon[67578]: osdmap e6: 1 total, 0 up, 1 in 2024-03-19T03:24:28.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:28 smithi178 ceph-mon[67578]: from='osd.0 [v2:172.21.15.117:6802/3014934900,v1:172.21.15.117:6803/3014934900]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-19T03:24:28.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:28 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:24:28.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:28 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:28.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:28 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:28.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:28 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:24:28.612 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T03:24:28.731 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:28 smithi119 ceph-mon[66861]: from='osd.0 [v2:172.21.15.117:6802/3014934900,v1:172.21.15.117:6803/3014934900]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-19T03:24:28.732 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:28 smithi119 ceph-mon[66861]: osdmap e6: 1 total, 0 up, 1 in 2024-03-19T03:24:28.732 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:28 smithi119 ceph-mon[66861]: from='osd.0 [v2:172.21.15.117:6802/3014934900,v1:172.21.15.117:6803/3014934900]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-19T03:24:28.732 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:28 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:24:28.732 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:28 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:28.732 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:28 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:28.732 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:28 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:24:29.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:29 smithi178 ceph-mon[67578]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:29.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:29 smithi178 ceph-mon[67578]: from='osd.0 [v2:172.21.15.117:6802/3014934900,v1:172.21.15.117:6803/3014934900]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-19T03:24:29.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:29 smithi178 ceph-mon[67578]: osdmap e7: 1 total, 0 up, 1 in 2024-03-19T03:24:29.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:29 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:24:29.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:29 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:24:29.644 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:29 smithi117 ceph-mon[31838]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:29.645 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:29 smithi117 ceph-mon[31838]: from='osd.0 [v2:172.21.15.117:6802/3014934900,v1:172.21.15.117:6803/3014934900]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-19T03:24:29.645 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:29 smithi117 ceph-mon[31838]: osdmap e7: 1 total, 0 up, 1 in 2024-03-19T03:24:29.645 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:29 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:24:29.645 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:29 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:24:29.662 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:29 smithi119 ceph-mon[66861]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:29.662 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:29 smithi119 ceph-mon[66861]: from='osd.0 [v2:172.21.15.117:6802/3014934900,v1:172.21.15.117:6803/3014934900]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-19T03:24:29.662 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:29 smithi119 ceph-mon[66861]: osdmap e7: 1 total, 0 up, 1 in 2024-03-19T03:24:29.662 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:29 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:24:29.662 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:29 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:24:30.500 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-19T03:24:30.500 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-19T03:24:30.500 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-03-19T03:24:30.500 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-03-19T03:24:30.500 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111418 s, 941 MB/s 2024-03-19T03:24:30.500 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-03-19T03:24:30.501 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:30 smithi119 ceph-mon[66861]: purged_snaps scrub starts 2024-03-19T03:24:30.501 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:30 smithi119 ceph-mon[66861]: purged_snaps scrub ok 2024-03-19T03:24:30.501 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:30 smithi119 ceph-mon[66861]: osd.0 [v2:172.21.15.117:6802/3014934900,v1:172.21.15.117:6803/3014934900] boot 2024-03-19T03:24:30.501 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:30 smithi119 ceph-mon[66861]: osdmap e8: 1 total, 1 up, 1 in 2024-03-19T03:24:30.501 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:30 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:24:30.501 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:30 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:24:30.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:30 smithi117 ceph-mon[31838]: purged_snaps scrub starts 2024-03-19T03:24:30.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:30 smithi117 ceph-mon[31838]: purged_snaps scrub ok 2024-03-19T03:24:30.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:30 smithi117 ceph-mon[31838]: osd.0 [v2:172.21.15.117:6802/3014934900,v1:172.21.15.117:6803/3014934900] boot 2024-03-19T03:24:30.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:30 smithi117 ceph-mon[31838]: osdmap e8: 1 total, 1 up, 1 in 2024-03-19T03:24:30.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:30 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:24:30.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:30 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:24:30.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:30 smithi178 ceph-mon[67578]: purged_snaps scrub starts 2024-03-19T03:24:30.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:30 smithi178 ceph-mon[67578]: purged_snaps scrub ok 2024-03-19T03:24:30.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:30 smithi178 ceph-mon[67578]: osd.0 [v2:172.21.15.117:6802/3014934900,v1:172.21.15.117:6803/3014934900] boot 2024-03-19T03:24:30.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:30 smithi178 ceph-mon[67578]: osdmap e8: 1 total, 1 up, 1 in 2024-03-19T03:24:30.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:30 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:24:30.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:30 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T03:24:31.120 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi119:vg_nvme/lv_4 2024-03-19T03:24:31.351 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T03:24:31.470 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:31 smithi119 ceph-mon[66861]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:31.470 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:31 smithi119 ceph-mon[66861]: osdmap e9: 1 total, 1 up, 1 in 2024-03-19T03:24:31.693 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:31 smithi117 ceph-mon[31838]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:31.693 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:31 smithi117 ceph-mon[31838]: osdmap e9: 1 total, 1 up, 1 in 2024-03-19T03:24:31.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:31 smithi178 ceph-mon[67578]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T03:24:31.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:31 smithi178 ceph-mon[67578]: osdmap e9: 1 total, 1 up, 1 in 2024-03-19T03:24:32.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.830+0000 7fe4e8387700 1 -- 172.21.15.119:0/3275740339 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4e0101560 msgr2=0x7fe4e01019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:24:32.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.830+0000 7fe4e8387700 1 --2- 172.21.15.119:0/3275740339 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4e0101560 0x7fe4e01019c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0009990 tx=0x7fe4d0009ca0 comp rx=0 tx=0).stop 2024-03-19T03:24:32.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.831+0000 7fe4e8387700 1 -- 172.21.15.119:0/3275740339 shutdown_connections 2024-03-19T03:24:32.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.831+0000 7fe4e8387700 1 --2- 172.21.15.119:0/3275740339 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe4e0101f00 0x7fe4e01097c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:24:32.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.831+0000 7fe4e8387700 1 --2- 172.21.15.119:0/3275740339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4e0102820 0x7fe4e0102c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:24:32.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.831+0000 7fe4e8387700 1 --2- 172.21.15.119:0/3275740339 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4e0101560 0x7fe4e01019c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:24:32.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.831+0000 7fe4e8387700 1 -- 172.21.15.119:0/3275740339 >> 172.21.15.119:0/3275740339 conn(0x7fe4e00fbfb0 msgr2=0x7fe4e00fe3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:24:32.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.831+0000 7fe4e8387700 1 -- 172.21.15.119:0/3275740339 shutdown_connections 2024-03-19T03:24:32.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.831+0000 7fe4e8387700 1 -- 172.21.15.119:0/3275740339 wait complete. 2024-03-19T03:24:32.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.831+0000 7fe4e8387700 1 Processor -- start 2024-03-19T03:24:32.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.831+0000 7fe4e8387700 1 -- start start 2024-03-19T03:24:32.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.831+0000 7fe4e8387700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe4e0101560 0x7fe4e0100fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:24:32.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.831+0000 7fe4e8387700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4e0101f00 0x7fe4e019d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:24:32.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.831+0000 7fe4e8387700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4e0102820 0x7fe4e019f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:24:32.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.831+0000 7fe4e8387700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e010cf60 con 0x7fe4e0101f00 2024-03-19T03:24:32.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.831+0000 7fe4e8387700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e010cde0 con 0x7fe4e0102820 2024-03-19T03:24:32.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.831+0000 7fe4e8387700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e010d0e0 con 0x7fe4e0101560 2024-03-19T03:24:32.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.832+0000 7fe4e6924700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4e0102820 0x7fe4e019f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:24:32.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.832+0000 7fe4e6123700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe4e0101560 0x7fe4e0100fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:24:32.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.832+0000 7fe4e6123700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe4e0101560 0x7fe4e0100fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.119:52700/0 (socket says 172.21.15.119:52700) 2024-03-19T03:24:32.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.832+0000 7fe4e6123700 1 -- 172.21.15.119:0/2598666758 learned_addr learned my addr 172.21.15.119:0/2598666758 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-19T03:24:32.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.832+0000 7fe4e6924700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4e0102820 0x7fe4e019f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41186/0 (socket says 172.21.15.119:41186) 2024-03-19T03:24:32.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.832+0000 7fe4e5922700 1 --2- 172.21.15.119:0/2598666758 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4e0101f00 0x7fe4e019d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:24:32.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.832+0000 7fe4e6924700 1 -- 172.21.15.119:0/2598666758 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe4e0101560 msgr2=0x7fe4e0100fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:24:32.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.832+0000 7fe4e6924700 1 --2- 172.21.15.119:0/2598666758 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe4e0101560 0x7fe4e0100fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:24:32.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.832+0000 7fe4e6924700 1 -- 172.21.15.119:0/2598666758 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4e0101f00 msgr2=0x7fe4e019d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:24:32.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.832+0000 7fe4e6924700 1 --2- 172.21.15.119:0/2598666758 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4e0101f00 0x7fe4e019d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:24:32.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.832+0000 7fe4e6924700 1 -- 172.21.15.119:0/2598666758 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4e01a0280 con 0x7fe4e0102820 2024-03-19T03:24:32.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.832+0000 7fe4e5922700 1 --2- 172.21.15.119:0/2598666758 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4e0101f00 0x7fe4e019d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T03:24:32.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.832+0000 7fe4e6924700 1 --2- 172.21.15.119:0/2598666758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4e0102820 0x7fe4e019f9b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8007f10 tx=0x7fe4d800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:24:32.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.832+0000 7fe4d77fe700 1 -- 172.21.15.119:0/2598666758 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d800ddd0 con 0x7fe4e0102820 2024-03-19T03:24:32.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.832+0000 7fe4e8387700 1 -- 172.21.15.119:0/2598666758 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4e01a0570 con 0x7fe4e0102820 2024-03-19T03:24:32.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.832+0000 7fe4d77fe700 1 -- 172.21.15.119:0/2598666758 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4d8009e20 con 0x7fe4e0102820 2024-03-19T03:24:32.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.832+0000 7fe4d77fe700 1 -- 172.21.15.119:0/2598666758 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d8015660 con 0x7fe4e0102820 2024-03-19T03:24:32.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.832+0000 7fe4e8387700 1 -- 172.21.15.119:0/2598666758 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4e01a0b30 con 0x7fe4e0102820 2024-03-19T03:24:32.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.833+0000 7fe4e8387700 1 -- 172.21.15.119:0/2598666758 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4e0068fd0 con 0x7fe4e0102820 2024-03-19T03:24:32.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.834+0000 7fe4d77fe700 1 -- 172.21.15.119:0/2598666758 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 15) ==== 137235+0+0 (secure 0 0 0) 0x7fe4d8004ad0 con 0x7fe4e0102820 2024-03-19T03:24:32.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.835+0000 7fe4d77fe700 1 --2- 172.21.15.119:0/2598666758 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe4cc0a2800 0x7fe4cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:24:32.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.835+0000 7fe4d77fe700 1 -- 172.21.15.119:0/2598666758 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fe4d80cd200 con 0x7fe4e0102820 2024-03-19T03:24:32.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.836+0000 7fe4e6123700 1 --2- 172.21.15.119:0/2598666758 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe4cc0a2800 0x7fe4cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:24:32.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.837+0000 7fe4e6123700 1 --2- 172.21.15.119:0/2598666758 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe4cc0a2800 0x7fe4cc0a4cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc009aa0 tx=0x7fe4dc009240 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:24:32.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:32.839+0000 7fe4d77fe700 1 -- 172.21.15.119:0/2598666758 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe4d8010070 con 0x7fe4e0102820 2024-03-19T03:24:33.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:33.053+0000 7fe4e8387700 1 -- 172.21.15.119:0/2598666758 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fe4e0062750 con 0x7fe4cc0a2800 2024-03-19T03:24:33.671 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:33 smithi119 ceph-mon[66861]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:33.671 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:33.671 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:33.671 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:24:33.671 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:33.671 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:33.671 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:24:33.671 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:33.671 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:24:33.672 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:24:33.672 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:33.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:33 smithi117 ceph-mon[31838]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:33.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:33.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:33.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:24:33.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:33.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:33.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:24:33.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:33.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:24:33.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:24:33.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:33 smithi178 ceph-mon[67578]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:24:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:24:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:24:33.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:24:33.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:34.534 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:34 smithi119 ceph-mon[66861]: Detected new or changed devices on smithi117 2024-03-19T03:24:34.534 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:34 smithi119 ceph-mon[66861]: Adjusting osd_memory_target on smithi117 to 17091M 2024-03-19T03:24:34.535 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:34 smithi119 ceph-mon[66861]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:24:34.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:34 smithi117 ceph-mon[31838]: Detected new or changed devices on smithi117 2024-03-19T03:24:34.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:34 smithi117 ceph-mon[31838]: Adjusting osd_memory_target on smithi117 to 17091M 2024-03-19T03:24:34.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:34 smithi117 ceph-mon[31838]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:24:34.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:34 smithi178 ceph-mon[67578]: Detected new or changed devices on smithi117 2024-03-19T03:24:34.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:34 smithi178 ceph-mon[67578]: Adjusting osd_memory_target on smithi117 to 17091M 2024-03-19T03:24:34.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:34 smithi178 ceph-mon[67578]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:24:35.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:35 smithi119 ceph-mon[66861]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:35.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:35 smithi117 ceph-mon[31838]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:35.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:35 smithi178 ceph-mon[67578]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:36.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:36 smithi119 ceph-mon[66861]: from='client.? 172.21.15.119:0/3897384706' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e07585be-c016-4466-b07c-abbc22a58de4"}]: dispatch 2024-03-19T03:24:36.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:36 smithi119 ceph-mon[66861]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e07585be-c016-4466-b07c-abbc22a58de4"}]: dispatch 2024-03-19T03:24:36.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:36 smithi119 ceph-mon[66861]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e07585be-c016-4466-b07c-abbc22a58de4"}]': finished 2024-03-19T03:24:36.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:36 smithi119 ceph-mon[66861]: osdmap e10: 2 total, 1 up, 2 in 2024-03-19T03:24:36.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:36 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:24:36.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:36 smithi119 ceph-mon[66861]: from='client.? 172.21.15.119:0/3955495543' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:24:36.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:36 smithi117 ceph-mon[31838]: from='client.? 172.21.15.119:0/3897384706' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e07585be-c016-4466-b07c-abbc22a58de4"}]: dispatch 2024-03-19T03:24:36.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:36 smithi117 ceph-mon[31838]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e07585be-c016-4466-b07c-abbc22a58de4"}]: dispatch 2024-03-19T03:24:36.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:36 smithi117 ceph-mon[31838]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e07585be-c016-4466-b07c-abbc22a58de4"}]': finished 2024-03-19T03:24:36.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:36 smithi117 ceph-mon[31838]: osdmap e10: 2 total, 1 up, 2 in 2024-03-19T03:24:36.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:36 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:24:36.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:36 smithi117 ceph-mon[31838]: from='client.? 172.21.15.119:0/3955495543' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:24:36.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:36 smithi178 ceph-mon[67578]: from='client.? 172.21.15.119:0/3897384706' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e07585be-c016-4466-b07c-abbc22a58de4"}]: dispatch 2024-03-19T03:24:36.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:36 smithi178 ceph-mon[67578]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e07585be-c016-4466-b07c-abbc22a58de4"}]: dispatch 2024-03-19T03:24:36.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:36 smithi178 ceph-mon[67578]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e07585be-c016-4466-b07c-abbc22a58de4"}]': finished 2024-03-19T03:24:36.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:36 smithi178 ceph-mon[67578]: osdmap e10: 2 total, 1 up, 2 in 2024-03-19T03:24:36.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:36 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:24:36.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:36 smithi178 ceph-mon[67578]: from='client.? 172.21.15.119:0/3955495543' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:24:37.693 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:37 smithi117 ceph-mon[31838]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:37.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:37 smithi119 ceph-mon[66861]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:37.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:37 smithi178 ceph-mon[67578]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:38.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:38 smithi119 ceph-mon[66861]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:38.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:38 smithi117 ceph-mon[31838]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:38.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:38 smithi178 ceph-mon[67578]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:41.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:41 smithi119 ceph-mon[66861]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:41.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:41 smithi117 ceph-mon[31838]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:41.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:41 smithi178 ceph-mon[67578]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:43.498 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:43 smithi119 ceph-mon[66861]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:43.498 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:43 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-19T03:24:43.498 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:43 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:43.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:43 smithi117 ceph-mon[31838]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:43.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:43 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-19T03:24:43.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:43 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:43.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:43 smithi178 ceph-mon[67578]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:43.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:43 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-19T03:24:43.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:43 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:44.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:44 smithi119 ceph-mon[66861]: Deploying daemon osd.1 on smithi119 2024-03-19T03:24:44.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:44 smithi117 ceph-mon[31838]: Deploying daemon osd.1 on smithi119 2024-03-19T03:24:44.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:44 smithi178 ceph-mon[67578]: Deploying daemon osd.1 on smithi119 2024-03-19T03:24:45.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:45 smithi117 ceph-mon[31838]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:45.576 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:45 smithi119 ceph-mon[66861]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:45.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:45 smithi178 ceph-mon[67578]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:47.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:47 smithi117 ceph-mon[31838]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:47.584 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:47 smithi119 ceph-mon[66861]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:47.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:47 smithi178 ceph-mon[67578]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:49.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:49 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:49.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:49 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:24:49.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:49 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:49.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:49 smithi119 ceph-mon[66861]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:49.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:49 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:49.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:49 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:24:49.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:49 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:49.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:49 smithi117 ceph-mon[31838]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:49.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:49 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:49.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:49 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:24:49.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:49 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:49.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:49 smithi178 ceph-mon[67578]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:51.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:51.132+0000 7fe4d77fe700 1 -- 172.21.15.119:0/2598666758 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe4e0062750 con 0x7fe4cc0a2800 2024-03-19T03:24:51.133 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 1 on host 'smithi119' 2024-03-19T03:24:51.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:51.134+0000 7fe4e8387700 1 -- 172.21.15.119:0/2598666758 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe4cc0a2800 msgr2=0x7fe4cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:24:51.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:51.134+0000 7fe4e8387700 1 --2- 172.21.15.119:0/2598666758 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe4cc0a2800 0x7fe4cc0a4cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc009aa0 tx=0x7fe4dc009240 comp rx=0 tx=0).stop 2024-03-19T03:24:51.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:51.134+0000 7fe4e8387700 1 -- 172.21.15.119:0/2598666758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4e0102820 msgr2=0x7fe4e019f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:24:51.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:51.134+0000 7fe4e8387700 1 --2- 172.21.15.119:0/2598666758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4e0102820 0x7fe4e019f9b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8007f10 tx=0x7fe4d800d3b0 comp rx=0 tx=0).stop 2024-03-19T03:24:51.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:51.134+0000 7fe4e8387700 1 -- 172.21.15.119:0/2598666758 shutdown_connections 2024-03-19T03:24:51.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:51.134+0000 7fe4e8387700 1 --2- 172.21.15.119:0/2598666758 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe4cc0a2800 0x7fe4cc0a4cc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:24:51.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:51.134+0000 7fe4e8387700 1 --2- 172.21.15.119:0/2598666758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4e0102820 0x7fe4e019f9b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:24:51.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:51.134+0000 7fe4e8387700 1 --2- 172.21.15.119:0/2598666758 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe4e0101560 0x7fe4e0100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:24:51.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:51.134+0000 7fe4e8387700 1 --2- 172.21.15.119:0/2598666758 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4e0101f00 0x7fe4e019d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:24:51.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:51.134+0000 7fe4e8387700 1 -- 172.21.15.119:0/2598666758 >> 172.21.15.119:0/2598666758 conn(0x7fe4e00fbfb0 msgr2=0x7fe4e0105810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:24:51.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:51.134+0000 7fe4e8387700 1 -- 172.21.15.119:0/2598666758 shutdown_connections 2024-03-19T03:24:51.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:24:51.134+0000 7fe4e8387700 1 -- 172.21.15.119:0/2598666758 wait complete. 2024-03-19T03:24:51.475 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:51 smithi119 ceph-mon[66861]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:51.475 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:51 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:51.475 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:51 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:51.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:51 smithi117 ceph-mon[31838]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:51.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:51 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:51.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:51 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:51.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:51 smithi178 ceph-mon[67578]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:51.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:51 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:51.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:51 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:52.072 DEBUG:teuthology.orchestra.run.smithi119:osd.1> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.1.service 2024-03-19T03:24:52.074 INFO:tasks.cephadm:Deploying osd.2 on smithi178 with /dev/vg_nvme/lv_4... 2024-03-19T03:24:52.074 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-19T03:24:52.318 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T03:24:52.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:52 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:52.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:52 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:52.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:52 smithi119 ceph-mon[66861]: from='osd.1 [v2:172.21.15.119:6800/2425927341,v1:172.21.15.119:6801/2425927341]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T03:24:52.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:52 smithi119 ceph-mon[66861]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T03:24:52.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:52 smithi119 ceph-mon[66861]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:53.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:52 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:53.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:52 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:53.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:52 smithi117 ceph-mon[31838]: from='osd.1 [v2:172.21.15.119:6800/2425927341,v1:172.21.15.119:6801/2425927341]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T03:24:53.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:52 smithi117 ceph-mon[31838]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T03:24:53.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:52 smithi117 ceph-mon[31838]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:53.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:52 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:53.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:52 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:53.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:52 smithi178 ceph-mon[67578]: from='osd.1 [v2:172.21.15.119:6800/2425927341,v1:172.21.15.119:6801/2425927341]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T03:24:53.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:52 smithi178 ceph-mon[67578]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T03:24:53.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:52 smithi178 ceph-mon[67578]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:53.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:53 smithi119 ceph-mon[66861]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-19T03:24:53.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:53 smithi119 ceph-mon[66861]: osdmap e11: 2 total, 1 up, 2 in 2024-03-19T03:24:53.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:53 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:24:53.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:53 smithi119 ceph-mon[66861]: from='osd.1 [v2:172.21.15.119:6800/2425927341,v1:172.21.15.119:6801/2425927341]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-19T03:24:53.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:53 smithi119 ceph-mon[66861]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-19T03:24:54.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:53 smithi117 ceph-mon[31838]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-19T03:24:54.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:53 smithi117 ceph-mon[31838]: osdmap e11: 2 total, 1 up, 2 in 2024-03-19T03:24:54.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:53 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:24:54.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:53 smithi117 ceph-mon[31838]: from='osd.1 [v2:172.21.15.119:6800/2425927341,v1:172.21.15.119:6801/2425927341]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-19T03:24:54.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:53 smithi117 ceph-mon[31838]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-19T03:24:54.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:53 smithi178 ceph-mon[67578]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-19T03:24:54.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:53 smithi178 ceph-mon[67578]: osdmap e11: 2 total, 1 up, 2 in 2024-03-19T03:24:54.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:53 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:24:54.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:53 smithi178 ceph-mon[67578]: from='osd.1 [v2:172.21.15.119:6800/2425927341,v1:172.21.15.119:6801/2425927341]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-19T03:24:54.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:53 smithi178 ceph-mon[67578]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-19T03:24:54.172 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-19T03:24:54.172 INFO:teuthology.orchestra.run.smithi178.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-19T03:24:54.172 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10+0 records in 2024-03-19T03:24:54.172 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records out 2024-03-19T03:24:54.172 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00834313 s, 1.3 GB/s 2024-03-19T03:24:54.172 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping successful for: 2024-03-19T03:24:54.468 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:24:54 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-1[71454]: 2024-03-19T03:24:54.211+0000 7f2a9e4ac700 -1 osd.1 0 waiting for initial osdmap 2024-03-19T03:24:54.469 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:24:54 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-1[71454]: 2024-03-19T03:24:54.215+0000 7f2a95a96700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T03:24:54.771 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi178:vg_nvme/lv_4 2024-03-19T03:24:55.013 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T03:24:55.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:54 smithi117 ceph-mon[31838]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-19T03:24:55.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:54 smithi117 ceph-mon[31838]: osdmap e12: 2 total, 1 up, 2 in 2024-03-19T03:24:55.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:54 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:24:55.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:54 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:24:55.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:54 smithi117 ceph-mon[31838]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:55.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:54 smithi117 ceph-mon[31838]: from='osd.1 ' entity='osd.1' 2024-03-19T03:24:55.074 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:54 smithi119 ceph-mon[66861]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-19T03:24:55.074 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:54 smithi119 ceph-mon[66861]: osdmap e12: 2 total, 1 up, 2 in 2024-03-19T03:24:55.075 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:54 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:24:55.075 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:54 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:24:55.075 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:54 smithi119 ceph-mon[66861]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:55.075 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:54 smithi119 ceph-mon[66861]: from='osd.1 ' entity='osd.1' 2024-03-19T03:24:55.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:54 smithi178 ceph-mon[67578]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-19T03:24:55.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:54 smithi178 ceph-mon[67578]: osdmap e12: 2 total, 1 up, 2 in 2024-03-19T03:24:55.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:54 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:24:55.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:54 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:24:55.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:54 smithi178 ceph-mon[67578]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-19T03:24:55.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:54 smithi178 ceph-mon[67578]: from='osd.1 ' entity='osd.1' 2024-03-19T03:24:56.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:55 smithi117 ceph-mon[31838]: purged_snaps scrub starts 2024-03-19T03:24:56.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:55 smithi117 ceph-mon[31838]: purged_snaps scrub ok 2024-03-19T03:24:56.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:24:56.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:55 smithi117 ceph-mon[31838]: Detected new or changed devices on smithi119 2024-03-19T03:24:56.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:56.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:56.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:24:56.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:55 smithi117 ceph-mon[31838]: Adjusting osd_memory_target on smithi119 to 17080M 2024-03-19T03:24:56.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:56.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:56.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:24:56.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:56.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:55 smithi117 ceph-mon[31838]: osd.1 [v2:172.21.15.119:6800/2425927341,v1:172.21.15.119:6801/2425927341] boot 2024-03-19T03:24:56.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:55 smithi117 ceph-mon[31838]: osdmap e13: 2 total, 2 up, 2 in 2024-03-19T03:24:56.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:24:56.044 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:24:56.044 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:56.044 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:24:56.044 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:56.091 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:55 smithi178 ceph-mon[67578]: purged_snaps scrub starts 2024-03-19T03:24:56.091 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:55 smithi178 ceph-mon[67578]: purged_snaps scrub ok 2024-03-19T03:24:56.091 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:24:56.091 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:55 smithi178 ceph-mon[67578]: Detected new or changed devices on smithi119 2024-03-19T03:24:56.091 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:56.091 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:56.091 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:24:56.092 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:55 smithi178 ceph-mon[67578]: Adjusting osd_memory_target on smithi119 to 17080M 2024-03-19T03:24:56.092 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:56.092 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:56.092 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:24:56.092 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:56.092 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:55 smithi178 ceph-mon[67578]: osd.1 [v2:172.21.15.119:6800/2425927341,v1:172.21.15.119:6801/2425927341] boot 2024-03-19T03:24:56.092 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:55 smithi178 ceph-mon[67578]: osdmap e13: 2 total, 2 up, 2 in 2024-03-19T03:24:56.092 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:24:56.092 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:24:56.092 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:56.092 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:24:56.092 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:56.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:55 smithi119 ceph-mon[66861]: purged_snaps scrub starts 2024-03-19T03:24:56.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:55 smithi119 ceph-mon[66861]: purged_snaps scrub ok 2024-03-19T03:24:56.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:24:56.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:55 smithi119 ceph-mon[66861]: Detected new or changed devices on smithi119 2024-03-19T03:24:56.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:56.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:56.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:24:56.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:55 smithi119 ceph-mon[66861]: Adjusting osd_memory_target on smithi119 to 17080M 2024-03-19T03:24:56.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:56.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:56.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:24:56.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:56.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:55 smithi119 ceph-mon[66861]: osd.1 [v2:172.21.15.119:6800/2425927341,v1:172.21.15.119:6801/2425927341] boot 2024-03-19T03:24:56.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:55 smithi119 ceph-mon[66861]: osdmap e13: 2 total, 2 up, 2 in 2024-03-19T03:24:56.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T03:24:56.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:24:56.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:56.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:24:56.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:24:56.565 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.563+0000 7f6e305d1700 1 -- 172.21.15.178:0/1384198601 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e28103ed0 msgr2=0x7f6e2810a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:24:56.565 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.563+0000 7f6e305d1700 1 --2- 172.21.15.178:0/1384198601 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e28103ed0 0x7f6e2810a780 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6e2000b0a0 tx=0x7f6e2000b3b0 comp rx=0 tx=0).stop 2024-03-19T03:24:56.565 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.563+0000 7f6e305d1700 1 -- 172.21.15.178:0/1384198601 shutdown_connections 2024-03-19T03:24:56.565 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.563+0000 7f6e305d1700 1 --2- 172.21.15.178:0/1384198601 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e28103ed0 0x7f6e2810a780 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:24:56.565 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.563+0000 7f6e305d1700 1 --2- 172.21.15.178:0/1384198601 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e28102330 0x7f6e28102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:24:56.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.563+0000 7f6e305d1700 1 --2- 172.21.15.178:0/1384198601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e28103530 0x7f6e28103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:24:56.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.563+0000 7f6e305d1700 1 -- 172.21.15.178:0/1384198601 >> 172.21.15.178:0/1384198601 conn(0x7f6e280fd8f0 msgr2=0x7f6e280ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:24:56.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.563+0000 7f6e305d1700 1 -- 172.21.15.178:0/1384198601 shutdown_connections 2024-03-19T03:24:56.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.563+0000 7f6e305d1700 1 -- 172.21.15.178:0/1384198601 wait complete. 2024-03-19T03:24:56.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.564+0000 7f6e305d1700 1 Processor -- start 2024-03-19T03:24:56.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.564+0000 7f6e305d1700 1 -- start start 2024-03-19T03:24:56.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.564+0000 7f6e305d1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e28102330 0x7f6e2819a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:24:56.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.564+0000 7f6e305d1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e28103530 0x7f6e2819abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:24:56.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.564+0000 7f6e305d1700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e28103ed0 0x7f6e281a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:24:56.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.564+0000 7f6e305d1700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2810df90 con 0x7f6e28103530 2024-03-19T03:24:56.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.564+0000 7f6e305d1700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2810de10 con 0x7f6e28102330 2024-03-19T03:24:56.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.564+0000 7f6e305d1700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2810e110 con 0x7f6e28103ed0 2024-03-19T03:24:56.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.564+0000 7f6e2eb6e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e28103ed0 0x7f6e281a1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:24:56.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.564+0000 7f6e2eb6e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e28103ed0 0x7f6e281a1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:43440/0 (socket says 172.21.15.178:43440) 2024-03-19T03:24:56.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.564+0000 7f6e2eb6e700 1 -- 172.21.15.178:0/1629049771 learned_addr learned my addr 172.21.15.178:0/1629049771 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-19T03:24:56.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.564+0000 7f6e2e36d700 1 --2- 172.21.15.178:0/1629049771 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e28102330 0x7f6e2819a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:24:56.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.564+0000 7f6e2eb6e700 1 -- 172.21.15.178:0/1629049771 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e28102330 msgr2=0x7f6e2819a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:24:56.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.564+0000 7f6e2eb6e700 1 --2- 172.21.15.178:0/1629049771 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e28102330 0x7f6e2819a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:24:56.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.564+0000 7f6e2eb6e700 1 -- 172.21.15.178:0/1629049771 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e28103530 msgr2=0x7f6e2819abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:24:56.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.564+0000 7f6e2eb6e700 1 --2- 172.21.15.178:0/1629049771 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e28103530 0x7f6e2819abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:24:56.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.564+0000 7f6e2eb6e700 1 -- 172.21.15.178:0/1629049771 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e281a2300 con 0x7f6e28103ed0 2024-03-19T03:24:56.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.564+0000 7f6e2eb6e700 1 --2- 172.21.15.178:0/1629049771 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e28103ed0 0x7f6e281a1c60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6e2000be60 tx=0x7f6e20007820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:24:56.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.565+0000 7f6e1f7fe700 1 -- 172.21.15.178:0/1629049771 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e20021020 con 0x7f6e28103ed0 2024-03-19T03:24:56.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.565+0000 7f6e305d1700 1 -- 172.21.15.178:0/1629049771 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e281a2590 con 0x7f6e28103ed0 2024-03-19T03:24:56.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.565+0000 7f6e1f7fe700 1 -- 172.21.15.178:0/1629049771 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e20005520 con 0x7f6e28103ed0 2024-03-19T03:24:56.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.565+0000 7f6e1f7fe700 1 -- 172.21.15.178:0/1629049771 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e2001ba00 con 0x7f6e28103ed0 2024-03-19T03:24:56.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.565+0000 7f6e305d1700 1 -- 172.21.15.178:0/1629049771 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e281a2ac0 con 0x7f6e28103ed0 2024-03-19T03:24:56.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.566+0000 7f6e305d1700 1 -- 172.21.15.178:0/1629049771 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e28068fd0 con 0x7f6e28103ed0 2024-03-19T03:24:56.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.567+0000 7f6e1f7fe700 1 -- 172.21.15.178:0/1629049771 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 15) ==== 137235+0+0 (secure 0 0 0) 0x7f6e20024020 con 0x7f6e28103ed0 2024-03-19T03:24:56.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.567+0000 7f6e1f7fe700 1 --2- 172.21.15.178:0/1629049771 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f6e140a2800 0x7f6e140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:24:56.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.567+0000 7f6e1f7fe700 1 -- 172.21.15.178:0/1629049771 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2309+0+0 (secure 0 0 0) 0x7f6e2000f020 con 0x7f6e28103ed0 2024-03-19T03:24:56.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.568+0000 7f6e2e36d700 1 --2- 172.21.15.178:0/1629049771 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f6e140a2800 0x7f6e140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:24:56.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.568+0000 7f6e2e36d700 1 --2- 172.21.15.178:0/1629049771 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f6e140a2800 0x7f6e140a4cc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6e24006fd0 tx=0x7f6e24008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:24:56.572 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.572+0000 7f6e1f7fe700 1 -- 172.21.15.178:0/1629049771 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6e2009cd40 con 0x7f6e28103ed0 2024-03-19T03:24:56.773 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:24:56.773+0000 7f6e305d1700 1 -- 172.21.15.178:0/1629049771 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6e28075c60 con 0x7f6e140a2800 2024-03-19T03:24:57.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:56 smithi117 ceph-mon[31838]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:24:57.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:56 smithi178 ceph-mon[67578]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:24:57.136 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:56 smithi119 ceph-mon[66861]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:24:58.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:57 smithi178 ceph-mon[67578]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:24:58.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:57 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:24:58.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:57 smithi178 ceph-mon[67578]: osdmap e14: 2 total, 2 up, 2 in 2024-03-19T03:24:58.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:57 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:24:58.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:57 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:58.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:57 smithi119 ceph-mon[66861]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:24:58.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:57 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:24:58.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:57 smithi119 ceph-mon[66861]: osdmap e14: 2 total, 2 up, 2 in 2024-03-19T03:24:58.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:57 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:24:58.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:57 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:58.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:57 smithi117 ceph-mon[31838]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:24:58.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:57 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:24:58.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:57 smithi117 ceph-mon[31838]: osdmap e14: 2 total, 2 up, 2 in 2024-03-19T03:24:58.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:57 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:24:58.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:57 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:24:59.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:58 smithi178 ceph-mon[67578]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:24:59.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:58 smithi119 ceph-mon[66861]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:24:59.291 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:58 smithi117 ceph-mon[31838]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:00.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:59 smithi178 ceph-mon[67578]: from='client.? 172.21.15.178:0/2379663950' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ede9edda-3b9f-41ca-ad04-5dcb738ab757"}]: dispatch 2024-03-19T03:25:00.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:59 smithi178 ceph-mon[67578]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ede9edda-3b9f-41ca-ad04-5dcb738ab757"}]: dispatch 2024-03-19T03:25:00.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:59 smithi178 ceph-mon[67578]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ede9edda-3b9f-41ca-ad04-5dcb738ab757"}]': finished 2024-03-19T03:25:00.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:59 smithi178 ceph-mon[67578]: osdmap e15: 3 total, 2 up, 3 in 2024-03-19T03:25:00.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:24:59 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T03:25:00.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:59 smithi119 ceph-mon[66861]: from='client.? 172.21.15.178:0/2379663950' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ede9edda-3b9f-41ca-ad04-5dcb738ab757"}]: dispatch 2024-03-19T03:25:00.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:59 smithi119 ceph-mon[66861]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ede9edda-3b9f-41ca-ad04-5dcb738ab757"}]: dispatch 2024-03-19T03:25:00.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:59 smithi119 ceph-mon[66861]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ede9edda-3b9f-41ca-ad04-5dcb738ab757"}]': finished 2024-03-19T03:25:00.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:59 smithi119 ceph-mon[66861]: osdmap e15: 3 total, 2 up, 3 in 2024-03-19T03:25:00.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:24:59 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T03:25:00.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:59 smithi117 ceph-mon[31838]: from='client.? 172.21.15.178:0/2379663950' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ede9edda-3b9f-41ca-ad04-5dcb738ab757"}]: dispatch 2024-03-19T03:25:00.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:59 smithi117 ceph-mon[31838]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ede9edda-3b9f-41ca-ad04-5dcb738ab757"}]: dispatch 2024-03-19T03:25:00.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:59 smithi117 ceph-mon[31838]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ede9edda-3b9f-41ca-ad04-5dcb738ab757"}]': finished 2024-03-19T03:25:00.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:59 smithi117 ceph-mon[31838]: osdmap e15: 3 total, 2 up, 3 in 2024-03-19T03:25:00.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:24:59 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T03:25:01.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:00 smithi178 ceph-mon[67578]: from='client.? 172.21.15.178:0/4139169408' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:25:01.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:00 smithi178 ceph-mon[67578]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:01.219 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:00 smithi119 ceph-mon[66861]: from='client.? 172.21.15.178:0/4139169408' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:25:01.220 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:00 smithi119 ceph-mon[66861]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:01.291 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:00 smithi117 ceph-mon[31838]: from='client.? 172.21.15.178:0/4139169408' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:25:01.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:00 smithi117 ceph-mon[31838]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:03.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:03 smithi119 ceph-mon[66861]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:03.525 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:03 smithi178 ceph-mon[67578]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:03.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:03 smithi117 ceph-mon[31838]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:05.341 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:05 smithi178 ceph-mon[67578]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:05.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:05 smithi117 ceph-mon[31838]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:05.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:05 smithi119 ceph-mon[66861]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:07.292 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:07 smithi178 ceph-mon[67578]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:07.293 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:07 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-19T03:25:07.293 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:07 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:25:07.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:07 smithi117 ceph-mon[31838]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:07.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:07 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-19T03:25:07.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:07 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:25:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:07 smithi119 ceph-mon[66861]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:07 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-19T03:25:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:07 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:25:08.291 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:08 smithi117 ceph-mon[31838]: Deploying daemon osd.2 on smithi178 2024-03-19T03:25:08.564 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:08 smithi178 ceph-mon[67578]: Deploying daemon osd.2 on smithi178 2024-03-19T03:25:08.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:08 smithi119 ceph-mon[66861]: Deploying daemon osd.2 on smithi178 2024-03-19T03:25:09.508 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:09 smithi178 ceph-mon[67578]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:09.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:09 smithi117 ceph-mon[31838]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:09.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:09 smithi119 ceph-mon[66861]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:11.402 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:11 smithi178 ceph-mon[67578]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:11.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:11 smithi117 ceph-mon[31838]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:11.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:11 smithi119 ceph-mon[66861]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:12.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:12 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:12.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:12 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:12.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:12 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:25:12.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:12 smithi178 ceph-mon[67578]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:12.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:12 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:12.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:12 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:12.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:12 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:25:12.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:12 smithi119 ceph-mon[66861]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:13.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:12 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:13.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:12 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:13.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:12 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:25:13.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:12 smithi117 ceph-mon[31838]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:15.012 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:25:15.011+0000 7f6e1f7fe700 1 -- 172.21.15.178:0/1629049771 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6e28075c60 con 0x7f6e140a2800 2024-03-19T03:25:15.013 INFO:teuthology.orchestra.run.smithi178.stdout:Created osd(s) 2 on host 'smithi178' 2024-03-19T03:25:15.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:25:15.013+0000 7f6e305d1700 1 -- 172.21.15.178:0/1629049771 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f6e140a2800 msgr2=0x7f6e140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:25:15.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:25:15.013+0000 7f6e305d1700 1 --2- 172.21.15.178:0/1629049771 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f6e140a2800 0x7f6e140a4cc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6e24006fd0 tx=0x7f6e24008040 comp rx=0 tx=0).stop 2024-03-19T03:25:15.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:25:15.013+0000 7f6e305d1700 1 -- 172.21.15.178:0/1629049771 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e28103ed0 msgr2=0x7f6e281a1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:25:15.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:25:15.013+0000 7f6e305d1700 1 --2- 172.21.15.178:0/1629049771 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e28103ed0 0x7f6e281a1c60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6e2000be60 tx=0x7f6e20007820 comp rx=0 tx=0).stop 2024-03-19T03:25:15.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:25:15.013+0000 7f6e305d1700 1 -- 172.21.15.178:0/1629049771 shutdown_connections 2024-03-19T03:25:15.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:25:15.013+0000 7f6e305d1700 1 --2- 172.21.15.178:0/1629049771 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f6e140a2800 0x7f6e140a4cc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:25:15.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:25:15.013+0000 7f6e305d1700 1 --2- 172.21.15.178:0/1629049771 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e28103ed0 0x7f6e281a1c60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:25:15.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:25:15.013+0000 7f6e305d1700 1 --2- 172.21.15.178:0/1629049771 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e28102330 0x7f6e2819a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:25:15.016 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:25:15.013+0000 7f6e305d1700 1 --2- 172.21.15.178:0/1629049771 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e28103530 0x7f6e2819abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:25:15.016 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:25:15.013+0000 7f6e305d1700 1 -- 172.21.15.178:0/1629049771 >> 172.21.15.178:0/1629049771 conn(0x7f6e280fd8f0 msgr2=0x7f6e28074540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:25:15.016 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:25:15.014+0000 7f6e305d1700 1 -- 172.21.15.178:0/1629049771 shutdown_connections 2024-03-19T03:25:15.016 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:25:15.014+0000 7f6e305d1700 1 -- 172.21.15.178:0/1629049771 wait complete. 2024-03-19T03:25:15.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:15 smithi178 ceph-mon[67578]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:15.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:15 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:15.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:15 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:15.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:15 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:15.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:15 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:15.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:15 smithi117 ceph-mon[31838]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:15.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:15 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:15.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:15 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:15.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:15 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:15.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:15 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:15.652 DEBUG:teuthology.orchestra.run.smithi178:osd.2> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.2.service 2024-03-19T03:25:15.655 INFO:tasks.cephadm:Deploying osd.3 on smithi117 with /dev/vg_nvme/lv_3... 2024-03-19T03:25:15.655 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-19T03:25:15.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:15 smithi119 ceph-mon[66861]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:15.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:15 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:15.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:15 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:15.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:15 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:15.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:15 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:16.091 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:25:16.502 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:16 smithi117 ceph-mon[31838]: from='osd.2 [v2:172.21.15.178:6800/3733812989,v1:172.21.15.178:6801/3733812989]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T03:25:16.503 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:16 smithi117 ceph-mon[31838]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T03:25:16.563 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:16 smithi178 ceph-mon[67578]: from='osd.2 [v2:172.21.15.178:6800/3733812989,v1:172.21.15.178:6801/3733812989]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T03:25:16.563 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:16 smithi178 ceph-mon[67578]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T03:25:16.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:16 smithi119 ceph-mon[66861]: from='osd.2 [v2:172.21.15.178:6800/3733812989,v1:172.21.15.178:6801/3733812989]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T03:25:16.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:16 smithi119 ceph-mon[66861]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T03:25:17.390 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:17 smithi117 ceph-mon[31838]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:17.390 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:17 smithi117 ceph-mon[31838]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-19T03:25:17.390 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:17 smithi117 ceph-mon[31838]: from='osd.2 [v2:172.21.15.178:6800/3733812989,v1:172.21.15.178:6801/3733812989]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-19T03:25:17.391 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:17 smithi117 ceph-mon[31838]: osdmap e16: 3 total, 2 up, 3 in 2024-03-19T03:25:17.391 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:17 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T03:25:17.391 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:17 smithi117 ceph-mon[31838]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-19T03:25:17.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:17 smithi178 ceph-mon[67578]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:17.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:17 smithi178 ceph-mon[67578]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-19T03:25:17.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:17 smithi178 ceph-mon[67578]: from='osd.2 [v2:172.21.15.178:6800/3733812989,v1:172.21.15.178:6801/3733812989]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-19T03:25:17.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:17 smithi178 ceph-mon[67578]: osdmap e16: 3 total, 2 up, 3 in 2024-03-19T03:25:17.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:17 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T03:25:17.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:17 smithi178 ceph-mon[67578]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-19T03:25:17.597 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 19 03:25:17 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-2[72716]: 2024-03-19T03:25:17.257+0000 7fc572fd5700 -1 osd.2 0 waiting for initial osdmap 2024-03-19T03:25:17.597 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 19 03:25:17 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-2[72716]: 2024-03-19T03:25:17.261+0000 7fc56edc8700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T03:25:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:17 smithi119 ceph-mon[66861]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:17 smithi119 ceph-mon[66861]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-19T03:25:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:17 smithi119 ceph-mon[66861]: from='osd.2 [v2:172.21.15.178:6800/3733812989,v1:172.21.15.178:6801/3733812989]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-19T03:25:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:17 smithi119 ceph-mon[66861]: osdmap e16: 3 total, 2 up, 3 in 2024-03-19T03:25:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:17 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T03:25:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:17 smithi119 ceph-mon[66861]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-19T03:25:18.474 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:18 smithi178 ceph-mon[67578]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-03-19T03:25:18.474 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:18 smithi178 ceph-mon[67578]: osdmap e17: 3 total, 2 up, 3 in 2024-03-19T03:25:18.474 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:18 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T03:25:18.474 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:18 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T03:25:18.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:18 smithi117 ceph-mon[31838]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-03-19T03:25:18.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:18 smithi117 ceph-mon[31838]: osdmap e17: 3 total, 2 up, 3 in 2024-03-19T03:25:18.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:18 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T03:25:18.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:18 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T03:25:18.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:18 smithi119 ceph-mon[66861]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-03-19T03:25:18.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:18 smithi119 ceph-mon[66861]: osdmap e17: 3 total, 2 up, 3 in 2024-03-19T03:25:18.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:18 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T03:25:18.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:18 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T03:25:18.838 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-19T03:25:18.838 INFO:teuthology.orchestra.run.smithi117.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-19T03:25:18.838 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10+0 records in 2024-03-19T03:25:18.838 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2024-03-19T03:25:18.838 INFO:teuthology.orchestra.run.smithi117.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0341178 s, 307 MB/s 2024-03-19T03:25:18.838 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping successful for: 2024-03-19T03:25:19.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:19 smithi117 ceph-mon[31838]: purged_snaps scrub starts 2024-03-19T03:25:19.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:19 smithi117 ceph-mon[31838]: purged_snaps scrub ok 2024-03-19T03:25:19.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:19 smithi117 ceph-mon[31838]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:19.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:19 smithi117 ceph-mon[31838]: osd.2 [v2:172.21.15.178:6800/3733812989,v1:172.21.15.178:6801/3733812989] boot 2024-03-19T03:25:19.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:19 smithi117 ceph-mon[31838]: osdmap e18: 3 total, 3 up, 3 in 2024-03-19T03:25:19.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:19 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T03:25:19.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:19 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-19T03:25:19.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:19 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:19.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:19 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:19.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:19 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:25:19.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:19 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:19.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:19 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:25:19.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:19 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:25:19.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:19 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:19.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:19 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:25:19.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:19 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:25:19.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:19 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:25:19.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:19 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:19.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:19 smithi178 ceph-mon[67578]: purged_snaps scrub starts 2024-03-19T03:25:19.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:19 smithi178 ceph-mon[67578]: purged_snaps scrub ok 2024-03-19T03:25:19.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:19 smithi178 ceph-mon[67578]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:19.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:19 smithi178 ceph-mon[67578]: osd.2 [v2:172.21.15.178:6800/3733812989,v1:172.21.15.178:6801/3733812989] boot 2024-03-19T03:25:19.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:19 smithi178 ceph-mon[67578]: osdmap e18: 3 total, 3 up, 3 in 2024-03-19T03:25:19.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:19 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T03:25:19.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:19 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-19T03:25:19.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:19 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:19.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:19 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:19.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:19 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:25:19.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:19 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:19.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:19 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:25:19.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:19 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:25:19.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:19 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:19.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:19 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:25:19.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:19 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:25:19.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:19 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:25:19.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:19 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:19.662 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi117:vg_nvme/lv_3 2024-03-19T03:25:19.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:19 smithi119 ceph-mon[66861]: purged_snaps scrub starts 2024-03-19T03:25:19.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:19 smithi119 ceph-mon[66861]: purged_snaps scrub ok 2024-03-19T03:25:19.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:19 smithi119 ceph-mon[66861]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T03:25:19.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:19 smithi119 ceph-mon[66861]: osd.2 [v2:172.21.15.178:6800/3733812989,v1:172.21.15.178:6801/3733812989] boot 2024-03-19T03:25:19.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:19 smithi119 ceph-mon[66861]: osdmap e18: 3 total, 3 up, 3 in 2024-03-19T03:25:19.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:19 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T03:25:19.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:19 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-19T03:25:19.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:19 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:19.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:19 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:19.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:19 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:25:19.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:19 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:19.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:19 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:25:19.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:19 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:25:19.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:19 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:19.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:19 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:25:19.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:19 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:25:19.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:19 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:25:19.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:19 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:20.099 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:25:20.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:20 smithi117 ceph-mon[31838]: Detected new or changed devices on smithi178 2024-03-19T03:25:20.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:20 smithi117 ceph-mon[31838]: Adjusting osd_memory_target on smithi178 to 17080M 2024-03-19T03:25:20.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:20 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-19T03:25:20.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:20 smithi117 ceph-mon[31838]: osdmap e19: 3 total, 3 up, 3 in 2024-03-19T03:25:20.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:20 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T03:25:20.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:20 smithi178 ceph-mon[67578]: Detected new or changed devices on smithi178 2024-03-19T03:25:20.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:20 smithi178 ceph-mon[67578]: Adjusting osd_memory_target on smithi178 to 17080M 2024-03-19T03:25:20.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:20 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-19T03:25:20.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:20 smithi178 ceph-mon[67578]: osdmap e19: 3 total, 3 up, 3 in 2024-03-19T03:25:20.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:20 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T03:25:20.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:20 smithi119 ceph-mon[66861]: Detected new or changed devices on smithi178 2024-03-19T03:25:20.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:20 smithi119 ceph-mon[66861]: Adjusting osd_memory_target on smithi178 to 17080M 2024-03-19T03:25:20.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:20 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-19T03:25:20.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:20 smithi119 ceph-mon[66861]: osdmap e19: 3 total, 3 up, 3 in 2024-03-19T03:25:20.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:20 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T03:25:21.305 INFO:journalctl@ceph.osd.0.smithi117.stdout:Mar 19 03:25:21 smithi117 sudo[50876]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T03:25:21.305 INFO:journalctl@ceph.osd.0.smithi117.stdout:Mar 19 03:25:21 smithi117 sudo[50876]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T03:25:21.305 INFO:journalctl@ceph.osd.0.smithi117.stdout:Mar 19 03:25:21 smithi117 sudo[50876]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T03:25:21.306 INFO:journalctl@ceph.osd.0.smithi117.stdout:Mar 19 03:25:21 smithi117 sudo[50876]: pam_unix(sudo:session): session closed for user root 2024-03-19T03:25:21.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:21 smithi178 ceph-mon[67578]: pgmap v90: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:21.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:21 smithi178 ceph-mon[67578]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T03:25:21.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:21 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-19T03:25:21.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:21 smithi178 ceph-mon[67578]: osdmap e20: 3 total, 3 up, 3 in 2024-03-19T03:25:21.601 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:21 smithi117 ceph-mon[31838]: pgmap v90: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:21.601 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:21 smithi117 ceph-mon[31838]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T03:25:21.601 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:21 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-19T03:25:21.601 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:21 smithi117 ceph-mon[31838]: osdmap e20: 3 total, 3 up, 3 in 2024-03-19T03:25:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:21 smithi119 ceph-mon[66861]: pgmap v90: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:21 smithi119 ceph-mon[66861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T03:25:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:21 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-19T03:25:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:21 smithi119 ceph-mon[66861]: osdmap e20: 3 total, 3 up, 3 in 2024-03-19T03:25:21.892 INFO:journalctl@ceph.osd.0.smithi117.stdout:Mar 19 03:25:21 smithi117 sudo[50882]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T03:25:21.892 INFO:journalctl@ceph.osd.0.smithi117.stdout:Mar 19 03:25:21 smithi117 sudo[50882]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T03:25:21.892 INFO:journalctl@ceph.osd.0.smithi117.stdout:Mar 19 03:25:21 smithi117 sudo[50882]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T03:25:21.892 INFO:journalctl@ceph.osd.0.smithi117.stdout:Mar 19 03:25:21 smithi117 sudo[50882]: pam_unix(sudo:session): session closed for user root 2024-03-19T03:25:22.136 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:25:21 smithi119 sudo[75445]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T03:25:22.136 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:25:21 smithi119 sudo[75445]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T03:25:22.136 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:25:21 smithi119 sudo[75445]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T03:25:22.136 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:25:21 smithi119 sudo[75445]: pam_unix(sudo:session): session closed for user root 2024-03-19T03:25:22.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.192+0000 7f84b3a00700 1 -- 172.21.15.117:0/1079689662 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84ac101520 msgr2=0x7f84ac1019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:25:22.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.192+0000 7f84b3a00700 1 --2- 172.21.15.117:0/1079689662 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84ac101520 0x7f84ac1019a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8498009990 tx=0x7f8498009ca0 comp rx=0 tx=0).stop 2024-03-19T03:25:22.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.192+0000 7f84b3a00700 1 -- 172.21.15.117:0/1079689662 shutdown_connections 2024-03-19T03:25:22.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.192+0000 7f84b3a00700 1 --2- 172.21.15.117:0/1079689662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84ac101ee0 0x7f84ac102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:25:22.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.192+0000 7f84b3a00700 1 --2- 172.21.15.117:0/1079689662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84ac100bc0 0x7f84ac100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:25:22.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.192+0000 7f84b3a00700 1 --2- 172.21.15.117:0/1079689662 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84ac101520 0x7f84ac1019a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:25:22.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.192+0000 7f84b3a00700 1 -- 172.21.15.117:0/1079689662 >> 172.21.15.117:0/1079689662 conn(0x7f84ac0fc7e0 msgr2=0x7f84ac0fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:25:22.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.192+0000 7f84b3a00700 1 -- 172.21.15.117:0/1079689662 shutdown_connections 2024-03-19T03:25:22.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.192+0000 7f84b3a00700 1 -- 172.21.15.117:0/1079689662 wait complete. 2024-03-19T03:25:22.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.193+0000 7f84b3a00700 1 Processor -- start 2024-03-19T03:25:22.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.194+0000 7f84b3a00700 1 -- start start 2024-03-19T03:25:22.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.194+0000 7f84b3a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84ac100bc0 0x7f84ac19a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:25:22.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.194+0000 7f84b3a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84ac101520 0x7f84ac19ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:25:22.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.194+0000 7f84b3a00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84ac101ee0 0x7f84ac1a1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:25:22.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.194+0000 7f84b3a00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f84ac110110 con 0x7f84ac100bc0 2024-03-19T03:25:22.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.194+0000 7f84b3a00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f84ac10ff90 con 0x7f84ac101520 2024-03-19T03:25:22.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.194+0000 7f84b3a00700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f84ac110290 con 0x7f84ac101ee0 2024-03-19T03:25:22.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.194+0000 7f84b1000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84ac100bc0 0x7f84ac19a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:25:22.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.194+0000 7f84abe00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84ac101520 0x7f84ac19ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:25:22.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.194+0000 7f84b1000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84ac100bc0 0x7f84ac19a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:51226/0 (socket says 172.21.15.117:51226) 2024-03-19T03:25:22.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.194+0000 7f84b1000700 1 -- 172.21.15.117:0/2894125098 learned_addr learned my addr 172.21.15.117:0/2894125098 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:25:22.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.194+0000 7f84b1a00700 1 --2- 172.21.15.117:0/2894125098 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84ac101ee0 0x7f84ac1a1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:25:22.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.195+0000 7f84abe00700 1 -- 172.21.15.117:0/2894125098 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84ac101ee0 msgr2=0x7f84ac1a1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:25:22.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.195+0000 7f84abe00700 1 --2- 172.21.15.117:0/2894125098 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84ac101ee0 0x7f84ac1a1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:25:22.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.195+0000 7f84abe00700 1 -- 172.21.15.117:0/2894125098 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84ac100bc0 msgr2=0x7f84ac19a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:25:22.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.195+0000 7f84abe00700 1 --2- 172.21.15.117:0/2894125098 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84ac100bc0 0x7f84ac19a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:25:22.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.195+0000 7f84abe00700 1 -- 172.21.15.117:0/2894125098 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84ac1a2470 con 0x7f84ac101520 2024-03-19T03:25:22.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.195+0000 7f84b1000700 1 --2- 172.21.15.117:0/2894125098 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84ac100bc0 0x7f84ac19a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:25:22.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.195+0000 7f84b1a00700 1 --2- 172.21.15.117:0/2894125098 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84ac101ee0 0x7f84ac1a1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:25:22.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.195+0000 7f84abe00700 1 --2- 172.21.15.117:0/2894125098 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84ac101520 0x7f84ac19ad50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f84980098e0 tx=0x7f849800b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:25:22.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.195+0000 7f84a9600700 1 -- 172.21.15.117:0/2894125098 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8498026070 con 0x7f84ac101520 2024-03-19T03:25:22.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.195+0000 7f84a9600700 1 -- 172.21.15.117:0/2894125098 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8498003d70 con 0x7f84ac101520 2024-03-19T03:25:22.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.195+0000 7f84a9600700 1 -- 172.21.15.117:0/2894125098 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8498021410 con 0x7f84ac101520 2024-03-19T03:25:22.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.195+0000 7f84b3a00700 1 -- 172.21.15.117:0/2894125098 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84ac1a2700 con 0x7f84ac101520 2024-03-19T03:25:22.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.196+0000 7f84b3a00700 1 -- 172.21.15.117:0/2894125098 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84ac1a2bb0 con 0x7f84ac101520 2024-03-19T03:25:22.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.197+0000 7f84b3a00700 1 -- 172.21.15.117:0/2894125098 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84ac068fd0 con 0x7f84ac101520 2024-03-19T03:25:22.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.198+0000 7f84a9600700 1 -- 172.21.15.117:0/2894125098 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 15) ==== 137235+0+0 (secure 0 0 0) 0x7f8498029070 con 0x7f84ac101520 2024-03-19T03:25:22.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.199+0000 7f84a9600700 1 --2- 172.21.15.117:0/2894125098 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f84940a2880 0x7f84940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:25:22.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.199+0000 7f84a9600700 1 -- 172.21.15.117:0/2894125098 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7f84980ce310 con 0x7f84ac101520 2024-03-19T03:25:22.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.200+0000 7f84b1000700 1 --2- 172.21.15.117:0/2894125098 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f84940a2880 0x7f84940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:25:22.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.200+0000 7f84b1000700 1 --2- 172.21.15.117:0/2894125098 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f84940a2880 0x7f84940a4d40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f84a0009c10 tx=0x7f84a000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:25:22.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.204+0000 7f84a9600700 1 -- 172.21.15.117:0/2894125098 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f849805d740 con 0x7f84ac101520 2024-03-19T03:25:22.421 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:22.420+0000 7f84b3a00700 1 -- 172.21.15.117:0/2894125098 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f84ac062750 con 0x7f84940a2880 2024-03-19T03:25:22.484 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:25:22 smithi119 sudo[75448]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T03:25:22.484 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:25:22 smithi119 sudo[75448]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T03:25:22.484 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:25:22 smithi119 sudo[75448]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T03:25:22.485 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:25:22 smithi119 sudo[75448]: pam_unix(sudo:session): session closed for user root 2024-03-19T03:25:22.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:22 smithi119 ceph-mon[66861]: osdmap e21: 3 total, 3 up, 3 in 2024-03-19T03:25:22.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:22 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:22.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:22 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:22.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:22 smithi117 ceph-mon[31838]: osdmap e21: 3 total, 3 up, 3 in 2024-03-19T03:25:22.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:22 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:22.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:22 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:22.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:22 smithi178 ceph-mon[67578]: osdmap e21: 3 total, 3 up, 3 in 2024-03-19T03:25:22.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:22 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:22.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:22 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:22.596 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 19 03:25:22 smithi178 sudo[76357]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T03:25:22.597 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 19 03:25:22 smithi178 sudo[76357]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T03:25:22.597 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 19 03:25:22 smithi178 sudo[76357]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T03:25:22.597 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 19 03:25:22 smithi178 sudo[76357]: pam_unix(sudo:session): session closed for user root 2024-03-19T03:25:23.096 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 19 03:25:22 smithi178 sudo[76360]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T03:25:23.096 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 19 03:25:22 smithi178 sudo[76360]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T03:25:23.096 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 19 03:25:22 smithi178 sudo[76360]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T03:25:23.096 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 19 03:25:22 smithi178 sudo[76360]: pam_unix(sudo:session): session closed for user root 2024-03-19T03:25:23.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:23 smithi117 sudo[51048]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T03:25:23.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:23 smithi117 sudo[51048]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T03:25:23.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:23 smithi117 sudo[51048]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T03:25:23.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:23 smithi117 sudo[51048]: pam_unix(sudo:session): session closed for user root 2024-03-19T03:25:23.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:23.875+0000 7f84a9600700 1 -- 172.21.15.117:0/2894125098 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f8498009d10 con 0x7f84ac101520 2024-03-19T03:25:24.124 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:23 smithi178 ceph-mon[67578]: pgmap v93: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:24.125 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:23 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:25:24.125 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:23 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:25:24.125 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:23 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:25:24.125 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:23 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:24.125 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:23 smithi178 ceph-mon[67578]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T03:25:24.125 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 sudo[76364]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T03:25:24.125 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 sudo[76364]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T03:25:24.125 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 sudo[76364]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T03:25:24.125 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 sudo[76364]: pam_unix(sudo:session): session closed for user root 2024-03-19T03:25:24.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:23 smithi117 sudo[51069]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T03:25:24.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:23 smithi117 sudo[51069]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T03:25:24.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:23 smithi117 sudo[51069]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T03:25:24.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:23 smithi117 sudo[51069]: pam_unix(sudo:session): session closed for user root 2024-03-19T03:25:24.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:23 smithi117 ceph-mon[31838]: pgmap v93: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:24.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:23 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:25:24.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:23 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:25:24.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:23 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:25:24.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:23 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:24.212 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:23 smithi117 ceph-mon[31838]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T03:25:24.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:23 smithi119 ceph-mon[66861]: pgmap v93: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:24.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:23 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:25:24.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:23 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:25:24.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:23 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:25:24.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:23 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:24.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:23 smithi119 ceph-mon[66861]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T03:25:24.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 sudo[76367]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T03:25:24.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 sudo[76367]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T03:25:24.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 sudo[76367]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T03:25:24.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 sudo[76367]: pam_unix(sudo:session): session closed for user root 2024-03-19T03:25:24.874 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 sudo[75451]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T03:25:24.874 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 sudo[75451]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T03:25:24.874 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 sudo[75451]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T03:25:24.874 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 sudo[75451]: pam_unix(sudo:session): session closed for user root 2024-03-19T03:25:24.874 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 sudo[75454]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T03:25:24.874 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 sudo[75454]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T03:25:24.874 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 sudo[75454]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T03:25:24.874 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 sudo[75454]: pam_unix(sudo:session): session closed for user root 2024-03-19T03:25:25.150 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:24 smithi117 ceph-mon[31838]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:25:25.150 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:24 smithi117 ceph-mon[31838]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T03:25:25.150 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:24 smithi117 ceph-mon[31838]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-19T03:25:25.150 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:24 smithi117 ceph-mon[31838]: Cluster is now healthy 2024-03-19T03:25:25.150 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:24 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T03:25:25.150 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:24 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:25:25.151 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:24 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:25:25.151 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:24 smithi117 ceph-mon[31838]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T03:25:25.151 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:24 smithi117 ceph-mon[31838]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-19T03:25:25.151 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:24 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T03:25:25.151 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:24 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:25:25.151 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:24 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:25:25.151 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:24 smithi117 ceph-mon[31838]: pgmap v94: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:25.151 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:24 smithi117 ceph-mon[31838]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T03:25:25.151 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:24 smithi117 ceph-mon[31838]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T03:25:25.151 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:24 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T03:25:25.151 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:24 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:25:25.151 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:24 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:25:25.151 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:24 smithi117 ceph-mon[31838]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T03:25:25.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 ceph-mon[66861]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:25:25.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 ceph-mon[66861]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T03:25:25.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 ceph-mon[66861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-19T03:25:25.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 ceph-mon[66861]: Cluster is now healthy 2024-03-19T03:25:25.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T03:25:25.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:25:25.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:25:25.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 ceph-mon[66861]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T03:25:25.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 ceph-mon[66861]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-19T03:25:25.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T03:25:25.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:25:25.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:25:25.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 ceph-mon[66861]: pgmap v94: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:25.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 ceph-mon[66861]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T03:25:25.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 ceph-mon[66861]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T03:25:25.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T03:25:25.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:25:25.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:25:25.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:24 smithi119 ceph-mon[66861]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T03:25:25.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 ceph-mon[67578]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:25:25.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 ceph-mon[67578]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T03:25:25.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 ceph-mon[67578]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-19T03:25:25.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 ceph-mon[67578]: Cluster is now healthy 2024-03-19T03:25:25.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T03:25:25.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:25:25.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:25:25.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 ceph-mon[67578]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T03:25:25.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 ceph-mon[67578]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-19T03:25:25.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T03:25:25.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:25:25.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:25:25.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 ceph-mon[67578]: pgmap v94: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:25.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 ceph-mon[67578]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T03:25:25.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 ceph-mon[67578]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T03:25:25.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T03:25:25.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T03:25:25.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T03:25:25.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:24 smithi178 ceph-mon[67578]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T03:25:27.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:27 smithi117 ceph-mon[31838]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 7.4 KiB/s rd, 376 KiB/s wr, 25 op/s 2024-03-19T03:25:27.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:27 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3873002785' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f81c41e1-3a69-40db-a90f-58151286b081"}]: dispatch 2024-03-19T03:25:27.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:27 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3873002785' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f81c41e1-3a69-40db-a90f-58151286b081"}]': finished 2024-03-19T03:25:27.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:27 smithi117 ceph-mon[31838]: osdmap e22: 4 total, 3 up, 4 in 2024-03-19T03:25:27.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:27 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:25:27.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:27 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1935358120' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:25:27.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:27 smithi178 ceph-mon[67578]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 7.4 KiB/s rd, 376 KiB/s wr, 25 op/s 2024-03-19T03:25:27.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:27 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3873002785' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f81c41e1-3a69-40db-a90f-58151286b081"}]: dispatch 2024-03-19T03:25:27.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:27 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3873002785' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f81c41e1-3a69-40db-a90f-58151286b081"}]': finished 2024-03-19T03:25:27.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:27 smithi178 ceph-mon[67578]: osdmap e22: 4 total, 3 up, 4 in 2024-03-19T03:25:27.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:27 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:25:27.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:27 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1935358120' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:25:27.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:27 smithi119 ceph-mon[66861]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 7.4 KiB/s rd, 376 KiB/s wr, 25 op/s 2024-03-19T03:25:27.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:27 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3873002785' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f81c41e1-3a69-40db-a90f-58151286b081"}]: dispatch 2024-03-19T03:25:27.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:27 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3873002785' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f81c41e1-3a69-40db-a90f-58151286b081"}]': finished 2024-03-19T03:25:27.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:27 smithi119 ceph-mon[66861]: osdmap e22: 4 total, 3 up, 4 in 2024-03-19T03:25:27.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:27 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:25:27.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:27 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1935358120' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:25:29.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:29 smithi117 ceph-mon[31838]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 5.2 KiB/s rd, 266 KiB/s wr, 17 op/s 2024-03-19T03:25:29.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:29 smithi178 ceph-mon[67578]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 5.2 KiB/s rd, 266 KiB/s wr, 17 op/s 2024-03-19T03:25:29.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:29 smithi119 ceph-mon[66861]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 5.2 KiB/s rd, 266 KiB/s wr, 17 op/s 2024-03-19T03:25:31.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:31 smithi117 ceph-mon[31838]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 4.1 KiB/s rd, 206 KiB/s wr, 13 op/s 2024-03-19T03:25:31.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:31 smithi178 ceph-mon[67578]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 4.1 KiB/s rd, 206 KiB/s wr, 13 op/s 2024-03-19T03:25:31.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:31 smithi119 ceph-mon[66861]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 4.1 KiB/s rd, 206 KiB/s wr, 13 op/s 2024-03-19T03:25:33.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:33 smithi117 ceph-mon[31838]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-19T03:25:33.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:33 smithi178 ceph-mon[67578]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-19T03:25:33.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:33 smithi119 ceph-mon[66861]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-19T03:25:35.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:35 smithi117 ceph-mon[31838]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-19T03:25:35.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:35 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-19T03:25:35.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:35 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:25:35.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:35 smithi178 ceph-mon[67578]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-19T03:25:35.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:35 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-19T03:25:35.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:35 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:25:35.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:35 smithi119 ceph-mon[66861]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-03-19T03:25:35.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:35 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-19T03:25:35.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:35 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:25:36.514 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:36 smithi117 ceph-mon[31838]: Deploying daemon osd.3 on smithi117 2024-03-19T03:25:36.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:36 smithi178 ceph-mon[67578]: Deploying daemon osd.3 on smithi117 2024-03-19T03:25:36.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:36 smithi119 ceph-mon[66861]: Deploying daemon osd.3 on smithi117 2024-03-19T03:25:37.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:37 smithi117 ceph-mon[31838]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:37.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:37 smithi178 ceph-mon[67578]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:37.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:37 smithi119 ceph-mon[66861]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:39.434 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:39 smithi117 ceph-mon[31838]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:39.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:39 smithi178 ceph-mon[67578]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:39.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:39 smithi119 ceph-mon[66861]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:41.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:41 smithi117 ceph-mon[31838]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:41.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:41 smithi178 ceph-mon[67578]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:41.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:41 smithi119 ceph-mon[66861]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:43.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:43 smithi117 ceph-mon[31838]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:43.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:43 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:43.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:43 smithi178 ceph-mon[67578]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:43.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:43 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:43.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:43 smithi119 ceph-mon[66861]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:43.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:43 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:44.522 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:44 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:25:44.523 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:44 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:44.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:44 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:25:44.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:44 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:44.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:44 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:25:44.741 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:44 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:45.355 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:45 smithi117 ceph-mon[31838]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:45.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:45 smithi178 ceph-mon[67578]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:45.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:45 smithi119 ceph-mon[66861]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:47.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:47 smithi117 ceph-mon[31838]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:47.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:47 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:47.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:47 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:47.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:47 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:25:47.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:47 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:25:47.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:47 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:47.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:47 smithi117 ceph-mon[31838]: from='osd.3 [v2:172.21.15.117:6810/2178310135,v1:172.21.15.117:6811/2178310135]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-19T03:25:47.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:47 smithi178 ceph-mon[67578]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:47.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:47 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:47.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:47 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:47.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:47 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:25:47.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:47 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:25:47.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:47 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:47.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:47 smithi178 ceph-mon[67578]: from='osd.3 [v2:172.21.15.117:6810/2178310135,v1:172.21.15.117:6811/2178310135]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-19T03:25:47.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:47 smithi119 ceph-mon[66861]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:47.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:47 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:47.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:47 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:47.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:47 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:25:47.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:47 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:25:47.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:47 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:47.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:47 smithi119 ceph-mon[66861]: from='osd.3 [v2:172.21.15.117:6810/2178310135,v1:172.21.15.117:6811/2178310135]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-19T03:25:48.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:48.212+0000 7f84a9600700 1 -- 172.21.15.117:0/2894125098 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f84ac062750 con 0x7f84940a2880 2024-03-19T03:25:48.213 INFO:teuthology.orchestra.run.smithi117.stdout:Created osd(s) 3 on host 'smithi117' 2024-03-19T03:25:48.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:48.216+0000 7f84b3a00700 1 -- 172.21.15.117:0/2894125098 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f84940a2880 msgr2=0x7f84940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:25:48.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:48.216+0000 7f84b3a00700 1 --2- 172.21.15.117:0/2894125098 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f84940a2880 0x7f84940a4d40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f84a0009c10 tx=0x7f84a000b040 comp rx=0 tx=0).stop 2024-03-19T03:25:48.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:48.216+0000 7f84b3a00700 1 -- 172.21.15.117:0/2894125098 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84ac101520 msgr2=0x7f84ac19ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:25:48.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:48.216+0000 7f84b3a00700 1 --2- 172.21.15.117:0/2894125098 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84ac101520 0x7f84ac19ad50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f84980098e0 tx=0x7f849800b980 comp rx=0 tx=0).stop 2024-03-19T03:25:48.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:48.216+0000 7f84b3a00700 1 -- 172.21.15.117:0/2894125098 shutdown_connections 2024-03-19T03:25:48.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:48.216+0000 7f84b3a00700 1 --2- 172.21.15.117:0/2894125098 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f84940a2880 0x7f84940a4d40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:25:48.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:48.216+0000 7f84b3a00700 1 --2- 172.21.15.117:0/2894125098 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84ac101ee0 0x7f84ac1a1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:25:48.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:48.216+0000 7f84b3a00700 1 --2- 172.21.15.117:0/2894125098 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84ac100bc0 0x7f84ac19a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:25:48.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:48.216+0000 7f84b3a00700 1 --2- 172.21.15.117:0/2894125098 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84ac101520 0x7f84ac19ad50 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:25:48.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:48.216+0000 7f84b3a00700 1 -- 172.21.15.117:0/2894125098 >> 172.21.15.117:0/2894125098 conn(0x7f84ac0fc7e0 msgr2=0x7f84ac0fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:25:48.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:48.217+0000 7f84b3a00700 1 -- 172.21.15.117:0/2894125098 shutdown_connections 2024-03-19T03:25:48.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:25:48.217+0000 7f84b3a00700 1 -- 172.21.15.117:0/2894125098 wait complete. 2024-03-19T03:25:48.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:48 smithi178 ceph-mon[67578]: from='osd.3 [v2:172.21.15.117:6810/2178310135,v1:172.21.15.117:6811/2178310135]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-19T03:25:48.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:48 smithi178 ceph-mon[67578]: osdmap e23: 4 total, 3 up, 4 in 2024-03-19T03:25:48.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:48 smithi178 ceph-mon[67578]: from='osd.3 [v2:172.21.15.117:6810/2178310135,v1:172.21.15.117:6811/2178310135]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-19T03:25:48.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:48 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:25:48.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:48 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:48.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:48 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:25:48.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:48 smithi178 ceph-mon[67578]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:48.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:48 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:48.982 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:48 smithi117 ceph-mon[31838]: from='osd.3 [v2:172.21.15.117:6810/2178310135,v1:172.21.15.117:6811/2178310135]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-19T03:25:48.982 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:48 smithi117 ceph-mon[31838]: osdmap e23: 4 total, 3 up, 4 in 2024-03-19T03:25:48.982 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:48 smithi117 ceph-mon[31838]: from='osd.3 [v2:172.21.15.117:6810/2178310135,v1:172.21.15.117:6811/2178310135]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-19T03:25:48.982 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:48 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:25:48.982 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:48 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:48.983 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:48 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:25:48.983 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:48 smithi117 ceph-mon[31838]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:48.983 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:48 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:48.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:48 smithi119 ceph-mon[66861]: from='osd.3 [v2:172.21.15.117:6810/2178310135,v1:172.21.15.117:6811/2178310135]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-19T03:25:48.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:48 smithi119 ceph-mon[66861]: osdmap e23: 4 total, 3 up, 4 in 2024-03-19T03:25:48.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:48 smithi119 ceph-mon[66861]: from='osd.3 [v2:172.21.15.117:6810/2178310135,v1:172.21.15.117:6811/2178310135]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-19T03:25:48.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:48 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:25:48.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:48 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:48.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:48 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:25:48.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:48 smithi119 ceph-mon[66861]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-19T03:25:48.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:48 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:49.025 DEBUG:teuthology.orchestra.run.smithi117:osd.3> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.3.service 2024-03-19T03:25:49.030 INFO:tasks.cephadm:Deploying osd.4 on smithi119 with /dev/vg_nvme/lv_3... 2024-03-19T03:25:49.030 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-19T03:25:49.273 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T03:25:49.657 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:49 smithi119 ceph-mon[66861]: from='osd.3 [v2:172.21.15.117:6810/2178310135,v1:172.21.15.117:6811/2178310135]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-19T03:25:49.658 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:49 smithi119 ceph-mon[66861]: osdmap e24: 4 total, 3 up, 4 in 2024-03-19T03:25:49.658 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:49 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:25:49.658 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:49 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:25:49.658 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:49 smithi119 ceph-mon[66861]: osd.3 [v2:172.21.15.117:6810/2178310135,v1:172.21.15.117:6811/2178310135] boot 2024-03-19T03:25:49.658 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:49 smithi119 ceph-mon[66861]: osdmap e25: 4 total, 4 up, 4 in 2024-03-19T03:25:49.658 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:49 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:25:49.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:49 smithi178 ceph-mon[67578]: from='osd.3 [v2:172.21.15.117:6810/2178310135,v1:172.21.15.117:6811/2178310135]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-19T03:25:49.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:49 smithi178 ceph-mon[67578]: osdmap e24: 4 total, 3 up, 4 in 2024-03-19T03:25:49.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:49 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:25:49.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:49 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:25:49.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:49 smithi178 ceph-mon[67578]: osd.3 [v2:172.21.15.117:6810/2178310135,v1:172.21.15.117:6811/2178310135] boot 2024-03-19T03:25:49.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:49 smithi178 ceph-mon[67578]: osdmap e25: 4 total, 4 up, 4 in 2024-03-19T03:25:49.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:49 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:25:50.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:49 smithi117 ceph-mon[31838]: from='osd.3 [v2:172.21.15.117:6810/2178310135,v1:172.21.15.117:6811/2178310135]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-19T03:25:50.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:49 smithi117 ceph-mon[31838]: osdmap e24: 4 total, 3 up, 4 in 2024-03-19T03:25:50.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:49 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:25:50.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:49 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:25:50.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:49 smithi117 ceph-mon[31838]: osd.3 [v2:172.21.15.117:6810/2178310135,v1:172.21.15.117:6811/2178310135] boot 2024-03-19T03:25:50.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:49 smithi117 ceph-mon[31838]: osdmap e25: 4 total, 4 up, 4 in 2024-03-19T03:25:50.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:49 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T03:25:51.002 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-19T03:25:51.002 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-19T03:25:51.003 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-03-19T03:25:51.003 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-03-19T03:25:51.003 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128215 s, 818 MB/s 2024-03-19T03:25:51.003 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-03-19T03:25:51.003 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:50 smithi119 ceph-mon[66861]: purged_snaps scrub starts 2024-03-19T03:25:51.003 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:50 smithi119 ceph-mon[66861]: purged_snaps scrub ok 2024-03-19T03:25:51.003 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:50 smithi119 ceph-mon[66861]: osdmap e26: 4 total, 4 up, 4 in 2024-03-19T03:25:51.003 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:50 smithi119 ceph-mon[66861]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:25:51.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:50 smithi117 ceph-mon[31838]: purged_snaps scrub starts 2024-03-19T03:25:51.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:50 smithi117 ceph-mon[31838]: purged_snaps scrub ok 2024-03-19T03:25:51.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:50 smithi117 ceph-mon[31838]: osdmap e26: 4 total, 4 up, 4 in 2024-03-19T03:25:51.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:50 smithi117 ceph-mon[31838]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:25:51.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:50 smithi178 ceph-mon[67578]: purged_snaps scrub starts 2024-03-19T03:25:51.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:50 smithi178 ceph-mon[67578]: purged_snaps scrub ok 2024-03-19T03:25:51.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:50 smithi178 ceph-mon[67578]: osdmap e26: 4 total, 4 up, 4 in 2024-03-19T03:25:51.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:50 smithi178 ceph-mon[67578]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:25:51.655 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi119:vg_nvme/lv_3 2024-03-19T03:25:51.887 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:51 smithi119 ceph-mon[66861]: osdmap e27: 4 total, 4 up, 4 in 2024-03-19T03:25:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T03:25:52.291 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:51 smithi117 ceph-mon[31838]: osdmap e27: 4 total, 4 up, 4 in 2024-03-19T03:25:52.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:51 smithi178 ceph-mon[67578]: osdmap e27: 4 total, 4 up, 4 in 2024-03-19T03:25:52.967 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:52 smithi119 ceph-mon[66861]: osdmap e28: 4 total, 4 up, 4 in 2024-03-19T03:25:52.967 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:52 smithi119 ceph-mon[66861]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:25:53.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:52 smithi117 ceph-mon[31838]: osdmap e28: 4 total, 4 up, 4 in 2024-03-19T03:25:53.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:52 smithi117 ceph-mon[31838]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:25:53.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:52 smithi178 ceph-mon[67578]: osdmap e28: 4 total, 4 up, 4 in 2024-03-19T03:25:53.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:52 smithi178 ceph-mon[67578]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:25:53.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.438+0000 7f60fa4a5700 1 -- 172.21.15.119:0/3845535121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60f4102560 msgr2=0x7f60f4108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:25:53.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.438+0000 7f60fa4a5700 1 --2- 172.21.15.119:0/3845535121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60f4102560 0x7f60f4108e10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f60ec00b0d0 tx=0x7f60ec00b3e0 comp rx=0 tx=0).stop 2024-03-19T03:25:53.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.438+0000 7f60fa4a5700 1 -- 172.21.15.119:0/3845535121 shutdown_connections 2024-03-19T03:25:53.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.438+0000 7f60fa4a5700 1 --2- 172.21.15.119:0/3845535121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60f4102560 0x7f60f4108e10 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:25:53.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.438+0000 7f60fa4a5700 1 --2- 172.21.15.119:0/3845535121 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f60f41009c0 0x7f60f4100de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:25:53.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.438+0000 7f60fa4a5700 1 --2- 172.21.15.119:0/3845535121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60f4101bc0 0x7f60f4102020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:25:53.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.438+0000 7f60fa4a5700 1 -- 172.21.15.119:0/3845535121 >> 172.21.15.119:0/3845535121 conn(0x7f60f40fbf80 msgr2=0x7f60f40fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:25:53.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.438+0000 7f60fa4a5700 1 -- 172.21.15.119:0/3845535121 shutdown_connections 2024-03-19T03:25:53.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.438+0000 7f60fa4a5700 1 -- 172.21.15.119:0/3845535121 wait complete. 2024-03-19T03:25:53.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.438+0000 7f60fa4a5700 1 Processor -- start 2024-03-19T03:25:53.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.438+0000 7f60fa4a5700 1 -- start start 2024-03-19T03:25:53.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.439+0000 7f60fa4a5700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60f41009c0 0x7f60f4196ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:25:53.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.439+0000 7f60fa4a5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60f4101bc0 0x7f60f4197000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:25:53.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.439+0000 7f60fa4a5700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f60f4102560 0x7f60f419fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:25:53.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.439+0000 7f60fa4a5700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f60f410c6a0 con 0x7f60f41009c0 2024-03-19T03:25:53.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.439+0000 7f60fa4a5700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f60f410c520 con 0x7f60f4101bc0 2024-03-19T03:25:53.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.439+0000 7f60fa4a5700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f60f410c820 con 0x7f60f4102560 2024-03-19T03:25:53.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.439+0000 7f60f37fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60f4101bc0 0x7f60f4197000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:25:53.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.439+0000 7f60f37fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60f4101bc0 0x7f60f4197000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46298/0 (socket says 172.21.15.119:46298) 2024-03-19T03:25:53.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.439+0000 7f60f37fe700 1 -- 172.21.15.119:0/1774431030 learned_addr learned my addr 172.21.15.119:0/1774431030 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-19T03:25:53.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.439+0000 7f60f37fe700 1 -- 172.21.15.119:0/1774431030 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f60f4102560 msgr2=0x7f60f419fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:25:53.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.439+0000 7f60f37fe700 1 --2- 172.21.15.119:0/1774431030 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f60f4102560 0x7f60f419fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:25:53.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.439+0000 7f60f37fe700 1 -- 172.21.15.119:0/1774431030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60f41009c0 msgr2=0x7f60f4196ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:25:53.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.439+0000 7f60f37fe700 1 --2- 172.21.15.119:0/1774431030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60f41009c0 0x7f60f4196ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:25:53.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.439+0000 7f60f37fe700 1 -- 172.21.15.119:0/1774431030 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60f41a02e0 con 0x7f60f4101bc0 2024-03-19T03:25:53.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.439+0000 7f60f37fe700 1 --2- 172.21.15.119:0/1774431030 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60f4101bc0 0x7f60f4197000 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f60e800b530 tx=0x7f60e800b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:25:53.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.440+0000 7f60f17fa700 1 -- 172.21.15.119:0/1774431030 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e8015020 con 0x7f60f4101bc0 2024-03-19T03:25:53.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.440+0000 7f60fa4a5700 1 -- 172.21.15.119:0/1774431030 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60f41a05d0 con 0x7f60f4101bc0 2024-03-19T03:25:53.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.440+0000 7f60f17fa700 1 -- 172.21.15.119:0/1774431030 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60e8011d10 con 0x7f60f4101bc0 2024-03-19T03:25:53.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.440+0000 7f60f17fa700 1 -- 172.21.15.119:0/1774431030 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e8010e50 con 0x7f60f4101bc0 2024-03-19T03:25:53.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.440+0000 7f60fa4a5700 1 -- 172.21.15.119:0/1774431030 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60f41a0be0 con 0x7f60f4101bc0 2024-03-19T03:25:53.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.441+0000 7f60fa4a5700 1 -- 172.21.15.119:0/1774431030 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60f4068fd0 con 0x7f60f4101bc0 2024-03-19T03:25:53.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.442+0000 7f60f17fa700 1 -- 172.21.15.119:0/1774431030 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f60e8018020 con 0x7f60f4101bc0 2024-03-19T03:25:53.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.442+0000 7f60f17fa700 1 --2- 172.21.15.119:0/1774431030 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f60dc0a2a30 0x7f60dc0a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:25:53.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.442+0000 7f60f17fa700 1 -- 172.21.15.119:0/1774431030 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 3669+0+0 (secure 0 0 0) 0x7f60e80cd0a0 con 0x7f60f4101bc0 2024-03-19T03:25:53.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.443+0000 7f60f3fff700 1 --2- 172.21.15.119:0/1774431030 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f60dc0a2a30 0x7f60dc0a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:25:53.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.443+0000 7f60f3fff700 1 --2- 172.21.15.119:0/1774431030 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f60dc0a2a30 0x7f60dc0a4ef0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f60f4101a20 tx=0x7f60e4006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:25:53.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.447+0000 7f60f17fa700 1 -- 172.21.15.119:0/1774431030 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f60e801f020 con 0x7f60f4101bc0 2024-03-19T03:25:53.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:25:53.651+0000 7f60fa4a5700 1 -- 172.21.15.119:0/1774431030 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f60f4062750 con 0x7f60dc0a2a30 2024-03-19T03:25:54.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:53 smithi117 ceph-mon[31838]: osdmap e29: 4 total, 4 up, 4 in 2024-03-19T03:25:54.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:53 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:25:54.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:53 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:25:54.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:53 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:25:54.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:53 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:54.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:53 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:54.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:53 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:25:54.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:53 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:25:54.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:53 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:54.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:53 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:25:54.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:53 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:25:54.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:53 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:54.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:53 smithi119 ceph-mon[66861]: osdmap e29: 4 total, 4 up, 4 in 2024-03-19T03:25:54.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:53 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:25:54.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:53 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:25:54.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:53 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:25:54.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:53 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:54.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:53 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:54.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:53 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:25:54.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:53 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:25:54.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:53 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:54.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:53 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:25:54.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:53 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:25:54.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:53 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:54.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:53 smithi178 ceph-mon[67578]: osdmap e29: 4 total, 4 up, 4 in 2024-03-19T03:25:54.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:53 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:25:54.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:53 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:25:54.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:53 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:25:54.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:53 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:54.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:53 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:54.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:53 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:25:54.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:53 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:25:54.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:53 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:54.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:53 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:25:54.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:53 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:25:54.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:53 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:25:55.202 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:54 smithi119 ceph-mon[66861]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:25:55.202 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:54 smithi119 ceph-mon[66861]: Detected new or changed devices on smithi117 2024-03-19T03:25:55.202 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:54 smithi119 ceph-mon[66861]: Adjusting osd_memory_target on smithi117 to 8545M 2024-03-19T03:25:55.202 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:54 smithi119 ceph-mon[66861]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:25:55.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:54 smithi117 ceph-mon[31838]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:25:55.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:54 smithi117 ceph-mon[31838]: Detected new or changed devices on smithi117 2024-03-19T03:25:55.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:54 smithi117 ceph-mon[31838]: Adjusting osd_memory_target on smithi117 to 8545M 2024-03-19T03:25:55.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:54 smithi117 ceph-mon[31838]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:25:55.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:54 smithi178 ceph-mon[67578]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:25:55.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:54 smithi178 ceph-mon[67578]: Detected new or changed devices on smithi117 2024-03-19T03:25:55.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:54 smithi178 ceph-mon[67578]: Adjusting osd_memory_target on smithi117 to 8545M 2024-03-19T03:25:55.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:54 smithi178 ceph-mon[67578]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:25:57.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:57 smithi117 ceph-mon[31838]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T03:25:57.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:57 smithi117 ceph-mon[31838]: from='client.? 172.21.15.119:0/511651277' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "caf7ce86-caf0-4eaf-bbab-bd2d93e63c33"}]: dispatch 2024-03-19T03:25:57.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:57 smithi117 ceph-mon[31838]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "caf7ce86-caf0-4eaf-bbab-bd2d93e63c33"}]: dispatch 2024-03-19T03:25:57.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:57 smithi117 ceph-mon[31838]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "caf7ce86-caf0-4eaf-bbab-bd2d93e63c33"}]': finished 2024-03-19T03:25:57.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:57 smithi117 ceph-mon[31838]: osdmap e30: 5 total, 4 up, 5 in 2024-03-19T03:25:57.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:57 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T03:25:57.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:57 smithi117 ceph-mon[31838]: from='client.? 172.21.15.119:0/3645186443' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:25:57.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:57 smithi178 ceph-mon[67578]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T03:25:57.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:57 smithi178 ceph-mon[67578]: from='client.? 172.21.15.119:0/511651277' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "caf7ce86-caf0-4eaf-bbab-bd2d93e63c33"}]: dispatch 2024-03-19T03:25:57.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:57 smithi178 ceph-mon[67578]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "caf7ce86-caf0-4eaf-bbab-bd2d93e63c33"}]: dispatch 2024-03-19T03:25:57.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:57 smithi178 ceph-mon[67578]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "caf7ce86-caf0-4eaf-bbab-bd2d93e63c33"}]': finished 2024-03-19T03:25:57.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:57 smithi178 ceph-mon[67578]: osdmap e30: 5 total, 4 up, 5 in 2024-03-19T03:25:57.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:57 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T03:25:57.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:57 smithi178 ceph-mon[67578]: from='client.? 172.21.15.119:0/3645186443' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:25:57.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:57 smithi119 ceph-mon[66861]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T03:25:57.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:57 smithi119 ceph-mon[66861]: from='client.? 172.21.15.119:0/511651277' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "caf7ce86-caf0-4eaf-bbab-bd2d93e63c33"}]: dispatch 2024-03-19T03:25:57.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:57 smithi119 ceph-mon[66861]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "caf7ce86-caf0-4eaf-bbab-bd2d93e63c33"}]: dispatch 2024-03-19T03:25:57.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:57 smithi119 ceph-mon[66861]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "caf7ce86-caf0-4eaf-bbab-bd2d93e63c33"}]': finished 2024-03-19T03:25:57.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:57 smithi119 ceph-mon[66861]: osdmap e30: 5 total, 4 up, 5 in 2024-03-19T03:25:57.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:57 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T03:25:57.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:57 smithi119 ceph-mon[66861]: from='client.? 172.21.15.119:0/3645186443' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:25:59.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:25:59 smithi117 ceph-mon[31838]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-19T03:25:59.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:25:59 smithi178 ceph-mon[67578]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-19T03:25:59.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:25:59 smithi119 ceph-mon[66861]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-19T03:26:01.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:01 smithi117 ceph-mon[31838]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T03:26:01.549 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:01 smithi119 ceph-mon[66861]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T03:26:01.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:01 smithi178 ceph-mon[67578]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T03:26:03.534 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:03 smithi119 ceph-mon[66861]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-19T03:26:03.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:03 smithi117 ceph-mon[31838]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-19T03:26:03.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:03 smithi178 ceph-mon[67578]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-19T03:26:04.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:04 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-19T03:26:04.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:04 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:04.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:04 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-19T03:26:04.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:04 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:04.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:04 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-19T03:26:04.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:04 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:05.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:05 smithi119 ceph-mon[66861]: Deploying daemon osd.4 on smithi119 2024-03-19T03:26:05.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:05 smithi119 ceph-mon[66861]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T03:26:05.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:05 smithi117 ceph-mon[31838]: Deploying daemon osd.4 on smithi119 2024-03-19T03:26:05.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:05 smithi117 ceph-mon[31838]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T03:26:05.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:05 smithi178 ceph-mon[67578]: Deploying daemon osd.4 on smithi119 2024-03-19T03:26:05.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:05 smithi178 ceph-mon[67578]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T03:26:07.537 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:07 smithi119 ceph-mon[66861]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:26:07.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:07 smithi117 ceph-mon[31838]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:26:07.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:07 smithi178 ceph-mon[67578]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:26:09.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:09 smithi117 ceph-mon[31838]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:26:09.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:09 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:09.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:09 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:09.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:09 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:26:09.565 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:09 smithi119 ceph-mon[66861]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:26:09.565 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:09 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:09.565 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:09 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:09.565 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:09 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:26:09.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:09 smithi178 ceph-mon[67578]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:26:09.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:09 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:09.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:09 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:09.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:09 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:26:11.339 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:11 smithi119 ceph-mon[66861]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:26:11.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:11 smithi117 ceph-mon[31838]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:26:11.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:11 smithi178 ceph-mon[67578]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:26:12.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:26:12.543+0000 7f60f17fa700 1 -- 172.21.15.119:0/1774431030 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f60f4062750 con 0x7f60dc0a2a30 2024-03-19T03:26:12.544 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 4 on host 'smithi119' 2024-03-19T03:26:12.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:26:12.545+0000 7f60fa4a5700 1 -- 172.21.15.119:0/1774431030 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f60dc0a2a30 msgr2=0x7f60dc0a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:26:12.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:26:12.545+0000 7f60fa4a5700 1 --2- 172.21.15.119:0/1774431030 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f60dc0a2a30 0x7f60dc0a4ef0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f60f4101a20 tx=0x7f60e4006d20 comp rx=0 tx=0).stop 2024-03-19T03:26:12.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:26:12.545+0000 7f60fa4a5700 1 -- 172.21.15.119:0/1774431030 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60f4101bc0 msgr2=0x7f60f4197000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:26:12.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:26:12.545+0000 7f60fa4a5700 1 --2- 172.21.15.119:0/1774431030 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60f4101bc0 0x7f60f4197000 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f60e800b530 tx=0x7f60e800b8f0 comp rx=0 tx=0).stop 2024-03-19T03:26:12.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:26:12.545+0000 7f60fa4a5700 1 -- 172.21.15.119:0/1774431030 shutdown_connections 2024-03-19T03:26:12.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:26:12.545+0000 7f60fa4a5700 1 --2- 172.21.15.119:0/1774431030 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f60dc0a2a30 0x7f60dc0a4ef0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:26:12.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:26:12.545+0000 7f60fa4a5700 1 --2- 172.21.15.119:0/1774431030 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f60f4102560 0x7f60f419fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:26:12.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:26:12.545+0000 7f60fa4a5700 1 --2- 172.21.15.119:0/1774431030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60f41009c0 0x7f60f4196ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:26:12.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:26:12.545+0000 7f60fa4a5700 1 --2- 172.21.15.119:0/1774431030 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60f4101bc0 0x7f60f4197000 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:26:12.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:26:12.545+0000 7f60fa4a5700 1 -- 172.21.15.119:0/1774431030 >> 172.21.15.119:0/1774431030 conn(0x7f60f40fbf80 msgr2=0x7f60f4104f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:26:12.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:26:12.545+0000 7f60fa4a5700 1 -- 172.21.15.119:0/1774431030 shutdown_connections 2024-03-19T03:26:12.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:26:12.545+0000 7f60fa4a5700 1 -- 172.21.15.119:0/1774431030 wait complete. 2024-03-19T03:26:13.196 DEBUG:teuthology.orchestra.run.smithi119:osd.4> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.4.service 2024-03-19T03:26:13.198 INFO:tasks.cephadm:Deploying osd.5 on smithi178 with /dev/vg_nvme/lv_3... 2024-03-19T03:26:13.199 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-19T03:26:13.447 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T03:26:13.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:13 smithi119 ceph-mon[66861]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:26:13.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:13 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:13.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:13 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:13.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:13 smithi119 ceph-mon[66861]: from='osd.4 [v2:172.21.15.119:6808/3455398086,v1:172.21.15.119:6809/3455398086]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T03:26:13.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:13 smithi119 ceph-mon[66861]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T03:26:13.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:13 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:13.569 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:13 smithi178 ceph-mon[67578]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:26:13.569 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:13 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:13.569 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:13 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:13.569 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:13 smithi178 ceph-mon[67578]: from='osd.4 [v2:172.21.15.119:6808/3455398086,v1:172.21.15.119:6809/3455398086]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T03:26:13.569 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:13 smithi178 ceph-mon[67578]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T03:26:13.569 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:13 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:13.794 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:13 smithi117 ceph-mon[31838]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:26:13.794 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:13 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:13.794 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:13 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:13.794 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:13 smithi117 ceph-mon[31838]: from='osd.4 [v2:172.21.15.119:6808/3455398086,v1:172.21.15.119:6809/3455398086]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T03:26:13.795 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:13 smithi117 ceph-mon[31838]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T03:26:13.795 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:13 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:14.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:14 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:14.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:14 smithi119 ceph-mon[66861]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-19T03:26:14.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:14 smithi119 ceph-mon[66861]: from='osd.4 [v2:172.21.15.119:6808/3455398086,v1:172.21.15.119:6809/3455398086]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-19T03:26:14.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:14 smithi119 ceph-mon[66861]: osdmap e31: 5 total, 4 up, 5 in 2024-03-19T03:26:14.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:14 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T03:26:14.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:14 smithi119 ceph-mon[66861]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-19T03:26:14.589 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:14 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:14.589 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:14 smithi178 ceph-mon[67578]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-19T03:26:14.589 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:14 smithi178 ceph-mon[67578]: from='osd.4 [v2:172.21.15.119:6808/3455398086,v1:172.21.15.119:6809/3455398086]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-19T03:26:14.589 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:14 smithi178 ceph-mon[67578]: osdmap e31: 5 total, 4 up, 5 in 2024-03-19T03:26:14.589 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:14 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T03:26:14.589 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:14 smithi178 ceph-mon[67578]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-19T03:26:14.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:14 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:14.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:14 smithi117 ceph-mon[31838]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-19T03:26:14.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:14 smithi117 ceph-mon[31838]: from='osd.4 [v2:172.21.15.119:6808/3455398086,v1:172.21.15.119:6809/3455398086]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-19T03:26:14.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:14 smithi117 ceph-mon[31838]: osdmap e31: 5 total, 4 up, 5 in 2024-03-19T03:26:14.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:14 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T03:26:14.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:14 smithi117 ceph-mon[31838]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-19T03:26:14.924 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 19 03:26:14 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-4[78276]: 2024-03-19T03:26:14.534+0000 7f9b6d0b0700 -1 osd.4 0 waiting for initial osdmap 2024-03-19T03:26:14.924 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 19 03:26:14 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-4[78276]: 2024-03-19T03:26:14.540+0000 7f9b68ea3700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T03:26:15.462 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-19T03:26:15.462 INFO:teuthology.orchestra.run.smithi178.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-19T03:26:15.462 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10+0 records in 2024-03-19T03:26:15.462 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records out 2024-03-19T03:26:15.462 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00828524 s, 1.3 GB/s 2024-03-19T03:26:15.462 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping successful for: 2024-03-19T03:26:15.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:15 smithi119 ceph-mon[66861]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:26:15.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:15 smithi119 ceph-mon[66861]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-19T03:26:15.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:15 smithi119 ceph-mon[66861]: osdmap e32: 5 total, 4 up, 5 in 2024-03-19T03:26:15.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:15 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T03:26:15.579 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:15 smithi178 ceph-mon[67578]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:26:15.579 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:15 smithi178 ceph-mon[67578]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-19T03:26:15.579 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:15 smithi178 ceph-mon[67578]: osdmap e32: 5 total, 4 up, 5 in 2024-03-19T03:26:15.580 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:15 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T03:26:15.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:15 smithi117 ceph-mon[31838]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:26:15.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:15 smithi117 ceph-mon[31838]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-19T03:26:15.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:15 smithi117 ceph-mon[31838]: osdmap e32: 5 total, 4 up, 5 in 2024-03-19T03:26:15.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:15 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T03:26:16.092 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi178:vg_nvme/lv_3 2024-03-19T03:26:16.336 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T03:26:16.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:16 smithi178 ceph-mon[67578]: purged_snaps scrub starts 2024-03-19T03:26:16.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:16 smithi178 ceph-mon[67578]: purged_snaps scrub ok 2024-03-19T03:26:16.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:16 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T03:26:16.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:16 smithi178 ceph-mon[67578]: osd.4 [v2:172.21.15.119:6808/3455398086,v1:172.21.15.119:6809/3455398086] boot 2024-03-19T03:26:16.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:16 smithi178 ceph-mon[67578]: osdmap e33: 5 total, 5 up, 5 in 2024-03-19T03:26:16.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:16 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T03:26:16.665 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:16 smithi119 ceph-mon[66861]: purged_snaps scrub starts 2024-03-19T03:26:16.666 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:16 smithi119 ceph-mon[66861]: purged_snaps scrub ok 2024-03-19T03:26:16.666 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:16 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T03:26:16.666 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:16 smithi119 ceph-mon[66861]: osd.4 [v2:172.21.15.119:6808/3455398086,v1:172.21.15.119:6809/3455398086] boot 2024-03-19T03:26:16.666 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:16 smithi119 ceph-mon[66861]: osdmap e33: 5 total, 5 up, 5 in 2024-03-19T03:26:16.666 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:16 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T03:26:16.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:16 smithi117 ceph-mon[31838]: purged_snaps scrub starts 2024-03-19T03:26:16.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:16 smithi117 ceph-mon[31838]: purged_snaps scrub ok 2024-03-19T03:26:16.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:16 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T03:26:16.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:16 smithi117 ceph-mon[31838]: osd.4 [v2:172.21.15.119:6808/3455398086,v1:172.21.15.119:6809/3455398086] boot 2024-03-19T03:26:16.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:16 smithi117 ceph-mon[31838]: osdmap e33: 5 total, 5 up, 5 in 2024-03-19T03:26:16.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:16 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T03:26:17.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:17 smithi178 ceph-mon[67578]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:26:17.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:17 smithi178 ceph-mon[67578]: osdmap e34: 5 total, 5 up, 5 in 2024-03-19T03:26:17.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:17 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:17.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:17 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:17.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:17 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:26:17.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:17 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:26:17.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:17 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:17.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:17 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:17.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:17 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:26:17.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:17 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:17.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:17 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:26:17.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:17 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:17.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:17 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:26:17.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:17 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:17 smithi119 ceph-mon[66861]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:26:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:17 smithi119 ceph-mon[66861]: osdmap e34: 5 total, 5 up, 5 in 2024-03-19T03:26:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:17 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:17 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:17.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:17 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:26:17.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:17 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:26:17.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:17 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:17.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:17 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:17.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:17 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:26:17.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:17 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:17.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:17 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:26:17.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:17 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:17.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:17 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:26:17.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:17 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:17.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:17 smithi117 ceph-mon[31838]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-19T03:26:17.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:17 smithi117 ceph-mon[31838]: osdmap e34: 5 total, 5 up, 5 in 2024-03-19T03:26:17.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:17 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:17.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:17 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:17.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:17 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:26:17.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:17 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:26:17.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:17 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:17.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:17 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:17.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:17 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:26:17.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:17 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:17.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:17 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:26:17.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:17 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:17.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:17 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:26:17.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:17 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:17.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.852+0000 7fe9cba36700 1 -- 172.21.15.178:0/3022239640 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9c4102490 msgr2=0x7fe9c41028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:26:17.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.852+0000 7fe9cba36700 1 --2- 172.21.15.178:0/3022239640 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9c4102490 0x7fe9c41028f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4009990 tx=0x7fe9b4009ca0 comp rx=0 tx=0).stop 2024-03-19T03:26:17.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.852+0000 7fe9cba36700 1 -- 172.21.15.178:0/3022239640 shutdown_connections 2024-03-19T03:26:17.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.852+0000 7fe9cba36700 1 --2- 172.21.15.178:0/3022239640 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9c4102e30 0x7fe9c41096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:26:17.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.852+0000 7fe9cba36700 1 --2- 172.21.15.178:0/3022239640 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9c4101290 0x7fe9c41016b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:26:17.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.852+0000 7fe9cba36700 1 --2- 172.21.15.178:0/3022239640 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9c4102490 0x7fe9c41028f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:26:17.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.852+0000 7fe9cba36700 1 -- 172.21.15.178:0/3022239640 >> 172.21.15.178:0/3022239640 conn(0x7fe9c40fc830 msgr2=0x7fe9c40fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:26:17.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.852+0000 7fe9cba36700 1 -- 172.21.15.178:0/3022239640 shutdown_connections 2024-03-19T03:26:17.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.852+0000 7fe9cba36700 1 -- 172.21.15.178:0/3022239640 wait complete. 2024-03-19T03:26:17.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.852+0000 7fe9cba36700 1 Processor -- start 2024-03-19T03:26:17.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.852+0000 7fe9cba36700 1 -- start start 2024-03-19T03:26:17.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.853+0000 7fe9cba36700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9c4101290 0x7fe9c41975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:26:17.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.853+0000 7fe9cba36700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9c4102490 0x7fe9c4197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:26:17.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.853+0000 7fe9cba36700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9c4102e30 0x7fe9c419eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:26:17.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.853+0000 7fe9cba36700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c410cf70 con 0x7fe9c4101290 2024-03-19T03:26:17.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.853+0000 7fe9cba36700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c410cdf0 con 0x7fe9c4102e30 2024-03-19T03:26:17.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.853+0000 7fe9cba36700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c410d0f0 con 0x7fe9c4102490 2024-03-19T03:26:17.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.853+0000 7fe9c8fd1700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9c4102490 0x7fe9c4197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:26:17.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.853+0000 7fe9c8fd1700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9c4102490 0x7fe9c4197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:44312/0 (socket says 172.21.15.178:44312) 2024-03-19T03:26:17.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.853+0000 7fe9c8fd1700 1 -- 172.21.15.178:0/3433228068 learned_addr learned my addr 172.21.15.178:0/3433228068 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-19T03:26:17.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.853+0000 7fe9c9fd3700 1 --2- 172.21.15.178:0/3433228068 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9c4102e30 0x7fe9c419eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:26:17.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.853+0000 7fe9c8fd1700 1 -- 172.21.15.178:0/3433228068 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9c4102e30 msgr2=0x7fe9c419eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:26:17.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.853+0000 7fe9c8fd1700 1 --2- 172.21.15.178:0/3433228068 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9c4102e30 0x7fe9c419eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:26:17.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.853+0000 7fe9c8fd1700 1 -- 172.21.15.178:0/3433228068 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9c4101290 msgr2=0x7fe9c41975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:26:17.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.853+0000 7fe9c8fd1700 1 --2- 172.21.15.178:0/3433228068 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9c4101290 0x7fe9c41975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:26:17.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.853+0000 7fe9c8fd1700 1 -- 172.21.15.178:0/3433228068 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9c419f210 con 0x7fe9c4102490 2024-03-19T03:26:17.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.853+0000 7fe9c8fd1700 1 --2- 172.21.15.178:0/3433228068 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9c4102490 0x7fe9c4197af0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe9b40047b0 tx=0x7fe9b4003680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:26:17.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.853+0000 7fe9ba7fc700 1 -- 172.21.15.178:0/3433228068 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4027070 con 0x7fe9c4102490 2024-03-19T03:26:17.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.853+0000 7fe9ba7fc700 1 -- 172.21.15.178:0/3433228068 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9b4022470 con 0x7fe9c4102490 2024-03-19T03:26:17.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.853+0000 7fe9ba7fc700 1 -- 172.21.15.178:0/3433228068 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b400f6c0 con 0x7fe9c4102490 2024-03-19T03:26:17.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.853+0000 7fe9cba36700 1 -- 172.21.15.178:0/3433228068 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9c419f4a0 con 0x7fe9c4102490 2024-03-19T03:26:17.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.853+0000 7fe9cba36700 1 -- 172.21.15.178:0/3433228068 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9c419f9d0 con 0x7fe9c4102490 2024-03-19T03:26:17.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.854+0000 7fe9cba36700 1 -- 172.21.15.178:0/3433228068 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9c4068fd0 con 0x7fe9c4102490 2024-03-19T03:26:17.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.856+0000 7fe9ba7fc700 1 -- 172.21.15.178:0/3433228068 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fe9b402a070 con 0x7fe9c4102490 2024-03-19T03:26:17.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.856+0000 7fe9ba7fc700 1 --2- 172.21.15.178:0/3433228068 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe9b00a2910 0x7fe9b00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:26:17.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.856+0000 7fe9ba7fc700 1 -- 172.21.15.178:0/3433228068 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7fe9b40cef20 con 0x7fe9c4102490 2024-03-19T03:26:17.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.857+0000 7fe9c97d2700 1 --2- 172.21.15.178:0/3433228068 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe9b00a2910 0x7fe9b00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:26:17.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.857+0000 7fe9c97d2700 1 --2- 172.21.15.178:0/3433228068 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe9b00a2910 0x7fe9b00a4dd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe9c41022f0 tx=0x7fe9c0006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:26:17.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:17.860+0000 7fe9ba7fc700 1 -- 172.21.15.178:0/3433228068 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe9b4031070 con 0x7fe9c4102490 2024-03-19T03:26:18.063 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:18.061+0000 7fe9cba36700 1 -- 172.21.15.178:0/3433228068 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fe9c4062750 con 0x7fe9b00a2910 2024-03-19T03:26:18.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:18 smithi178 ceph-mon[67578]: Detected new or changed devices on smithi119 2024-03-19T03:26:18.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:18 smithi178 ceph-mon[67578]: Adjusting osd_memory_target on smithi119 to 8540M 2024-03-19T03:26:18.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:18 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:26:18.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:18 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:26:18.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:18 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:18.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:18 smithi119 ceph-mon[66861]: Detected new or changed devices on smithi119 2024-03-19T03:26:18.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:18 smithi119 ceph-mon[66861]: Adjusting osd_memory_target on smithi119 to 8540M 2024-03-19T03:26:18.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:18 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:26:18.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:18 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:26:18.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:18 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:18.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:18 smithi117 ceph-mon[31838]: Detected new or changed devices on smithi119 2024-03-19T03:26:18.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:18 smithi117 ceph-mon[31838]: Adjusting osd_memory_target on smithi119 to 8540M 2024-03-19T03:26:18.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:18 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:26:18.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:18 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:26:18.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:18 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:19.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:19 smithi178 ceph-mon[67578]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:26:19.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:19 smithi178 ceph-mon[67578]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:19.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:19 smithi119 ceph-mon[66861]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:26:19.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:19 smithi119 ceph-mon[66861]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:19.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:19 smithi117 ceph-mon[31838]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:26:19.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:19 smithi117 ceph-mon[31838]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:21.497 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:21 smithi178 ceph-mon[67578]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:21.497 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:21 smithi178 ceph-mon[67578]: from='client.? 172.21.15.178:0/597077970' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20ba0b8a-2e0f-4ea7-b435-430e44766200"}]: dispatch 2024-03-19T03:26:21.497 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:21 smithi178 ceph-mon[67578]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20ba0b8a-2e0f-4ea7-b435-430e44766200"}]: dispatch 2024-03-19T03:26:21.497 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:21 smithi178 ceph-mon[67578]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20ba0b8a-2e0f-4ea7-b435-430e44766200"}]': finished 2024-03-19T03:26:21.498 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:21 smithi178 ceph-mon[67578]: osdmap e35: 6 total, 5 up, 6 in 2024-03-19T03:26:21.498 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:21 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T03:26:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:21 smithi119 ceph-mon[66861]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:21 smithi119 ceph-mon[66861]: from='client.? 172.21.15.178:0/597077970' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20ba0b8a-2e0f-4ea7-b435-430e44766200"}]: dispatch 2024-03-19T03:26:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:21 smithi119 ceph-mon[66861]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20ba0b8a-2e0f-4ea7-b435-430e44766200"}]: dispatch 2024-03-19T03:26:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:21 smithi119 ceph-mon[66861]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20ba0b8a-2e0f-4ea7-b435-430e44766200"}]': finished 2024-03-19T03:26:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:21 smithi119 ceph-mon[66861]: osdmap e35: 6 total, 5 up, 6 in 2024-03-19T03:26:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:21 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T03:26:21.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:21 smithi117 ceph-mon[31838]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:21.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:21 smithi117 ceph-mon[31838]: from='client.? 172.21.15.178:0/597077970' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20ba0b8a-2e0f-4ea7-b435-430e44766200"}]: dispatch 2024-03-19T03:26:21.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:21 smithi117 ceph-mon[31838]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20ba0b8a-2e0f-4ea7-b435-430e44766200"}]: dispatch 2024-03-19T03:26:21.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:21 smithi117 ceph-mon[31838]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20ba0b8a-2e0f-4ea7-b435-430e44766200"}]': finished 2024-03-19T03:26:21.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:21 smithi117 ceph-mon[31838]: osdmap e35: 6 total, 5 up, 6 in 2024-03-19T03:26:21.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:21 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T03:26:22.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:22 smithi178 ceph-mon[67578]: from='client.? 172.21.15.178:0/313479640' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:26:22.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:22 smithi119 ceph-mon[66861]: from='client.? 172.21.15.178:0/313479640' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:26:22.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:22 smithi117 ceph-mon[31838]: from='client.? 172.21.15.178:0/313479640' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:26:23.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:23 smithi178 ceph-mon[67578]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:23.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:23 smithi119 ceph-mon[66861]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:23.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:23 smithi117 ceph-mon[31838]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:25.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:25 smithi178 ceph-mon[67578]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:25.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:25 smithi119 ceph-mon[66861]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:25.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:25 smithi117 ceph-mon[31838]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:27.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:27 smithi178 ceph-mon[67578]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:27.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:27 smithi119 ceph-mon[66861]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:27.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:27 smithi117 ceph-mon[31838]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:28.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:28 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-19T03:26:28.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:28 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:28.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:28 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-19T03:26:28.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:28 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:28.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:28 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-19T03:26:28.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:28 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:29.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:29 smithi178 ceph-mon[67578]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:29.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:29 smithi178 ceph-mon[67578]: Deploying daemon osd.5 on smithi178 2024-03-19T03:26:29.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:29 smithi119 ceph-mon[66861]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:29.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:29 smithi119 ceph-mon[66861]: Deploying daemon osd.5 on smithi178 2024-03-19T03:26:29.794 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:29 smithi117 ceph-mon[31838]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:29.795 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:29 smithi117 ceph-mon[31838]: Deploying daemon osd.5 on smithi178 2024-03-19T03:26:31.653 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:31 smithi178 ceph-mon[67578]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:31.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:31 smithi119 ceph-mon[66861]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:31.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:31 smithi117 ceph-mon[31838]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:33.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:33 smithi178 ceph-mon[67578]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:33.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:33 smithi119 ceph-mon[66861]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:33.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:33 smithi117 ceph-mon[31838]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:35.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:34 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:35.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:34 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:26:35.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:34 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:35.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:34 smithi119 ceph-mon[66861]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:35.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:34 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:35.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:34 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:26:35.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:34 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:35.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:34 smithi117 ceph-mon[31838]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:35.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:34 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:35.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:34 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:26:35.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:34 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:35.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:34 smithi178 ceph-mon[67578]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:37.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:37 smithi117 ceph-mon[31838]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:37.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:37.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:37.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:37.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:26:37.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:37.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:37 smithi178 ceph-mon[67578]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:37.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:37.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:37.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:37.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:26:37.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:37.701 INFO:teuthology.orchestra.run.smithi178.stdout:Created osd(s) 5 on host 'smithi178' 2024-03-19T03:26:37.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:37.701+0000 7fe9ba7fc700 1 -- 172.21.15.178:0/3433228068 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe9c4062750 con 0x7fe9b00a2910 2024-03-19T03:26:37.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:37.703+0000 7fe9cba36700 1 -- 172.21.15.178:0/3433228068 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe9b00a2910 msgr2=0x7fe9b00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:26:37.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:37.703+0000 7fe9cba36700 1 --2- 172.21.15.178:0/3433228068 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe9b00a2910 0x7fe9b00a4dd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe9c41022f0 tx=0x7fe9c0006d20 comp rx=0 tx=0).stop 2024-03-19T03:26:37.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:37.703+0000 7fe9cba36700 1 -- 172.21.15.178:0/3433228068 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9c4102490 msgr2=0x7fe9c4197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:26:37.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:37.703+0000 7fe9cba36700 1 --2- 172.21.15.178:0/3433228068 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9c4102490 0x7fe9c4197af0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe9b40047b0 tx=0x7fe9b4003680 comp rx=0 tx=0).stop 2024-03-19T03:26:37.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:37.703+0000 7fe9cba36700 1 -- 172.21.15.178:0/3433228068 shutdown_connections 2024-03-19T03:26:37.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:37.703+0000 7fe9cba36700 1 --2- 172.21.15.178:0/3433228068 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe9b00a2910 0x7fe9b00a4dd0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:26:37.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:37.703+0000 7fe9cba36700 1 --2- 172.21.15.178:0/3433228068 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9c4102e30 0x7fe9c419eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:26:37.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:37.703+0000 7fe9cba36700 1 --2- 172.21.15.178:0/3433228068 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9c4101290 0x7fe9c41975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:26:37.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:37.703+0000 7fe9cba36700 1 --2- 172.21.15.178:0/3433228068 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9c4102490 0x7fe9c4197af0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:26:37.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:37.703+0000 7fe9cba36700 1 -- 172.21.15.178:0/3433228068 >> 172.21.15.178:0/3433228068 conn(0x7fe9c40fc830 msgr2=0x7fe9c4105860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:26:37.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:37.703+0000 7fe9cba36700 1 -- 172.21.15.178:0/3433228068 shutdown_connections 2024-03-19T03:26:37.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:26:37.703+0000 7fe9cba36700 1 -- 172.21.15.178:0/3433228068 wait complete. 2024-03-19T03:26:37.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:37 smithi119 ceph-mon[66861]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:37.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:37 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:37.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:37 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:37.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:37 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:37.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:37 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:26:37.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:37 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:38.366 DEBUG:teuthology.orchestra.run.smithi178:osd.5> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.5.service 2024-03-19T03:26:38.369 INFO:tasks.cephadm:Deploying osd.6 on smithi117 with /dev/vg_nvme/lv_2... 2024-03-19T03:26:38.369 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-19T03:26:38.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:38 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:38.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:38 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:26:38.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:38 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:38.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:38 smithi117 ceph-mon[31838]: from='osd.5 [v2:172.21.15.178:6808/1259631912,v1:172.21.15.178:6809/1259631912]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T03:26:38.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:38 smithi117 ceph-mon[31838]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T03:26:38.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:38 smithi117 ceph-mon[31838]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:38.798 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:26:38.885 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:38 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:38.885 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:38 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:26:38.885 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:38 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:38.885 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:38 smithi178 ceph-mon[67578]: from='osd.5 [v2:172.21.15.178:6808/1259631912,v1:172.21.15.178:6809/1259631912]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T03:26:38.885 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:38 smithi178 ceph-mon[67578]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T03:26:38.885 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:38 smithi178 ceph-mon[67578]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:38.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:38 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:38.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:38 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:26:38.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:38 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:38.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:38 smithi119 ceph-mon[66861]: from='osd.5 [v2:172.21.15.178:6808/1259631912,v1:172.21.15.178:6809/1259631912]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T03:26:38.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:38 smithi119 ceph-mon[66861]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T03:26:38.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:38 smithi119 ceph-mon[66861]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:39.969 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:39 smithi117 ceph-mon[31838]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-19T03:26:39.969 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:39 smithi117 ceph-mon[31838]: from='osd.5 [v2:172.21.15.178:6808/1259631912,v1:172.21.15.178:6809/1259631912]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-19T03:26:39.969 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:39 smithi117 ceph-mon[31838]: osdmap e36: 6 total, 5 up, 6 in 2024-03-19T03:26:39.969 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T03:26:39.969 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:39 smithi117 ceph-mon[31838]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-19T03:26:39.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:39 smithi119 ceph-mon[66861]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-19T03:26:39.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:39 smithi119 ceph-mon[66861]: from='osd.5 [v2:172.21.15.178:6808/1259631912,v1:172.21.15.178:6809/1259631912]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-19T03:26:39.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:39 smithi119 ceph-mon[66861]: osdmap e36: 6 total, 5 up, 6 in 2024-03-19T03:26:39.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T03:26:39.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:39 smithi119 ceph-mon[66861]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-19T03:26:39.989 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:39 smithi178 ceph-mon[67578]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-19T03:26:39.989 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:39 smithi178 ceph-mon[67578]: from='osd.5 [v2:172.21.15.178:6808/1259631912,v1:172.21.15.178:6809/1259631912]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-19T03:26:39.989 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:39 smithi178 ceph-mon[67578]: osdmap e36: 6 total, 5 up, 6 in 2024-03-19T03:26:39.989 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T03:26:39.989 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:39 smithi178 ceph-mon[67578]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-19T03:26:39.989 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:26:39 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:26:39.669+0000 7fb073084700 -1 osd.5 0 waiting for initial osdmap 2024-03-19T03:26:39.989 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:26:39 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:26:39.675+0000 7fb06ee77700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T03:26:40.897 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:40 smithi117 ceph-mon[31838]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-03-19T03:26:40.897 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:40 smithi117 ceph-mon[31838]: osdmap e37: 6 total, 5 up, 6 in 2024-03-19T03:26:40.897 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:40 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T03:26:40.897 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:40 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T03:26:40.897 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:40 smithi117 ceph-mon[31838]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:40.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:40 smithi119 ceph-mon[66861]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-03-19T03:26:40.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:40 smithi119 ceph-mon[66861]: osdmap e37: 6 total, 5 up, 6 in 2024-03-19T03:26:40.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:40 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T03:26:40.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:40 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T03:26:40.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:40 smithi119 ceph-mon[66861]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:41.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:40 smithi178 ceph-mon[67578]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-03-19T03:26:41.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:40 smithi178 ceph-mon[67578]: osdmap e37: 6 total, 5 up, 6 in 2024-03-19T03:26:41.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:40 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T03:26:41.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:40 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T03:26:41.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:40 smithi178 ceph-mon[67578]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-03-19T03:26:41.782 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-19T03:26:41.782 INFO:teuthology.orchestra.run.smithi117.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-19T03:26:41.782 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10+0 records in 2024-03-19T03:26:41.782 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2024-03-19T03:26:41.782 INFO:teuthology.orchestra.run.smithi117.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0330073 s, 318 MB/s 2024-03-19T03:26:41.782 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping successful for: 2024-03-19T03:26:41.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:41 smithi178 ceph-mon[67578]: purged_snaps scrub starts 2024-03-19T03:26:41.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:41 smithi178 ceph-mon[67578]: purged_snaps scrub ok 2024-03-19T03:26:41.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:41 smithi178 ceph-mon[67578]: osd.5 [v2:172.21.15.178:6808/1259631912,v1:172.21.15.178:6809/1259631912] boot 2024-03-19T03:26:41.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:41 smithi178 ceph-mon[67578]: osdmap e38: 6 total, 6 up, 6 in 2024-03-19T03:26:41.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T03:26:41.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:41.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:41.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:26:41.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:26:41.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:41.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:41.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:26:41.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:41.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:41 smithi178 ceph-mon[67578]: osdmap e39: 6 total, 6 up, 6 in 2024-03-19T03:26:41.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:41 smithi119 ceph-mon[66861]: purged_snaps scrub starts 2024-03-19T03:26:41.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:41 smithi119 ceph-mon[66861]: purged_snaps scrub ok 2024-03-19T03:26:41.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:41 smithi119 ceph-mon[66861]: osd.5 [v2:172.21.15.178:6808/1259631912,v1:172.21.15.178:6809/1259631912] boot 2024-03-19T03:26:41.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:41 smithi119 ceph-mon[66861]: osdmap e38: 6 total, 6 up, 6 in 2024-03-19T03:26:41.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T03:26:41.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:41.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:41.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:26:41.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:26:41.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:41.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:41.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:26:41.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:41.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:41 smithi119 ceph-mon[66861]: osdmap e39: 6 total, 6 up, 6 in 2024-03-19T03:26:41.992 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:41 smithi117 ceph-mon[31838]: purged_snaps scrub starts 2024-03-19T03:26:41.992 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:41 smithi117 ceph-mon[31838]: purged_snaps scrub ok 2024-03-19T03:26:41.992 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:41 smithi117 ceph-mon[31838]: osd.5 [v2:172.21.15.178:6808/1259631912,v1:172.21.15.178:6809/1259631912] boot 2024-03-19T03:26:41.992 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:41 smithi117 ceph-mon[31838]: osdmap e38: 6 total, 6 up, 6 in 2024-03-19T03:26:41.992 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T03:26:41.992 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:41.992 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:41.992 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:26:41.992 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:26:41.992 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:41.992 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:41.993 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:26:41.993 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:26:41.993 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:41 smithi117 ceph-mon[31838]: osdmap e39: 6 total, 6 up, 6 in 2024-03-19T03:26:42.777 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi117:vg_nvme/lv_2 2024-03-19T03:26:42.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:42 smithi119 ceph-mon[66861]: Detected new or changed devices on smithi178 2024-03-19T03:26:42.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:42 smithi119 ceph-mon[66861]: Adjusting osd_memory_target on smithi178 to 8540M 2024-03-19T03:26:42.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:42 smithi119 ceph-mon[66861]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 164 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:26:42.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:42 smithi119 ceph-mon[66861]: osdmap e40: 6 total, 6 up, 6 in 2024-03-19T03:26:43.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:42 smithi117 ceph-mon[31838]: Detected new or changed devices on smithi178 2024-03-19T03:26:43.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:42 smithi117 ceph-mon[31838]: Adjusting osd_memory_target on smithi178 to 8540M 2024-03-19T03:26:43.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:42 smithi117 ceph-mon[31838]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 164 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:26:43.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:42 smithi117 ceph-mon[31838]: osdmap e40: 6 total, 6 up, 6 in 2024-03-19T03:26:43.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:42 smithi178 ceph-mon[67578]: Detected new or changed devices on smithi178 2024-03-19T03:26:43.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:42 smithi178 ceph-mon[67578]: Adjusting osd_memory_target on smithi178 to 8540M 2024-03-19T03:26:43.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:42 smithi178 ceph-mon[67578]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 164 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:26:43.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:42 smithi178 ceph-mon[67578]: osdmap e40: 6 total, 6 up, 6 in 2024-03-19T03:26:43.198 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:26:44.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:44 smithi119 ceph-mon[66861]: osdmap e41: 6 total, 6 up, 6 in 2024-03-19T03:26:44.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:44 smithi119 ceph-mon[66861]: pgmap v154: 1 pgs: 1 remapped+peering; 577 KiB data, 164 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:26:45.019 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:44 smithi117 ceph-mon[31838]: osdmap e41: 6 total, 6 up, 6 in 2024-03-19T03:26:45.020 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:44 smithi117 ceph-mon[31838]: pgmap v154: 1 pgs: 1 remapped+peering; 577 KiB data, 164 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:26:45.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:44 smithi178 ceph-mon[67578]: osdmap e41: 6 total, 6 up, 6 in 2024-03-19T03:26:45.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:44 smithi178 ceph-mon[67578]: pgmap v154: 1 pgs: 1 remapped+peering; 577 KiB data, 164 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:26:45.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.540+0000 7f4388200700 1 -- 172.21.15.117:0/3071214869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4380108030 msgr2=0x7f438010a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:26:45.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.541+0000 7f4388200700 1 --2- 172.21.15.117:0/3071214869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4380108030 0x7f438010a4a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4378009f90 tx=0x7f437800b330 comp rx=0 tx=0).stop 2024-03-19T03:26:45.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.541+0000 7f4388200700 1 -- 172.21.15.117:0/3071214869 shutdown_connections 2024-03-19T03:26:45.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.541+0000 7f4388200700 1 --2- 172.21.15.117:0/3071214869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4380108030 0x7f438010a4a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:26:45.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.541+0000 7f4388200700 1 --2- 172.21.15.117:0/3071214869 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4380102dd0 0x7f43801051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:26:45.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.541+0000 7f4388200700 1 --2- 172.21.15.117:0/3071214869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4380105700 0x7f4380107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:26:45.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.541+0000 7f4388200700 1 -- 172.21.15.117:0/3071214869 >> 172.21.15.117:0/3071214869 conn(0x7f43800fc7e0 msgr2=0x7f43800fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:26:45.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.541+0000 7f4388200700 1 -- 172.21.15.117:0/3071214869 shutdown_connections 2024-03-19T03:26:45.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.541+0000 7f4388200700 1 -- 172.21.15.117:0/3071214869 wait complete. 2024-03-19T03:26:45.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.542+0000 7f4388200700 1 Processor -- start 2024-03-19T03:26:45.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.542+0000 7f4388200700 1 -- start start 2024-03-19T03:26:45.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.543+0000 7f4388200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4380102dd0 0x7f438019a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:26:45.543 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.543+0000 7f4388200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4380105700 0x7f438019ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:26:45.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.543+0000 7f4388200700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4380108030 0x7f43801a1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:26:45.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.543+0000 7f4388200700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f438010dfa0 con 0x7f4380105700 2024-03-19T03:26:45.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.543+0000 7f4388200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f438010de20 con 0x7f4380102dd0 2024-03-19T03:26:45.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.543+0000 7f4388200700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f438010e120 con 0x7f4380108030 2024-03-19T03:26:45.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.543+0000 7f4384e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4380105700 0x7f438019ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:26:45.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.543+0000 7f4384e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4380105700 0x7f438019ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:52882/0 (socket says 172.21.15.117:52882) 2024-03-19T03:26:45.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.543+0000 7f4384e00700 1 -- 172.21.15.117:0/615988360 learned_addr learned my addr 172.21.15.117:0/615988360 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:26:45.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.543+0000 7f4385800700 1 --2- 172.21.15.117:0/615988360 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4380102dd0 0x7f438019a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:26:45.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.543+0000 7f4386200700 1 --2- 172.21.15.117:0/615988360 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4380108030 0x7f43801a1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:26:45.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.544+0000 7f4385800700 1 -- 172.21.15.117:0/615988360 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4380108030 msgr2=0x7f43801a1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:26:45.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.544+0000 7f4385800700 1 --2- 172.21.15.117:0/615988360 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4380108030 0x7f43801a1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:26:45.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.544+0000 7f4385800700 1 -- 172.21.15.117:0/615988360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4380105700 msgr2=0x7f438019ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:26:45.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.544+0000 7f4385800700 1 --2- 172.21.15.117:0/615988360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4380105700 0x7f438019ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:26:45.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.544+0000 7f4385800700 1 -- 172.21.15.117:0/615988360 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43801a2400 con 0x7f4380102dd0 2024-03-19T03:26:45.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.544+0000 7f4386200700 1 --2- 172.21.15.117:0/615988360 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4380108030 0x7f43801a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:26:45.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.544+0000 7f4384e00700 1 --2- 172.21.15.117:0/615988360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4380105700 0x7f438019ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:26:45.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.544+0000 7f4385800700 1 --2- 172.21.15.117:0/615988360 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4380102dd0 0x7f438019a7a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f437400e9f0 tx=0x7f437400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:26:45.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.544+0000 7f437e000700 1 -- 172.21.15.117:0/615988360 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437400c9c0 con 0x7f4380102dd0 2024-03-19T03:26:45.546 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.544+0000 7f4388200700 1 -- 172.21.15.117:0/615988360 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43801a26f0 con 0x7f4380102dd0 2024-03-19T03:26:45.546 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.544+0000 7f437e000700 1 -- 172.21.15.117:0/615988360 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f437400cb60 con 0x7f4380102dd0 2024-03-19T03:26:45.546 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.544+0000 7f437e000700 1 -- 172.21.15.117:0/615988360 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4374018680 con 0x7f4380102dd0 2024-03-19T03:26:45.546 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.544+0000 7f4388200700 1 -- 172.21.15.117:0/615988360 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43801a2c80 con 0x7f4380102dd0 2024-03-19T03:26:45.546 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.546+0000 7f4388200700 1 -- 172.21.15.117:0/615988360 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4380068fd0 con 0x7f4380102dd0 2024-03-19T03:26:45.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.547+0000 7f437e000700 1 -- 172.21.15.117:0/615988360 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f4374010430 con 0x7f4380102dd0 2024-03-19T03:26:45.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.549+0000 7f437e000700 1 --2- 172.21.15.117:0/615988360 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f43680a2910 0x7f43680a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:26:45.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.549+0000 7f437e000700 1 -- 172.21.15.117:0/615988360 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4533+0+0 (secure 0 0 0) 0x7f43740cdcb0 con 0x7f4380102dd0 2024-03-19T03:26:45.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.549+0000 7f4384e00700 1 --2- 172.21.15.117:0/615988360 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f43680a2910 0x7f43680a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:26:45.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.549+0000 7f4384e00700 1 --2- 172.21.15.117:0/615988360 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f43680a2910 0x7f43680a4dd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f436c005950 tx=0x7f436c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:26:45.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.553+0000 7f437e000700 1 -- 172.21.15.117:0/615988360 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f437409bb70 con 0x7f4380102dd0 2024-03-19T03:26:45.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:26:45.786+0000 7f4388200700 1 -- 172.21.15.117:0/615988360 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f4380062750 con 0x7f43680a2910 2024-03-19T03:26:45.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:45 smithi119 ceph-mon[66861]: osdmap e42: 6 total, 6 up, 6 in 2024-03-19T03:26:46.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:45 smithi117 ceph-mon[31838]: osdmap e42: 6 total, 6 up, 6 in 2024-03-19T03:26:46.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:45 smithi178 ceph-mon[67578]: osdmap e42: 6 total, 6 up, 6 in 2024-03-19T03:26:46.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:46 smithi119 ceph-mon[66861]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:26:46.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:46 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:26:46.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:46 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:26:46.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:46 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:46.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:46 smithi119 ceph-mon[66861]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-03-19T03:26:47.030 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:46 smithi117 ceph-mon[31838]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:26:47.031 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:46 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:26:47.031 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:46 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:26:47.031 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:46 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:47.031 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:46 smithi117 ceph-mon[31838]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-03-19T03:26:47.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:46 smithi178 ceph-mon[67578]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:26:47.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:46 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:26:47.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:46 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:26:47.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:46 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:47.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:46 smithi178 ceph-mon[67578]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-03-19T03:26:49.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:49 smithi117 ceph-mon[31838]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T03:26:49.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:49 smithi178 ceph-mon[67578]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T03:26:49.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:49 smithi119 ceph-mon[66861]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T03:26:50.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:50 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3268287340' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75c2ddd7-f837-42e0-ac47-8cfbcf97a52a"}]: dispatch 2024-03-19T03:26:50.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:50 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3268287340' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75c2ddd7-f837-42e0-ac47-8cfbcf97a52a"}]': finished 2024-03-19T03:26:50.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:50 smithi117 ceph-mon[31838]: osdmap e43: 7 total, 6 up, 7 in 2024-03-19T03:26:50.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:50 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T03:26:50.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:50 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3268287340' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75c2ddd7-f837-42e0-ac47-8cfbcf97a52a"}]: dispatch 2024-03-19T03:26:50.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:50 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3268287340' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75c2ddd7-f837-42e0-ac47-8cfbcf97a52a"}]': finished 2024-03-19T03:26:50.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:50 smithi178 ceph-mon[67578]: osdmap e43: 7 total, 6 up, 7 in 2024-03-19T03:26:50.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:50 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T03:26:50.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:50 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3268287340' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75c2ddd7-f837-42e0-ac47-8cfbcf97a52a"}]: dispatch 2024-03-19T03:26:50.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:50 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3268287340' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75c2ddd7-f837-42e0-ac47-8cfbcf97a52a"}]': finished 2024-03-19T03:26:50.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:50 smithi119 ceph-mon[66861]: osdmap e43: 7 total, 6 up, 7 in 2024-03-19T03:26:50.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:50 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T03:26:51.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:51 smithi117 ceph-mon[31838]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-19T03:26:51.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:51 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2498758709' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:26:51.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:51 smithi178 ceph-mon[67578]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-19T03:26:51.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:51 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2498758709' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:26:51.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:51 smithi119 ceph-mon[66861]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-19T03:26:51.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:51 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2498758709' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:26:53.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:53 smithi117 ceph-mon[31838]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T03:26:53.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:53 smithi178 ceph-mon[67578]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T03:26:53.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:53 smithi119 ceph-mon[66861]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T03:26:55.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:55 smithi117 ceph-mon[31838]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-19T03:26:55.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:55 smithi178 ceph-mon[67578]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-19T03:26:55.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:55 smithi119 ceph-mon[66861]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-19T03:26:57.498 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:57 smithi117 ceph-mon[31838]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:26:57.536 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:57 smithi119 ceph-mon[66861]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:26:57.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:57 smithi178 ceph-mon[67578]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:26:59.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:59 smithi117 ceph-mon[31838]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:26:59.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:59 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-19T03:26:59.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:26:59 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:59.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:59 smithi178 ceph-mon[67578]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:26:59.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:59 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-19T03:26:59.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:26:59 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:26:59.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:59 smithi119 ceph-mon[66861]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:26:59.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:59 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-19T03:26:59.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:26:59 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:27:00.461 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:00 smithi117 ceph-mon[31838]: Deploying daemon osd.6 on smithi117 2024-03-19T03:27:00.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:00 smithi178 ceph-mon[67578]: Deploying daemon osd.6 on smithi117 2024-03-19T03:27:00.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:00 smithi119 ceph-mon[66861]: Deploying daemon osd.6 on smithi117 2024-03-19T03:27:01.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:01 smithi178 ceph-mon[67578]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:27:01.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:01 smithi119 ceph-mon[66861]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:27:01.746 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:01 smithi117 ceph-mon[31838]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:27:03.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:03 smithi117 ceph-mon[31838]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:27:03.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:03 smithi178 ceph-mon[67578]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:27:03.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:03 smithi119 ceph-mon[66861]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:27:05.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:05 smithi178 ceph-mon[67578]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:27:05.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:05 smithi119 ceph-mon[66861]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:27:05.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:05 smithi117 ceph-mon[31838]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:27:07.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:07 smithi178 ceph-mon[67578]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:27:07.722 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:07 smithi117 ceph-mon[31838]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:27:07.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:07 smithi119 ceph-mon[66861]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:27:08.857 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:08.857 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:08.857 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:27:08.857 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:08 smithi117 ceph-mon[31838]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:27:08.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:08.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:08.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:27:08.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:08 smithi119 ceph-mon[66861]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:27:09.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:09.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:09.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:27:09.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:08 smithi178 ceph-mon[67578]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:27:11.533 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:11 smithi117 ceph-mon[31838]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:27:11.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:11 smithi178 ceph-mon[67578]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:27:11.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:11 smithi119 ceph-mon[66861]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:27:12.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:27:12.705+0000 7f437e000700 1 -- 172.21.15.117:0/615988360 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4380062750 con 0x7f43680a2910 2024-03-19T03:27:12.707 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:12 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:12.707 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:12 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:12.707 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:12 smithi117 ceph-mon[31838]: from='osd.6 [v2:172.21.15.117:6818/864146757,v1:172.21.15.117:6819/864146757]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-19T03:27:12.707 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:12 smithi117 ceph-mon[31838]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-19T03:27:12.708 INFO:teuthology.orchestra.run.smithi117.stdout:Created osd(s) 6 on host 'smithi117' 2024-03-19T03:27:12.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:27:12.709+0000 7f4388200700 1 -- 172.21.15.117:0/615988360 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f43680a2910 msgr2=0x7f43680a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:27:12.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:27:12.709+0000 7f4388200700 1 --2- 172.21.15.117:0/615988360 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f43680a2910 0x7f43680a4dd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f436c005950 tx=0x7f436c009310 comp rx=0 tx=0).stop 2024-03-19T03:27:12.711 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:27:12.709+0000 7f4388200700 1 -- 172.21.15.117:0/615988360 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4380102dd0 msgr2=0x7f438019a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:27:12.711 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:27:12.709+0000 7f4388200700 1 --2- 172.21.15.117:0/615988360 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4380102dd0 0x7f438019a7a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f437400e9f0 tx=0x7f437400ed00 comp rx=0 tx=0).stop 2024-03-19T03:27:12.711 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:27:12.709+0000 7f4388200700 1 -- 172.21.15.117:0/615988360 shutdown_connections 2024-03-19T03:27:12.711 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:27:12.709+0000 7f4388200700 1 --2- 172.21.15.117:0/615988360 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f43680a2910 0x7f43680a4dd0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:27:12.711 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:27:12.709+0000 7f4388200700 1 --2- 172.21.15.117:0/615988360 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4380108030 0x7f43801a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:27:12.711 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:27:12.709+0000 7f4388200700 1 --2- 172.21.15.117:0/615988360 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4380102dd0 0x7f438019a7a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:27:12.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:27:12.709+0000 7f4388200700 1 --2- 172.21.15.117:0/615988360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4380105700 0x7f438019ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:27:12.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:27:12.709+0000 7f4388200700 1 -- 172.21.15.117:0/615988360 >> 172.21.15.117:0/615988360 conn(0x7f43800fc7e0 msgr2=0x7f4380102520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:27:12.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:27:12.709+0000 7f4388200700 1 -- 172.21.15.117:0/615988360 shutdown_connections 2024-03-19T03:27:12.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:27:12.709+0000 7f4388200700 1 -- 172.21.15.117:0/615988360 wait complete. 2024-03-19T03:27:12.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:12 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:12.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:12 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:12.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:12 smithi119 ceph-mon[66861]: from='osd.6 [v2:172.21.15.117:6818/864146757,v1:172.21.15.117:6819/864146757]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-19T03:27:12.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:12 smithi119 ceph-mon[66861]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-19T03:27:12.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:12 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:12.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:12 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:12.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:12 smithi178 ceph-mon[67578]: from='osd.6 [v2:172.21.15.117:6818/864146757,v1:172.21.15.117:6819/864146757]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-19T03:27:12.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:12 smithi178 ceph-mon[67578]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-19T03:27:13.500 DEBUG:teuthology.orchestra.run.smithi117:osd.6> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.6.service 2024-03-19T03:27:13.506 INFO:tasks.cephadm:Deploying osd.7 on smithi119 with /dev/vg_nvme/lv_2... 2024-03-19T03:27:13.507 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-19T03:27:13.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:13 smithi119 ceph-mon[66861]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:27:13.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:13 smithi119 ceph-mon[66861]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-19T03:27:13.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:13 smithi119 ceph-mon[66861]: from='osd.6 [v2:172.21.15.117:6818/864146757,v1:172.21.15.117:6819/864146757]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-19T03:27:13.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:13 smithi119 ceph-mon[66861]: osdmap e44: 7 total, 6 up, 7 in 2024-03-19T03:27:13.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:13 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T03:27:13.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:13 smithi119 ceph-mon[66861]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-19T03:27:13.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:13 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:27:13.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:13 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:27:13.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:13 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:13.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:13 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:13.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:13 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:13.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:13 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:27:13.751 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T03:27:13.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:13 smithi117 ceph-mon[31838]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:27:13.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:13 smithi117 ceph-mon[31838]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-19T03:27:13.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:13 smithi117 ceph-mon[31838]: from='osd.6 [v2:172.21.15.117:6818/864146757,v1:172.21.15.117:6819/864146757]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-19T03:27:13.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:13 smithi117 ceph-mon[31838]: osdmap e44: 7 total, 6 up, 7 in 2024-03-19T03:27:13.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:13 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T03:27:13.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:13 smithi117 ceph-mon[31838]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-19T03:27:13.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:13 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:27:13.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:13 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:27:13.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:13 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:13.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:13 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:13.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:13 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:13.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:13 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:27:13.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:13 smithi178 ceph-mon[67578]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:27:13.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:13 smithi178 ceph-mon[67578]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-19T03:27:13.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:13 smithi178 ceph-mon[67578]: from='osd.6 [v2:172.21.15.117:6818/864146757,v1:172.21.15.117:6819/864146757]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-19T03:27:13.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:13 smithi178 ceph-mon[67578]: osdmap e44: 7 total, 6 up, 7 in 2024-03-19T03:27:13.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:13 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T03:27:13.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:13 smithi178 ceph-mon[67578]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-19T03:27:13.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:13 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:27:13.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:13 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:27:13.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:13 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:13.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:13 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:13.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:13 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:13.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:13 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:27:14.419 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:27:14 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:27:14.071+0000 7f36cde00700 -1 osd.6 0 waiting for initial osdmap 2024-03-19T03:27:14.419 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:27:14 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:27:14.092+0000 7f36c3c00700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T03:27:14.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:14 smithi119 ceph-mon[66861]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-19T03:27:14.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:14 smithi119 ceph-mon[66861]: osdmap e45: 7 total, 6 up, 7 in 2024-03-19T03:27:14.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:14 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T03:27:14.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:14 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T03:27:14.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:14 smithi119 ceph-mon[66861]: from='osd.6 ' entity='osd.6' 2024-03-19T03:27:14.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:14 smithi119 ceph-mon[66861]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:27:14.725 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:14 smithi117 ceph-mon[31838]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-19T03:27:14.725 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:14 smithi117 ceph-mon[31838]: osdmap e45: 7 total, 6 up, 7 in 2024-03-19T03:27:14.725 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:14 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T03:27:14.725 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:14 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T03:27:14.726 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:14 smithi117 ceph-mon[31838]: from='osd.6 ' entity='osd.6' 2024-03-19T03:27:14.726 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:14 smithi117 ceph-mon[31838]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:27:14.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:14 smithi178 ceph-mon[67578]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-19T03:27:14.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:14 smithi178 ceph-mon[67578]: osdmap e45: 7 total, 6 up, 7 in 2024-03-19T03:27:14.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:14 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T03:27:14.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:14 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T03:27:14.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:14 smithi178 ceph-mon[67578]: from='osd.6 ' entity='osd.6' 2024-03-19T03:27:14.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:14 smithi178 ceph-mon[67578]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-03-19T03:27:15.638 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:15 smithi117 ceph-mon[31838]: purged_snaps scrub starts 2024-03-19T03:27:15.638 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:15 smithi117 ceph-mon[31838]: purged_snaps scrub ok 2024-03-19T03:27:15.638 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:15 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T03:27:15.638 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:15 smithi117 ceph-mon[31838]: osd.6 [v2:172.21.15.117:6818/864146757,v1:172.21.15.117:6819/864146757] boot 2024-03-19T03:27:15.638 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:15 smithi117 ceph-mon[31838]: osdmap e46: 7 total, 7 up, 7 in 2024-03-19T03:27:15.638 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:15 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T03:27:15.709 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:15 smithi119 ceph-mon[66861]: purged_snaps scrub starts 2024-03-19T03:27:15.710 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:15 smithi119 ceph-mon[66861]: purged_snaps scrub ok 2024-03-19T03:27:15.710 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:15 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T03:27:15.710 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:15 smithi119 ceph-mon[66861]: osd.6 [v2:172.21.15.117:6818/864146757,v1:172.21.15.117:6819/864146757] boot 2024-03-19T03:27:15.710 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:15 smithi119 ceph-mon[66861]: osdmap e46: 7 total, 7 up, 7 in 2024-03-19T03:27:15.710 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:15 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T03:27:15.710 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-19T03:27:15.710 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-19T03:27:15.710 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-03-19T03:27:15.710 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-03-19T03:27:15.710 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0118235 s, 887 MB/s 2024-03-19T03:27:15.711 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-03-19T03:27:15.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:15 smithi178 ceph-mon[67578]: purged_snaps scrub starts 2024-03-19T03:27:15.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:15 smithi178 ceph-mon[67578]: purged_snaps scrub ok 2024-03-19T03:27:15.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:15 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T03:27:15.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:15 smithi178 ceph-mon[67578]: osd.6 [v2:172.21.15.117:6818/864146757,v1:172.21.15.117:6819/864146757] boot 2024-03-19T03:27:15.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:15 smithi178 ceph-mon[67578]: osdmap e46: 7 total, 7 up, 7 in 2024-03-19T03:27:15.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:15 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T03:27:16.408 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi119:vg_nvme/lv_2 2024-03-19T03:27:16.652 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T03:27:16.682 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:16 smithi119 ceph-mon[66861]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:16.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:16 smithi117 ceph-mon[31838]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:16.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:16 smithi178 ceph-mon[67578]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:17 smithi119 ceph-mon[66861]: osdmap e47: 7 total, 7 up, 7 in 2024-03-19T03:27:17.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:17 smithi117 ceph-mon[31838]: osdmap e47: 7 total, 7 up, 7 in 2024-03-19T03:27:17.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:17 smithi178 ceph-mon[67578]: osdmap e47: 7 total, 7 up, 7 in 2024-03-19T03:27:18.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.282+0000 7f32ab59d700 1 -- 172.21.15.119:0/169271918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32a4101270 msgr2=0x7f32a4101690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:27:18.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.282+0000 7f32ab59d700 1 --2- 172.21.15.119:0/169271918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32a4101270 0x7f32a4101690 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f32a00099c0 tx=0x7f32a0009cd0 comp rx=0 tx=0).stop 2024-03-19T03:27:18.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.283+0000 7f32ab59d700 1 -- 172.21.15.119:0/169271918 shutdown_connections 2024-03-19T03:27:18.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.283+0000 7f32ab59d700 1 --2- 172.21.15.119:0/169271918 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f32a4102e10 0x7f32a41096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:27:18.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.283+0000 7f32ab59d700 1 --2- 172.21.15.119:0/169271918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32a4101270 0x7f32a4101690 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:27:18.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.283+0000 7f32ab59d700 1 --2- 172.21.15.119:0/169271918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32a4102470 0x7f32a41028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:27:18.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.283+0000 7f32ab59d700 1 -- 172.21.15.119:0/169271918 >> 172.21.15.119:0/169271918 conn(0x7f32a40fc830 msgr2=0x7f32a40fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:27:18.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.283+0000 7f32ab59d700 1 -- 172.21.15.119:0/169271918 shutdown_connections 2024-03-19T03:27:18.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.283+0000 7f32ab59d700 1 -- 172.21.15.119:0/169271918 wait complete. 2024-03-19T03:27:18.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.283+0000 7f32ab59d700 1 Processor -- start 2024-03-19T03:27:18.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.283+0000 7f32ab59d700 1 -- start start 2024-03-19T03:27:18.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.283+0000 7f32ab59d700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f32a4101270 0x7f32a406b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:27:18.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.283+0000 7f32ab59d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32a4102470 0x7f32a419d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:27:18.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.283+0000 7f32ab59d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32a4102e10 0x7f32a419f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:27:18.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.284+0000 7f32ab59d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f32a410ceb0 con 0x7f32a4102470 2024-03-19T03:27:18.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.284+0000 7f32ab59d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f32a410cd30 con 0x7f32a4102e10 2024-03-19T03:27:18.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.284+0000 7f32ab59d700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f32a410d030 con 0x7f32a4101270 2024-03-19T03:27:18.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.284+0000 7f32a9b3a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32a4102e10 0x7f32a419f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:27:18.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.284+0000 7f32a9b3a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32a4102e10 0x7f32a419f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60464/0 (socket says 172.21.15.119:60464) 2024-03-19T03:27:18.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.284+0000 7f32a9b3a700 1 -- 172.21.15.119:0/1348084158 learned_addr learned my addr 172.21.15.119:0/1348084158 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-19T03:27:18.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.284+0000 7f32a9339700 1 --2- 172.21.15.119:0/1348084158 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f32a4101270 0x7f32a406b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:27:18.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.284+0000 7f32a9b3a700 1 -- 172.21.15.119:0/1348084158 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f32a4101270 msgr2=0x7f32a406b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:27:18.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.284+0000 7f32a9b3a700 1 --2- 172.21.15.119:0/1348084158 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f32a4101270 0x7f32a406b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:27:18.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.284+0000 7f32a9b3a700 1 -- 172.21.15.119:0/1348084158 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32a4102470 msgr2=0x7f32a419d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:27:18.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.284+0000 7f32a9b3a700 1 --2- 172.21.15.119:0/1348084158 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32a4102470 0x7f32a419d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:27:18.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.284+0000 7f32a9b3a700 1 -- 172.21.15.119:0/1348084158 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32a41a01c0 con 0x7f32a4102e10 2024-03-19T03:27:18.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.284+0000 7f32a9b3a700 1 --2- 172.21.15.119:0/1348084158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32a4102e10 0x7f32a419f8f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f329c007d70 tx=0x7f329c00a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:27:18.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.284+0000 7f329a7fc700 1 -- 172.21.15.119:0/1348084158 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f329c016020 con 0x7f32a4102e10 2024-03-19T03:27:18.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.284+0000 7f32ab59d700 1 -- 172.21.15.119:0/1348084158 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32a41a04b0 con 0x7f32a4102e10 2024-03-19T03:27:18.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.284+0000 7f329a7fc700 1 -- 172.21.15.119:0/1348084158 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f329c00ad40 con 0x7f32a4102e10 2024-03-19T03:27:18.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.284+0000 7f329a7fc700 1 -- 172.21.15.119:0/1348084158 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f329c0115c0 con 0x7f32a4102e10 2024-03-19T03:27:18.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.284+0000 7f32ab59d700 1 -- 172.21.15.119:0/1348084158 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32a41a0ac0 con 0x7f32a4102e10 2024-03-19T03:27:18.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.285+0000 7f32ab59d700 1 -- 172.21.15.119:0/1348084158 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32a4068fd0 con 0x7f32a4102e10 2024-03-19T03:27:18.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.286+0000 7f329a7fc700 1 -- 172.21.15.119:0/1348084158 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f329c019020 con 0x7f32a4102e10 2024-03-19T03:27:18.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.287+0000 7f329a7fc700 1 --2- 172.21.15.119:0/1348084158 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f32900a2890 0x7f32900a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:27:18.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.287+0000 7f329a7fc700 1 -- 172.21.15.119:0/1348084158 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7f329c0cdd00 con 0x7f32a4102e10 2024-03-19T03:27:18.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.287+0000 7f32a9339700 1 --2- 172.21.15.119:0/1348084158 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f32900a2890 0x7f32900a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:27:18.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.288+0000 7f32a9339700 1 --2- 172.21.15.119:0/1348084158 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f32900a2890 0x7f32900a4d50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f32a0003890 tx=0x7f32a0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:27:18.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.291+0000 7f329a7fc700 1 -- 172.21.15.119:0/1348084158 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f329c020020 con 0x7f32a4102e10 2024-03-19T03:27:18.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:18.498+0000 7f32ab59d700 1 -- 172.21.15.119:0/1348084158 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f32a4062750 con 0x7f32900a2890 2024-03-19T03:27:19.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:19 smithi119 ceph-mon[66861]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:19.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:19 smithi119 ceph-mon[66861]: Detected new or changed devices on smithi117 2024-03-19T03:27:19.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:19 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:19.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:19 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:19.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:19 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:27:19.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:19 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:27:19.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:19 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:27:19.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:19 smithi119 ceph-mon[66861]: Adjusting osd_memory_target on smithi117 to 5697M 2024-03-19T03:27:19.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:19 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:19.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:19 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:27:19.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:19 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:27:19.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:19 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:19.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:19 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:27:19.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:19 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:27:19.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:19 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:27:19.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:19 smithi117 ceph-mon[31838]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:19.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:19 smithi117 ceph-mon[31838]: Detected new or changed devices on smithi117 2024-03-19T03:27:19.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:19 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:19.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:19 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:19.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:19 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:27:19.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:19 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:27:19.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:19 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:27:19.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:19 smithi117 ceph-mon[31838]: Adjusting osd_memory_target on smithi117 to 5697M 2024-03-19T03:27:19.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:19 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:19.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:19 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:27:19.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:19 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:27:19.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:19 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:19.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:19 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:27:19.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:19 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:27:19.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:19 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:27:19.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:19 smithi178 ceph-mon[67578]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:19.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:19 smithi178 ceph-mon[67578]: Detected new or changed devices on smithi117 2024-03-19T03:27:19.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:19 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:19.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:19 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:19.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:19 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:27:19.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:19 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:27:19.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:19 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:27:19.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:19 smithi178 ceph-mon[67578]: Adjusting osd_memory_target on smithi117 to 5697M 2024-03-19T03:27:19.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:19 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:19.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:19 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:27:19.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:19 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:27:19.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:19 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:19.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:19 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:27:19.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:19 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:27:19.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:19 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:27:20.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:20 smithi119 ceph-mon[66861]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:27:20.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:20 smithi117 ceph-mon[31838]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:27:20.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:20 smithi178 ceph-mon[67578]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:27:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:21 smithi119 ceph-mon[66861]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:21 smithi119 ceph-mon[66861]: from='client.? 172.21.15.119:0/96164763' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c"}]: dispatch 2024-03-19T03:27:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:21 smithi119 ceph-mon[66861]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c"}]: dispatch 2024-03-19T03:27:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:21 smithi119 ceph-mon[66861]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c"}]': finished 2024-03-19T03:27:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:21 smithi119 ceph-mon[66861]: osdmap e48: 8 total, 7 up, 8 in 2024-03-19T03:27:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:21 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T03:27:21.747 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:21 smithi178 ceph-mon[67578]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:21.747 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:21 smithi178 ceph-mon[67578]: from='client.? 172.21.15.119:0/96164763' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c"}]: dispatch 2024-03-19T03:27:21.747 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:21 smithi178 ceph-mon[67578]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c"}]: dispatch 2024-03-19T03:27:21.747 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:21 smithi178 ceph-mon[67578]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c"}]': finished 2024-03-19T03:27:21.747 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:21 smithi178 ceph-mon[67578]: osdmap e48: 8 total, 7 up, 8 in 2024-03-19T03:27:21.747 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:21 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T03:27:21.795 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:21 smithi117 ceph-mon[31838]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:21.795 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:21 smithi117 ceph-mon[31838]: from='client.? 172.21.15.119:0/96164763' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c"}]: dispatch 2024-03-19T03:27:21.795 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:21 smithi117 ceph-mon[31838]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c"}]: dispatch 2024-03-19T03:27:21.795 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:21 smithi117 ceph-mon[31838]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c"}]': finished 2024-03-19T03:27:21.795 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:21 smithi117 ceph-mon[31838]: osdmap e48: 8 total, 7 up, 8 in 2024-03-19T03:27:21.795 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:21 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T03:27:22.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:22 smithi119 ceph-mon[66861]: from='client.? 172.21.15.119:0/4158168168' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:27:22.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:22 smithi117 ceph-mon[31838]: from='client.? 172.21.15.119:0/4158168168' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:27:22.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:22 smithi178 ceph-mon[67578]: from='client.? 172.21.15.119:0/4158168168' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:27:23.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:23 smithi119 ceph-mon[66861]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:23.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:23 smithi117 ceph-mon[31838]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:23.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:23 smithi178 ceph-mon[67578]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:24.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:24 smithi119 ceph-mon[66861]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:24.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:24 smithi117 ceph-mon[31838]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:24.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:24 smithi178 ceph-mon[67578]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:27.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:27 smithi117 ceph-mon[31838]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:27.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:27 smithi178 ceph-mon[67578]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:27.691 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:27 smithi119 ceph-mon[66861]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:29.465 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:29 smithi119 ceph-mon[66861]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:29.465 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:29 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-19T03:27:29.465 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:29 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:27:29.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:29 smithi117 ceph-mon[31838]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:29.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:29 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-19T03:27:29.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:29 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:27:29.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:29 smithi178 ceph-mon[67578]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:29.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:29 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-19T03:27:29.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:29 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:27:30.440 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:30 smithi119 ceph-mon[66861]: Deploying daemon osd.7 on smithi119 2024-03-19T03:27:30.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:30 smithi117 ceph-mon[31838]: Deploying daemon osd.7 on smithi119 2024-03-19T03:27:30.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:30 smithi178 ceph-mon[67578]: Deploying daemon osd.7 on smithi119 2024-03-19T03:27:31.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:31 smithi119 ceph-mon[66861]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:31.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:31 smithi178 ceph-mon[67578]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:31.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:31 smithi117 ceph-mon[31838]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:33.478 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:33 smithi119 ceph-mon[66861]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:33.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:33 smithi178 ceph-mon[67578]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:33.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:33 smithi117 ceph-mon[31838]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:35.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:35 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:35.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:35 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:27:35.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:35 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:35.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:35 smithi178 ceph-mon[67578]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:35.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:35 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:35.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:35 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:27:35.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:35 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:35.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:35 smithi119 ceph-mon[66861]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:35.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:35 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:35.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:35 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:27:35.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:35 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:35.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:35 smithi117 ceph-mon[31838]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:37.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:37 smithi117 ceph-mon[31838]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:37.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:37.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:37.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:27:37.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:27:37.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:37.584 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:37 smithi119 ceph-mon[66861]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:37.584 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:37 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:37.584 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:37 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:37.584 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:37 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:27:37.585 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:37 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:27:37.585 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:37 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:37.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:37 smithi178 ceph-mon[67578]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:37.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:37.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:37.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:27:37.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:27:37.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:37.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:37.981+0000 7f329a7fc700 1 -- 172.21.15.119:0/1348084158 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f32a4062750 con 0x7f32900a2890 2024-03-19T03:27:37.982 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 7 on host 'smithi119' 2024-03-19T03:27:37.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:37.983+0000 7f32ab59d700 1 -- 172.21.15.119:0/1348084158 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f32900a2890 msgr2=0x7f32900a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:27:37.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:37.983+0000 7f32ab59d700 1 --2- 172.21.15.119:0/1348084158 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f32900a2890 0x7f32900a4d50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f32a0003890 tx=0x7f32a0005fb0 comp rx=0 tx=0).stop 2024-03-19T03:27:37.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:37.983+0000 7f32ab59d700 1 -- 172.21.15.119:0/1348084158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32a4102e10 msgr2=0x7f32a419f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:27:37.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:37.983+0000 7f32ab59d700 1 --2- 172.21.15.119:0/1348084158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32a4102e10 0x7f32a419f8f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f329c007d70 tx=0x7f329c00a340 comp rx=0 tx=0).stop 2024-03-19T03:27:37.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:37.983+0000 7f32ab59d700 1 -- 172.21.15.119:0/1348084158 shutdown_connections 2024-03-19T03:27:37.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:37.983+0000 7f32ab59d700 1 --2- 172.21.15.119:0/1348084158 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f32900a2890 0x7f32900a4d50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:27:37.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:37.983+0000 7f32ab59d700 1 --2- 172.21.15.119:0/1348084158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32a4102e10 0x7f32a419f8f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:27:37.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:37.983+0000 7f32ab59d700 1 --2- 172.21.15.119:0/1348084158 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f32a4101270 0x7f32a406b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:27:37.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:37.983+0000 7f32ab59d700 1 --2- 172.21.15.119:0/1348084158 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32a4102470 0x7f32a419d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:27:37.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:37.983+0000 7f32ab59d700 1 -- 172.21.15.119:0/1348084158 >> 172.21.15.119:0/1348084158 conn(0x7f32a40fc830 msgr2=0x7f32a4105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:27:37.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:37.983+0000 7f32ab59d700 1 -- 172.21.15.119:0/1348084158 shutdown_connections 2024-03-19T03:27:37.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:27:37.983+0000 7f32ab59d700 1 -- 172.21.15.119:0/1348084158 wait complete. 2024-03-19T03:27:38.621 DEBUG:teuthology.orchestra.run.smithi119:osd.7> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.7.service 2024-03-19T03:27:38.624 INFO:tasks.cephadm:Deploying osd.8 on smithi178 with /dev/vg_nvme/lv_2... 2024-03-19T03:27:38.624 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-19T03:27:38.872 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T03:27:39.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:38 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:39.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:38 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:39.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:38 smithi117 ceph-mon[31838]: from='osd.7 [v2:172.21.15.119:6816/1273365545,v1:172.21.15.119:6817/1273365545]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-19T03:27:39.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:38 smithi117 ceph-mon[31838]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-19T03:27:39.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:38 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:27:39.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:38 smithi117 ceph-mon[31838]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:39.143 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:38 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:39.143 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:38 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:39.143 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:38 smithi119 ceph-mon[66861]: from='osd.7 [v2:172.21.15.119:6816/1273365545,v1:172.21.15.119:6817/1273365545]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-19T03:27:39.143 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:38 smithi119 ceph-mon[66861]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-19T03:27:39.143 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:38 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:27:39.143 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:38 smithi119 ceph-mon[66861]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:39.252 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:38 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:39.253 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:38 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:39.253 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:38 smithi178 ceph-mon[67578]: from='osd.7 [v2:172.21.15.119:6816/1273365545,v1:172.21.15.119:6817/1273365545]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-19T03:27:39.253 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:38 smithi178 ceph-mon[67578]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-19T03:27:39.253 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:38 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:27:39.253 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:38 smithi178 ceph-mon[67578]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:40.115 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:39 smithi119 ceph-mon[66861]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-19T03:27:40.115 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:39 smithi119 ceph-mon[66861]: from='osd.7 [v2:172.21.15.119:6816/1273365545,v1:172.21.15.119:6817/1273365545]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-19T03:27:40.115 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:39 smithi119 ceph-mon[66861]: osdmap e49: 8 total, 7 up, 8 in 2024-03-19T03:27:40.115 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T03:27:40.115 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:39 smithi119 ceph-mon[66861]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-19T03:27:40.277 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:39 smithi178 ceph-mon[67578]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-19T03:27:40.277 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:39 smithi178 ceph-mon[67578]: from='osd.7 [v2:172.21.15.119:6816/1273365545,v1:172.21.15.119:6817/1273365545]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-19T03:27:40.277 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:39 smithi178 ceph-mon[67578]: osdmap e49: 8 total, 7 up, 8 in 2024-03-19T03:27:40.277 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T03:27:40.277 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:39 smithi178 ceph-mon[67578]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-19T03:27:40.291 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:39 smithi117 ceph-mon[31838]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-19T03:27:40.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:39 smithi117 ceph-mon[31838]: from='osd.7 [v2:172.21.15.119:6816/1273365545,v1:172.21.15.119:6817/1273365545]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-19T03:27:40.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:39 smithi117 ceph-mon[31838]: osdmap e49: 8 total, 7 up, 8 in 2024-03-19T03:27:40.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T03:27:40.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:39 smithi117 ceph-mon[31838]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-19T03:27:40.734 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:27:40 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:27:40.378+0000 7fc5c458c700 -1 osd.7 0 waiting for initial osdmap 2024-03-19T03:27:40.734 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:27:40 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:27:40.387+0000 7fc5bc377700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T03:27:40.855 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-19T03:27:40.855 INFO:teuthology.orchestra.run.smithi178.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-19T03:27:40.855 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10+0 records in 2024-03-19T03:27:40.855 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records out 2024-03-19T03:27:40.855 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00862822 s, 1.2 GB/s 2024-03-19T03:27:40.855 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping successful for: 2024-03-19T03:27:41.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:40 smithi178 ceph-mon[67578]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-19T03:27:41.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:40 smithi178 ceph-mon[67578]: osdmap e50: 8 total, 7 up, 8 in 2024-03-19T03:27:41.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:40 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T03:27:41.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:40 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T03:27:41.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:40 smithi178 ceph-mon[67578]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:41.180 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:40 smithi178 ceph-mon[67578]: from='osd.7 ' entity='osd.7' 2024-03-19T03:27:41.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:40 smithi119 ceph-mon[66861]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-19T03:27:41.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:40 smithi119 ceph-mon[66861]: osdmap e50: 8 total, 7 up, 8 in 2024-03-19T03:27:41.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:40 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T03:27:41.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:40 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T03:27:41.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:40 smithi119 ceph-mon[66861]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:41.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:40 smithi119 ceph-mon[66861]: from='osd.7 ' entity='osd.7' 2024-03-19T03:27:41.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:40 smithi117 ceph-mon[31838]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-19T03:27:41.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:40 smithi117 ceph-mon[31838]: osdmap e50: 8 total, 7 up, 8 in 2024-03-19T03:27:41.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:40 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T03:27:41.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:40 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T03:27:41.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:40 smithi117 ceph-mon[31838]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-19T03:27:41.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:40 smithi117 ceph-mon[31838]: from='osd.7 ' entity='osd.7' 2024-03-19T03:27:41.542 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi178:vg_nvme/lv_2 2024-03-19T03:27:41.784 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T03:27:42.087 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:41 smithi178 ceph-mon[67578]: purged_snaps scrub starts 2024-03-19T03:27:42.087 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:41 smithi178 ceph-mon[67578]: purged_snaps scrub ok 2024-03-19T03:27:42.087 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:41 smithi178 ceph-mon[67578]: osd.7 [v2:172.21.15.119:6816/1273365545,v1:172.21.15.119:6817/1273365545] boot 2024-03-19T03:27:42.088 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:41 smithi178 ceph-mon[67578]: osdmap e51: 8 total, 8 up, 8 in 2024-03-19T03:27:42.088 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T03:27:42.088 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:42.088 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:42.088 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:27:42.088 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:27:42.088 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:27:42.088 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:42.088 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:27:42.088 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:27:42.088 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:42.088 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:41 smithi178 ceph-mon[67578]: osdmap e52: 8 total, 8 up, 8 in 2024-03-19T03:27:42.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:41 smithi119 ceph-mon[66861]: purged_snaps scrub starts 2024-03-19T03:27:42.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:41 smithi119 ceph-mon[66861]: purged_snaps scrub ok 2024-03-19T03:27:42.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:41 smithi119 ceph-mon[66861]: osd.7 [v2:172.21.15.119:6816/1273365545,v1:172.21.15.119:6817/1273365545] boot 2024-03-19T03:27:42.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:41 smithi119 ceph-mon[66861]: osdmap e51: 8 total, 8 up, 8 in 2024-03-19T03:27:42.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T03:27:42.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:42.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:42.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:27:42.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:27:42.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:27:42.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:42.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:27:42.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:27:42.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:42.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:41 smithi119 ceph-mon[66861]: osdmap e52: 8 total, 8 up, 8 in 2024-03-19T03:27:42.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:41 smithi117 ceph-mon[31838]: purged_snaps scrub starts 2024-03-19T03:27:42.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:41 smithi117 ceph-mon[31838]: purged_snaps scrub ok 2024-03-19T03:27:42.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:41 smithi117 ceph-mon[31838]: osd.7 [v2:172.21.15.119:6816/1273365545,v1:172.21.15.119:6817/1273365545] boot 2024-03-19T03:27:42.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:41 smithi117 ceph-mon[31838]: osdmap e51: 8 total, 8 up, 8 in 2024-03-19T03:27:42.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T03:27:42.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:42.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:42.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:27:42.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:27:42.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:27:42.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:42.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:27:42.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:27:42.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:27:42.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:41 smithi117 ceph-mon[31838]: osdmap e52: 8 total, 8 up, 8 in 2024-03-19T03:27:43.091 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:42 smithi178 ceph-mon[67578]: Detected new or changed devices on smithi119 2024-03-19T03:27:43.091 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:42 smithi178 ceph-mon[67578]: Adjusting osd_memory_target on smithi119 to 5693M 2024-03-19T03:27:43.091 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:42 smithi178 ceph-mon[67578]: pgmap v194: 1 pgs: 1 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:43.091 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:42 smithi178 ceph-mon[67578]: osdmap e53: 8 total, 8 up, 8 in 2024-03-19T03:27:43.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:42 smithi119 ceph-mon[66861]: Detected new or changed devices on smithi119 2024-03-19T03:27:43.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:42 smithi119 ceph-mon[66861]: Adjusting osd_memory_target on smithi119 to 5693M 2024-03-19T03:27:43.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:42 smithi119 ceph-mon[66861]: pgmap v194: 1 pgs: 1 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:43.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:42 smithi119 ceph-mon[66861]: osdmap e53: 8 total, 8 up, 8 in 2024-03-19T03:27:43.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:42 smithi117 ceph-mon[31838]: Detected new or changed devices on smithi119 2024-03-19T03:27:43.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:42 smithi117 ceph-mon[31838]: Adjusting osd_memory_target on smithi119 to 5693M 2024-03-19T03:27:43.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:42 smithi117 ceph-mon[31838]: pgmap v194: 1 pgs: 1 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:43.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:42 smithi117 ceph-mon[31838]: osdmap e53: 8 total, 8 up, 8 in 2024-03-19T03:27:43.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.508+0000 7fecc3751700 1 -- 172.21.15.178:0/222454508 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fecbc105700 msgr2=0x7fecbc107af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:27:43.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.508+0000 7fecc3751700 1 --2- 172.21.15.178:0/222454508 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fecbc105700 0x7fecbc107af0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fecac009990 tx=0x7fecac009ca0 comp rx=0 tx=0).stop 2024-03-19T03:27:43.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.508+0000 7fecc3751700 1 -- 172.21.15.178:0/222454508 shutdown_connections 2024-03-19T03:27:43.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.508+0000 7fecc3751700 1 --2- 172.21.15.178:0/222454508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fecbc108030 0x7fecbc10a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:27:43.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.508+0000 7fecc3751700 1 --2- 172.21.15.178:0/222454508 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fecbc102dd0 0x7fecbc1051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:27:43.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.508+0000 7fecc3751700 1 --2- 172.21.15.178:0/222454508 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fecbc105700 0x7fecbc107af0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:27:43.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.508+0000 7fecc3751700 1 -- 172.21.15.178:0/222454508 >> 172.21.15.178:0/222454508 conn(0x7fecbc0fc7e0 msgr2=0x7fecbc0fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:27:43.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.509+0000 7fecc3751700 1 -- 172.21.15.178:0/222454508 shutdown_connections 2024-03-19T03:27:43.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.509+0000 7fecc3751700 1 -- 172.21.15.178:0/222454508 wait complete. 2024-03-19T03:27:43.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.509+0000 7fecc3751700 1 Processor -- start 2024-03-19T03:27:43.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.509+0000 7fecc3751700 1 -- start start 2024-03-19T03:27:43.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.509+0000 7fecc3751700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fecbc102dd0 0x7fecbc198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:27:43.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.509+0000 7fecc3751700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fecbc105700 0x7fecbc198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:27:43.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.509+0000 7fecc3751700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fecbc108030 0x7fecbc19fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:27:43.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.509+0000 7fecc3751700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc10dfb0 con 0x7fecbc102dd0 2024-03-19T03:27:43.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.509+0000 7fecc3751700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc10de30 con 0x7fecbc105700 2024-03-19T03:27:43.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.509+0000 7fecc3751700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc10e130 con 0x7fecbc108030 2024-03-19T03:27:43.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.510+0000 7fecc1cee700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fecbc108030 0x7fecbc19fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:27:43.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.510+0000 7fecc1cee700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fecbc108030 0x7fecbc19fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:43180/0 (socket says 172.21.15.178:43180) 2024-03-19T03:27:43.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.510+0000 7fecc1cee700 1 -- 172.21.15.178:0/838870896 learned_addr learned my addr 172.21.15.178:0/838870896 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-19T03:27:43.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.510+0000 7fecc0cec700 1 --2- 172.21.15.178:0/838870896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fecbc105700 0x7fecbc198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:27:43.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.510+0000 7fecc1cee700 1 -- 172.21.15.178:0/838870896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fecbc105700 msgr2=0x7fecbc198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:27:43.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.510+0000 7fecc1cee700 1 --2- 172.21.15.178:0/838870896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fecbc105700 0x7fecbc198b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:27:43.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.510+0000 7fecc1cee700 1 -- 172.21.15.178:0/838870896 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fecbc102dd0 msgr2=0x7fecbc198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:27:43.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.510+0000 7fecc1cee700 1 --2- 172.21.15.178:0/838870896 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fecbc102dd0 0x7fecbc198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:27:43.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.510+0000 7fecc1cee700 1 -- 172.21.15.178:0/838870896 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecbc1a0270 con 0x7fecbc108030 2024-03-19T03:27:43.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.510+0000 7fecc1cee700 1 --2- 172.21.15.178:0/838870896 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fecbc108030 0x7fecbc19fbd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fecb400ba10 tx=0x7fecb400bd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:27:43.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.510+0000 7fecb27fc700 1 -- 172.21.15.178:0/838870896 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb4019070 con 0x7fecbc108030 2024-03-19T03:27:43.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.510+0000 7fecc3751700 1 -- 172.21.15.178:0/838870896 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecbc1a0560 con 0x7fecbc108030 2024-03-19T03:27:43.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.510+0000 7fecb27fc700 1 -- 172.21.15.178:0/838870896 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecb400ce10 con 0x7fecbc108030 2024-03-19T03:27:43.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.510+0000 7fecb27fc700 1 -- 172.21.15.178:0/838870896 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb4012340 con 0x7fecbc108030 2024-03-19T03:27:43.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.510+0000 7fecc3751700 1 -- 172.21.15.178:0/838870896 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecbc1a0b20 con 0x7fecbc108030 2024-03-19T03:27:43.515 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.511+0000 7fecc3751700 1 -- 172.21.15.178:0/838870896 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecbc068fd0 con 0x7fecbc108030 2024-03-19T03:27:43.515 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.512+0000 7fecb27fc700 1 -- 172.21.15.178:0/838870896 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fecb401c070 con 0x7fecbc108030 2024-03-19T03:27:43.515 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.513+0000 7fecb27fc700 1 --2- 172.21.15.178:0/838870896 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7feca80a2890 0x7feca80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:27:43.515 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.513+0000 7fecb27fc700 1 -- 172.21.15.178:0/838870896 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7fecb40cd260 con 0x7fecbc108030 2024-03-19T03:27:43.516 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.513+0000 7fecc14ed700 1 --2- 172.21.15.178:0/838870896 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7feca80a2890 0x7feca80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:27:43.516 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.514+0000 7fecc14ed700 1 --2- 172.21.15.178:0/838870896 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7feca80a2890 0x7feca80a4d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fecb8005950 tx=0x7fecb80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:27:43.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.517+0000 7fecb27fc700 1 -- 172.21.15.178:0/838870896 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fecb409ae40 con 0x7fecbc108030 2024-03-19T03:27:43.720 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:27:43.718+0000 7fecc3751700 1 -- 172.21.15.178:0/838870896 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fecbc062750 con 0x7feca80a2890 2024-03-19T03:27:44.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:43 smithi119 ceph-mon[66861]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-19T03:27:44.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:43 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:27:44.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:43 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:27:44.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:43 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:27:44.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:43 smithi119 ceph-mon[66861]: osdmap e54: 8 total, 8 up, 8 in 2024-03-19T03:27:44.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:43 smithi117 ceph-mon[31838]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-19T03:27:44.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:43 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:27:44.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:43 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:27:44.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:43 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:27:44.294 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:43 smithi117 ceph-mon[31838]: osdmap e54: 8 total, 8 up, 8 in 2024-03-19T03:27:44.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:43 smithi178 ceph-mon[67578]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-19T03:27:44.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:43 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:27:44.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:43 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:27:44.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:43 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:27:44.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:43 smithi178 ceph-mon[67578]: osdmap e54: 8 total, 8 up, 8 in 2024-03-19T03:27:45.291 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:44 smithi117 ceph-mon[31838]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:27:45.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:44 smithi117 ceph-mon[31838]: pgmap v197: 1 pgs: 1 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:45.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:44 smithi117 ceph-mon[31838]: osdmap e55: 8 total, 8 up, 8 in 2024-03-19T03:27:45.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:44 smithi178 ceph-mon[67578]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:27:45.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:44 smithi178 ceph-mon[67578]: pgmap v197: 1 pgs: 1 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:45.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:44 smithi178 ceph-mon[67578]: osdmap e55: 8 total, 8 up, 8 in 2024-03-19T03:27:45.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:44 smithi119 ceph-mon[66861]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:27:45.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:44 smithi119 ceph-mon[66861]: pgmap v197: 1 pgs: 1 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:45.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:44 smithi119 ceph-mon[66861]: osdmap e55: 8 total, 8 up, 8 in 2024-03-19T03:27:46.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:46 smithi178 ceph-mon[67578]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-19T03:27:46.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:46 smithi178 ceph-mon[67578]: Cluster is now healthy 2024-03-19T03:27:46.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:46 smithi119 ceph-mon[66861]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-19T03:27:46.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:46 smithi119 ceph-mon[66861]: Cluster is now healthy 2024-03-19T03:27:46.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:46 smithi117 ceph-mon[31838]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-19T03:27:46.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:46 smithi117 ceph-mon[31838]: Cluster is now healthy 2024-03-19T03:27:47.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:47 smithi178 ceph-mon[67578]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:47.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:47 smithi178 ceph-mon[67578]: from='client.? 172.21.15.178:0/1219823831' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd371461-a538-4781-92e1-5eead4ca3864"}]: dispatch 2024-03-19T03:27:47.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:47 smithi178 ceph-mon[67578]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd371461-a538-4781-92e1-5eead4ca3864"}]: dispatch 2024-03-19T03:27:47.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:47 smithi178 ceph-mon[67578]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd371461-a538-4781-92e1-5eead4ca3864"}]': finished 2024-03-19T03:27:47.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:47 smithi178 ceph-mon[67578]: osdmap e56: 9 total, 8 up, 9 in 2024-03-19T03:27:47.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:47 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T03:27:47.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:47 smithi178 ceph-mon[67578]: from='client.? 172.21.15.178:0/3379728481' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:27:47.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:47 smithi119 ceph-mon[66861]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:47.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:47 smithi119 ceph-mon[66861]: from='client.? 172.21.15.178:0/1219823831' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd371461-a538-4781-92e1-5eead4ca3864"}]: dispatch 2024-03-19T03:27:47.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:47 smithi119 ceph-mon[66861]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd371461-a538-4781-92e1-5eead4ca3864"}]: dispatch 2024-03-19T03:27:47.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:47 smithi119 ceph-mon[66861]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd371461-a538-4781-92e1-5eead4ca3864"}]': finished 2024-03-19T03:27:47.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:47 smithi119 ceph-mon[66861]: osdmap e56: 9 total, 8 up, 9 in 2024-03-19T03:27:47.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:47 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T03:27:47.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:47 smithi119 ceph-mon[66861]: from='client.? 172.21.15.178:0/3379728481' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:27:47.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:47 smithi117 ceph-mon[31838]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:47.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:47 smithi117 ceph-mon[31838]: from='client.? 172.21.15.178:0/1219823831' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd371461-a538-4781-92e1-5eead4ca3864"}]: dispatch 2024-03-19T03:27:47.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:47 smithi117 ceph-mon[31838]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd371461-a538-4781-92e1-5eead4ca3864"}]: dispatch 2024-03-19T03:27:47.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:47 smithi117 ceph-mon[31838]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd371461-a538-4781-92e1-5eead4ca3864"}]': finished 2024-03-19T03:27:47.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:47 smithi117 ceph-mon[31838]: osdmap e56: 9 total, 8 up, 9 in 2024-03-19T03:27:47.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:47 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T03:27:47.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:47 smithi117 ceph-mon[31838]: from='client.? 172.21.15.178:0/3379728481' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:27:49.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:49 smithi178 ceph-mon[67578]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:49.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:49 smithi119 ceph-mon[66861]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:49.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:49 smithi117 ceph-mon[31838]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:51.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:51 smithi178 ceph-mon[67578]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:51.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:51 smithi119 ceph-mon[66861]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:51.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:51 smithi117 ceph-mon[31838]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:53.634 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:53 smithi178 ceph-mon[67578]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:53.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:53 smithi119 ceph-mon[66861]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:53.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:53 smithi117 ceph-mon[31838]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:54.608 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:54 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-19T03:27:54.608 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:54 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:27:54.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:54 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-19T03:27:54.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:54 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:27:54.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:54 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-19T03:27:54.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:54 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:27:55.436 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:55 smithi178 ceph-mon[67578]: Deploying daemon osd.8 on smithi178 2024-03-19T03:27:55.436 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:55 smithi178 ceph-mon[67578]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:55.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:55 smithi119 ceph-mon[66861]: Deploying daemon osd.8 on smithi178 2024-03-19T03:27:55.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:55 smithi119 ceph-mon[66861]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:55.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:55 smithi117 ceph-mon[31838]: Deploying daemon osd.8 on smithi178 2024-03-19T03:27:55.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:55 smithi117 ceph-mon[31838]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:57.556 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:57 smithi178 ceph-mon[67578]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:57.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:57 smithi119 ceph-mon[66861]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:57.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:57 smithi117 ceph-mon[31838]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:59.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:27:59 smithi178 ceph-mon[67578]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:59.726 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:27:59 smithi119 ceph-mon[66861]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:27:59.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:27:59 smithi117 ceph-mon[31838]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:28:00.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:00 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:00.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:00 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:00.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:00 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:28:00.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:00 smithi178 ceph-mon[67578]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:28:00.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:00 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:00.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:00 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:00.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:00 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:28:00.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:00 smithi119 ceph-mon[66861]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:28:01.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:00 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:01.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:00 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:01.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:00 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:28:01.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:00 smithi117 ceph-mon[31838]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:28:03.103 INFO:teuthology.orchestra.run.smithi178.stdout:Created osd(s) 8 on host 'smithi178' 2024-03-19T03:28:03.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:28:03.102+0000 7fecb27fc700 1 -- 172.21.15.178:0/838870896 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fecbc062750 con 0x7feca80a2890 2024-03-19T03:28:03.105 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:28:03.104+0000 7fecc3751700 1 -- 172.21.15.178:0/838870896 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7feca80a2890 msgr2=0x7feca80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:28:03.106 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:28:03.104+0000 7fecc3751700 1 --2- 172.21.15.178:0/838870896 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7feca80a2890 0x7feca80a4d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fecb8005950 tx=0x7fecb80058e0 comp rx=0 tx=0).stop 2024-03-19T03:28:03.106 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:28:03.104+0000 7fecc3751700 1 -- 172.21.15.178:0/838870896 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fecbc108030 msgr2=0x7fecbc19fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:28:03.106 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:28:03.104+0000 7fecc3751700 1 --2- 172.21.15.178:0/838870896 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fecbc108030 0x7fecbc19fbd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fecb400ba10 tx=0x7fecb400bd20 comp rx=0 tx=0).stop 2024-03-19T03:28:03.106 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:28:03.104+0000 7fecc3751700 1 -- 172.21.15.178:0/838870896 shutdown_connections 2024-03-19T03:28:03.107 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:28:03.104+0000 7fecc3751700 1 --2- 172.21.15.178:0/838870896 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7feca80a2890 0x7feca80a4d50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:28:03.107 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:28:03.104+0000 7fecc3751700 1 --2- 172.21.15.178:0/838870896 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fecbc108030 0x7fecbc19fbd0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:28:03.107 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:28:03.104+0000 7fecc3751700 1 --2- 172.21.15.178:0/838870896 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fecbc102dd0 0x7fecbc198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:28:03.107 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:28:03.104+0000 7fecc3751700 1 --2- 172.21.15.178:0/838870896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fecbc105700 0x7fecbc198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:28:03.107 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:28:03.104+0000 7fecc3751700 1 -- 172.21.15.178:0/838870896 >> 172.21.15.178:0/838870896 conn(0x7fecbc0fc7e0 msgr2=0x7fecbc103710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:28:03.107 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:28:03.105+0000 7fecc3751700 1 -- 172.21.15.178:0/838870896 shutdown_connections 2024-03-19T03:28:03.108 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:28:03.105+0000 7fecc3751700 1 -- 172.21.15.178:0/838870896 wait complete. 2024-03-19T03:28:03.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:03 smithi178 ceph-mon[67578]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:28:03.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:03 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:03.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:03 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:03.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:03 smithi117 ceph-mon[31838]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:28:03.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:03 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:03.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:03 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:03.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:03 smithi119 ceph-mon[66861]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:28:03.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:03 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:03.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:03 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:03.741 DEBUG:teuthology.orchestra.run.smithi178:osd.8> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.8.service 2024-03-19T03:28:03.748 INFO:tasks.cephadm:Deploying osd.9 on smithi117 with /dev/vg_nvme/lv_1... 2024-03-19T03:28:03.748 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-19T03:28:04.198 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:28:04.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:04 smithi117 ceph-mon[31838]: from='osd.8 [v2:172.21.15.178:6816/1059587501,v1:172.21.15.178:6817/1059587501]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-19T03:28:04.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:04 smithi117 ceph-mon[31838]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-19T03:28:04.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:04 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:04.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:04 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:04.543 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:04 smithi178 ceph-mon[67578]: from='osd.8 [v2:172.21.15.178:6816/1059587501,v1:172.21.15.178:6817/1059587501]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-19T03:28:04.543 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:04 smithi178 ceph-mon[67578]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-19T03:28:04.543 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:04 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:04.543 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:04 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:04.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:04 smithi119 ceph-mon[66861]: from='osd.8 [v2:172.21.15.178:6816/1059587501,v1:172.21.15.178:6817/1059587501]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-19T03:28:04.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:04 smithi119 ceph-mon[66861]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-19T03:28:04.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:04 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:04.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:04 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:05.571 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:05 smithi178 ceph-mon[67578]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:28:05.571 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:05 smithi178 ceph-mon[67578]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-19T03:28:05.571 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:05 smithi178 ceph-mon[67578]: from='osd.8 [v2:172.21.15.178:6816/1059587501,v1:172.21.15.178:6817/1059587501]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-19T03:28:05.571 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:05 smithi178 ceph-mon[67578]: osdmap e57: 9 total, 8 up, 9 in 2024-03-19T03:28:05.571 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:05 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T03:28:05.572 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:05 smithi178 ceph-mon[67578]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-19T03:28:05.572 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:28:05 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:28:05.306+0000 7fcbc80c6700 -1 osd.8 0 waiting for initial osdmap 2024-03-19T03:28:05.572 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:28:05 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:28:05.318+0000 7fcbc3eb9700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T03:28:05.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:05 smithi119 ceph-mon[66861]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:28:05.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:05 smithi119 ceph-mon[66861]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-19T03:28:05.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:05 smithi119 ceph-mon[66861]: from='osd.8 [v2:172.21.15.178:6816/1059587501,v1:172.21.15.178:6817/1059587501]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-19T03:28:05.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:05 smithi119 ceph-mon[66861]: osdmap e57: 9 total, 8 up, 9 in 2024-03-19T03:28:05.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:05 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T03:28:05.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:05 smithi119 ceph-mon[66861]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-19T03:28:05.748 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:05 smithi117 ceph-mon[31838]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:28:05.748 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:05 smithi117 ceph-mon[31838]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-19T03:28:05.748 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:05 smithi117 ceph-mon[31838]: from='osd.8 [v2:172.21.15.178:6816/1059587501,v1:172.21.15.178:6817/1059587501]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-19T03:28:05.748 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:05 smithi117 ceph-mon[31838]: osdmap e57: 9 total, 8 up, 9 in 2024-03-19T03:28:05.748 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:05 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T03:28:05.748 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:05 smithi117 ceph-mon[31838]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-19T03:28:06.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:06 smithi178 ceph-mon[67578]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-03-19T03:28:06.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:06 smithi178 ceph-mon[67578]: osdmap e58: 9 total, 8 up, 9 in 2024-03-19T03:28:06.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:06 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T03:28:06.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:06 smithi178 ceph-mon[67578]: osd.8 [v2:172.21.15.178:6816/1059587501,v1:172.21.15.178:6817/1059587501] boot 2024-03-19T03:28:06.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:06 smithi178 ceph-mon[67578]: osdmap e59: 9 total, 9 up, 9 in 2024-03-19T03:28:06.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:06 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T03:28:06.626 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:06 smithi117 ceph-mon[31838]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-03-19T03:28:06.627 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:06 smithi117 ceph-mon[31838]: osdmap e58: 9 total, 8 up, 9 in 2024-03-19T03:28:06.627 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:06 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T03:28:06.627 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:06 smithi117 ceph-mon[31838]: osd.8 [v2:172.21.15.178:6816/1059587501,v1:172.21.15.178:6817/1059587501] boot 2024-03-19T03:28:06.627 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:06 smithi117 ceph-mon[31838]: osdmap e59: 9 total, 9 up, 9 in 2024-03-19T03:28:06.627 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:06 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T03:28:06.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:06 smithi119 ceph-mon[66861]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-03-19T03:28:06.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:06 smithi119 ceph-mon[66861]: osdmap e58: 9 total, 8 up, 9 in 2024-03-19T03:28:06.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:06 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T03:28:06.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:06 smithi119 ceph-mon[66861]: osd.8 [v2:172.21.15.178:6816/1059587501,v1:172.21.15.178:6817/1059587501] boot 2024-03-19T03:28:06.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:06 smithi119 ceph-mon[66861]: osdmap e59: 9 total, 9 up, 9 in 2024-03-19T03:28:06.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:06 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T03:28:07.229 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-19T03:28:07.229 INFO:teuthology.orchestra.run.smithi117.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-19T03:28:07.229 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10+0 records in 2024-03-19T03:28:07.229 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2024-03-19T03:28:07.229 INFO:teuthology.orchestra.run.smithi117.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0338366 s, 310 MB/s 2024-03-19T03:28:07.229 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping successful for: 2024-03-19T03:28:07.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:07 smithi117 ceph-mon[31838]: purged_snaps scrub starts 2024-03-19T03:28:07.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:07 smithi117 ceph-mon[31838]: purged_snaps scrub ok 2024-03-19T03:28:07.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:07 smithi117 ceph-mon[31838]: pgmap v212: 1 pgs: 1 peering; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:28:07.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:07 smithi117 ceph-mon[31838]: osdmap e60: 9 total, 9 up, 9 in 2024-03-19T03:28:07.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:07 smithi178 ceph-mon[67578]: purged_snaps scrub starts 2024-03-19T03:28:07.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:07 smithi178 ceph-mon[67578]: purged_snaps scrub ok 2024-03-19T03:28:07.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:07 smithi178 ceph-mon[67578]: pgmap v212: 1 pgs: 1 peering; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:28:07.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:07 smithi178 ceph-mon[67578]: osdmap e60: 9 total, 9 up, 9 in 2024-03-19T03:28:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:07 smithi119 ceph-mon[66861]: purged_snaps scrub starts 2024-03-19T03:28:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:07 smithi119 ceph-mon[66861]: purged_snaps scrub ok 2024-03-19T03:28:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:07 smithi119 ceph-mon[66861]: pgmap v212: 1 pgs: 1 peering; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-19T03:28:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:07 smithi119 ceph-mon[66861]: osdmap e60: 9 total, 9 up, 9 in 2024-03-19T03:28:08.298 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi117:vg_nvme/lv_1 2024-03-19T03:28:08.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:08 smithi117 ceph-mon[31838]: Detected new or changed devices on smithi178 2024-03-19T03:28:08.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:08.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:08.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:28:08.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:28:08.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:28:08.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:08 smithi117 ceph-mon[31838]: Adjusting osd_memory_target on smithi178 to 5693M 2024-03-19T03:28:08.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:08.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:28:08.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:28:08.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:08.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:28:08.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:28:08.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:28:08.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:08.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:08 smithi117 ceph-mon[31838]: pgmap v215: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:08.544 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:08 smithi117 ceph-mon[31838]: osdmap e61: 9 total, 9 up, 9 in 2024-03-19T03:28:08.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:08 smithi119 ceph-mon[66861]: Detected new or changed devices on smithi178 2024-03-19T03:28:08.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:08.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:08.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:28:08.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:28:08.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:28:08.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:08 smithi119 ceph-mon[66861]: Adjusting osd_memory_target on smithi178 to 5693M 2024-03-19T03:28:08.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:08.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:28:08.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:28:08.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:08.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:28:08.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:28:08.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:28:08.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:08.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:08 smithi119 ceph-mon[66861]: pgmap v215: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:08.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:08 smithi119 ceph-mon[66861]: osdmap e61: 9 total, 9 up, 9 in 2024-03-19T03:28:08.736 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:28:08.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:08 smithi178 ceph-mon[67578]: Detected new or changed devices on smithi178 2024-03-19T03:28:08.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:08.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:08.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:28:08.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:28:08.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:28:08.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:08 smithi178 ceph-mon[67578]: Adjusting osd_memory_target on smithi178 to 5693M 2024-03-19T03:28:08.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:08.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:28:08.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:28:08.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:08.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:28:08.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:28:08.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:28:08.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:08.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:08 smithi178 ceph-mon[67578]: pgmap v215: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:08.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:08 smithi178 ceph-mon[67578]: osdmap e61: 9 total, 9 up, 9 in 2024-03-19T03:28:10.215 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:09 smithi117 ceph-mon[31838]: osdmap e62: 9 total, 9 up, 9 in 2024-03-19T03:28:10.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:09 smithi119 ceph-mon[66861]: osdmap e62: 9 total, 9 up, 9 in 2024-03-19T03:28:10.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:09 smithi178 ceph-mon[67578]: osdmap e62: 9 total, 9 up, 9 in 2024-03-19T03:28:11.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.028+0000 7fadc6800700 1 -- 172.21.15.117:0/2596306089 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadc0108990 msgr2=0x7fadc0108db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:28:11.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.028+0000 7fadc6800700 1 --2- 172.21.15.117:0/2596306089 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadc0108990 0x7fadc0108db0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fadb00099c0 tx=0x7fadb0009cd0 comp rx=0 tx=0).stop 2024-03-19T03:28:11.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.028+0000 7fadc6800700 1 -- 172.21.15.117:0/2596306089 shutdown_connections 2024-03-19T03:28:11.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.028+0000 7fadc6800700 1 --2- 172.21.15.117:0/2596306089 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fadc010a530 0x7fadc0110de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:28:11.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.028+0000 7fadc6800700 1 --2- 172.21.15.117:0/2596306089 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadc0108990 0x7fadc0108db0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:28:11.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.028+0000 7fadc6800700 1 --2- 172.21.15.117:0/2596306089 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadc0109b90 0x7fadc0109ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:28:11.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.028+0000 7fadc6800700 1 -- 172.21.15.117:0/2596306089 >> 172.21.15.117:0/2596306089 conn(0x7fadc0077960 msgr2=0x7fadc0079da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:28:11.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.028+0000 7fadc6800700 1 -- 172.21.15.117:0/2596306089 shutdown_connections 2024-03-19T03:28:11.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.028+0000 7fadc6800700 1 -- 172.21.15.117:0/2596306089 wait complete. 2024-03-19T03:28:11.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.029+0000 7fadc6800700 1 Processor -- start 2024-03-19T03:28:11.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.029+0000 7fadc6800700 1 -- start start 2024-03-19T03:28:11.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.030+0000 7fadc6800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadc0108990 0x7fadc019eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:28:11.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.030+0000 7fadc6800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadc0109b90 0x7fadc019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:28:11.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.030+0000 7fadc6800700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fadc010a530 0x7fadc01a6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:28:11.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.030+0000 7fadc6800700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fadc0114570 con 0x7fadc0109b90 2024-03-19T03:28:11.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.030+0000 7fadc6800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fadc01143f0 con 0x7fadc0108990 2024-03-19T03:28:11.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.030+0000 7fadc6800700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fadc01146f0 con 0x7fadc010a530 2024-03-19T03:28:11.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.030+0000 7fadbf400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadc0108990 0x7fadc019eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:28:11.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.030+0000 7fadbea00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadc0109b90 0x7fadc019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:28:11.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.030+0000 7fadbf400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadc0108990 0x7fadc019eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:60962/0 (socket says 172.21.15.117:60962) 2024-03-19T03:28:11.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.030+0000 7fadbf400700 1 -- 172.21.15.117:0/132912697 learned_addr learned my addr 172.21.15.117:0/132912697 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:28:11.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.030+0000 7fadbfe00700 1 --2- 172.21.15.117:0/132912697 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fadc010a530 0x7fadc01a6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:28:11.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.031+0000 7fadbf400700 1 -- 172.21.15.117:0/132912697 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fadc010a530 msgr2=0x7fadc01a6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:28:11.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.031+0000 7fadbf400700 1 --2- 172.21.15.117:0/132912697 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fadc010a530 0x7fadc01a6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:28:11.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.031+0000 7fadbf400700 1 -- 172.21.15.117:0/132912697 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadc0109b90 msgr2=0x7fadc019f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:28:11.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.031+0000 7fadbf400700 1 --2- 172.21.15.117:0/132912697 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadc0109b90 0x7fadc019f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:28:11.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.031+0000 7fadbf400700 1 -- 172.21.15.117:0/132912697 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc01a67b0 con 0x7fadc0108990 2024-03-19T03:28:11.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.031+0000 7fadbfe00700 1 --2- 172.21.15.117:0/132912697 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fadc010a530 0x7fadc01a6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:28:11.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.031+0000 7fadbea00700 1 --2- 172.21.15.117:0/132912697 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadc0109b90 0x7fadc019f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:28:11.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.031+0000 7fadbf400700 1 --2- 172.21.15.117:0/132912697 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadc0108990 0x7fadc019eb50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fadb000b580 tx=0x7fadb000f710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:28:11.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.031+0000 7fadb7e00700 1 -- 172.21.15.117:0/132912697 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb0027070 con 0x7fadc0108990 2024-03-19T03:28:11.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.031+0000 7fadb7e00700 1 -- 172.21.15.117:0/132912697 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadb000fe20 con 0x7fadc0108990 2024-03-19T03:28:11.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.031+0000 7fadb7e00700 1 -- 172.21.15.117:0/132912697 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb00173f0 con 0x7fadc0108990 2024-03-19T03:28:11.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.031+0000 7fadc6800700 1 -- 172.21.15.117:0/132912697 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadc01a6a40 con 0x7fadc0108990 2024-03-19T03:28:11.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.032+0000 7fadc6800700 1 -- 172.21.15.117:0/132912697 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadc01a6ed0 con 0x7fadc0108990 2024-03-19T03:28:11.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.033+0000 7fadc6800700 1 -- 172.21.15.117:0/132912697 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadc0068fd0 con 0x7fadc0108990 2024-03-19T03:28:11.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.034+0000 7fadb7e00700 1 -- 172.21.15.117:0/132912697 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fadb002a070 con 0x7fadc0108990 2024-03-19T03:28:11.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.036+0000 7fadb7e00700 1 --2- 172.21.15.117:0/132912697 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fada40a2910 0x7fada40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:28:11.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.036+0000 7fadbea00700 1 --2- 172.21.15.117:0/132912697 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fada40a2910 0x7fada40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:28:11.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.036+0000 7fadb7e00700 1 -- 172.21.15.117:0/132912697 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7fadb00ced10 con 0x7fadc0108990 2024-03-19T03:28:11.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.037+0000 7fadbea00700 1 --2- 172.21.15.117:0/132912697 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fada40a2910 0x7fada40a4dd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fadac011490 tx=0x7fadac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:28:11.039 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:10 smithi117 ceph-mon[31838]: osdmap e63: 9 total, 9 up, 9 in 2024-03-19T03:28:11.039 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:10 smithi117 ceph-mon[31838]: pgmap v219: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:11.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.040+0000 7fadb7e00700 1 -- 172.21.15.117:0/132912697 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fadb005e740 con 0x7fadc0108990 2024-03-19T03:28:11.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:10 smithi119 ceph-mon[66861]: osdmap e63: 9 total, 9 up, 9 in 2024-03-19T03:28:11.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:10 smithi119 ceph-mon[66861]: pgmap v219: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:11.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:11.274+0000 7fadc6800700 1 -- 172.21.15.117:0/132912697 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fadc0062750 con 0x7fada40a2910 2024-03-19T03:28:11.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:10 smithi178 ceph-mon[67578]: osdmap e63: 9 total, 9 up, 9 in 2024-03-19T03:28:11.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:10 smithi178 ceph-mon[67578]: pgmap v219: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:12.227 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:11 smithi117 ceph-mon[31838]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:28:12.227 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:11 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:28:12.228 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:11 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:28:12.228 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:11 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:28:12.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:11 smithi119 ceph-mon[66861]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:28:12.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:11 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:28:12.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:11 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:28:12.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:11 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:28:12.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:11 smithi178 ceph-mon[67578]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:28:12.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:11 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:28:12.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:11 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:28:12.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:11 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:28:13.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:12 smithi119 ceph-mon[66861]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-03-19T03:28:13.249 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:12 smithi117 ceph-mon[31838]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-03-19T03:28:13.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:12 smithi178 ceph-mon[67578]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-03-19T03:28:15.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:15 smithi117 ceph-mon[31838]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T03:28:15.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:15 smithi178 ceph-mon[67578]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T03:28:15.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:15 smithi119 ceph-mon[66861]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T03:28:16.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:16 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1638646026' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8"}]: dispatch 2024-03-19T03:28:16.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:16 smithi117 ceph-mon[31838]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8"}]: dispatch 2024-03-19T03:28:16.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:16 smithi117 ceph-mon[31838]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8"}]': finished 2024-03-19T03:28:16.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:16 smithi117 ceph-mon[31838]: osdmap e64: 10 total, 9 up, 10 in 2024-03-19T03:28:16.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:16 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T03:28:16.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:16 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3216100106' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:28:16.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:16 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1638646026' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8"}]: dispatch 2024-03-19T03:28:16.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:16 smithi178 ceph-mon[67578]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8"}]: dispatch 2024-03-19T03:28:16.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:16 smithi178 ceph-mon[67578]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8"}]': finished 2024-03-19T03:28:16.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:16 smithi178 ceph-mon[67578]: osdmap e64: 10 total, 9 up, 10 in 2024-03-19T03:28:16.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:16 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T03:28:16.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:16 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3216100106' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:28:16.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:16 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1638646026' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8"}]: dispatch 2024-03-19T03:28:16.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:16 smithi119 ceph-mon[66861]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8"}]: dispatch 2024-03-19T03:28:16.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:16 smithi119 ceph-mon[66861]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8"}]': finished 2024-03-19T03:28:16.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:16 smithi119 ceph-mon[66861]: osdmap e64: 10 total, 9 up, 10 in 2024-03-19T03:28:16.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:16 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T03:28:16.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:16 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3216100106' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:28:17.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:17 smithi178 ceph-mon[67578]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-19T03:28:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:17 smithi119 ceph-mon[66861]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-19T03:28:17.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:17 smithi117 ceph-mon[31838]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-19T03:28:19.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:19 smithi178 ceph-mon[67578]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-19T03:28:19.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:19 smithi119 ceph-mon[66861]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-19T03:28:19.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:19 smithi117 ceph-mon[31838]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-19T03:28:21.573 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:21 smithi117 ceph-mon[31838]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T03:28:21.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:21 smithi178 ceph-mon[67578]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T03:28:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:21 smithi119 ceph-mon[66861]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T03:28:23.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:23 smithi178 ceph-mon[67578]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:23.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:23 smithi119 ceph-mon[66861]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:23.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:23 smithi117 ceph-mon[31838]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:25.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:25 smithi178 ceph-mon[67578]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:25.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:25 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-19T03:28:25.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:25 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:28:25.716 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:25 smithi117 ceph-mon[31838]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:25.716 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:25 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-19T03:28:25.716 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:25 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:28:25.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:25 smithi119 ceph-mon[66861]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:25.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:25 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-19T03:28:25.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:25 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:28:26.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:26 smithi117 ceph-mon[31838]: Deploying daemon osd.9 on smithi117 2024-03-19T03:28:26.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:26 smithi178 ceph-mon[67578]: Deploying daemon osd.9 on smithi117 2024-03-19T03:28:26.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:26 smithi119 ceph-mon[66861]: Deploying daemon osd.9 on smithi117 2024-03-19T03:28:27.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:27 smithi178 ceph-mon[67578]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:27.604 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:27 smithi117 ceph-mon[31838]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:27.635 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:27 smithi119 ceph-mon[66861]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:29.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:29 smithi178 ceph-mon[67578]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:29.724 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:29 smithi117 ceph-mon[31838]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:29.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:29 smithi119 ceph-mon[66861]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:31.613 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:31 smithi117 ceph-mon[31838]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:31.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:31 smithi119 ceph-mon[66861]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:31.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:31 smithi178 ceph-mon[67578]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:33.641 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:33 smithi117 ceph-mon[31838]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:33.641 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:33.641 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:28:33.641 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:33.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:33 smithi119 ceph-mon[66861]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:33.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:33.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:28:33.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:33.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:33 smithi178 ceph-mon[67578]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:33.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:28:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:35.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:35 smithi117 ceph-mon[31838]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:35.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:35 smithi119 ceph-mon[66861]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:35.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:35 smithi178 ceph-mon[67578]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:37.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:37 smithi119 ceph-mon[66861]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:37.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:37 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:37.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:37 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:37.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:37 smithi119 ceph-mon[66861]: from='osd.9 [v2:172.21.15.117:6826/3653875600,v1:172.21.15.117:6827/3653875600]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-19T03:28:37.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:37 smithi119 ceph-mon[66861]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-19T03:28:37.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:37 smithi117 ceph-mon[31838]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:37.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:37.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:37.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:37 smithi117 ceph-mon[31838]: from='osd.9 [v2:172.21.15.117:6826/3653875600,v1:172.21.15.117:6827/3653875600]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-19T03:28:37.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:37 smithi117 ceph-mon[31838]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-19T03:28:37.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:37 smithi178 ceph-mon[67578]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:37.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:37.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:37.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:37 smithi178 ceph-mon[67578]: from='osd.9 [v2:172.21.15.117:6826/3653875600,v1:172.21.15.117:6827/3653875600]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-19T03:28:37.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:37 smithi178 ceph-mon[67578]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-19T03:28:38.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:38.329+0000 7fadb7e00700 1 -- 172.21.15.117:0/132912697 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fadc0062750 con 0x7fada40a2910 2024-03-19T03:28:38.331 INFO:teuthology.orchestra.run.smithi117.stdout:Created osd(s) 9 on host 'smithi117' 2024-03-19T03:28:38.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:38.332+0000 7fadc6800700 1 -- 172.21.15.117:0/132912697 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fada40a2910 msgr2=0x7fada40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:28:38.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:38.332+0000 7fadc6800700 1 --2- 172.21.15.117:0/132912697 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fada40a2910 0x7fada40a4dd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fadac011490 tx=0x7fadac009260 comp rx=0 tx=0).stop 2024-03-19T03:28:38.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:38.332+0000 7fadc6800700 1 -- 172.21.15.117:0/132912697 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadc0108990 msgr2=0x7fadc019eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:28:38.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:38.332+0000 7fadc6800700 1 --2- 172.21.15.117:0/132912697 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadc0108990 0x7fadc019eb50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fadb000b580 tx=0x7fadb000f710 comp rx=0 tx=0).stop 2024-03-19T03:28:38.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:38.332+0000 7fadc6800700 1 -- 172.21.15.117:0/132912697 shutdown_connections 2024-03-19T03:28:38.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:38.332+0000 7fadc6800700 1 --2- 172.21.15.117:0/132912697 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fada40a2910 0x7fada40a4dd0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:28:38.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:38.332+0000 7fadc6800700 1 --2- 172.21.15.117:0/132912697 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fadc010a530 0x7fadc01a6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:28:38.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:38.332+0000 7fadc6800700 1 --2- 172.21.15.117:0/132912697 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fadc0108990 0x7fadc019eb50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:28:38.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:38.332+0000 7fadc6800700 1 --2- 172.21.15.117:0/132912697 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadc0109b90 0x7fadc019f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:28:38.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:38.332+0000 7fadc6800700 1 -- 172.21.15.117:0/132912697 >> 172.21.15.117:0/132912697 conn(0x7fadc0077960 msgr2=0x7fadc010ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:28:38.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:38.332+0000 7fadc6800700 1 -- 172.21.15.117:0/132912697 shutdown_connections 2024-03-19T03:28:38.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:28:38.332+0000 7fadc6800700 1 -- 172.21.15.117:0/132912697 wait complete. 2024-03-19T03:28:39.162 DEBUG:teuthology.orchestra.run.smithi117:osd.9> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.9.service 2024-03-19T03:28:39.166 INFO:tasks.cephadm:Deploying osd.10 on smithi119 with /dev/vg_nvme/lv_1... 2024-03-19T03:28:39.167 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-19T03:28:39.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:39 smithi178 ceph-mon[67578]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-19T03:28:39.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:39 smithi178 ceph-mon[67578]: osdmap e65: 10 total, 9 up, 10 in 2024-03-19T03:28:39.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T03:28:39.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:39 smithi178 ceph-mon[67578]: from='osd.9 [v2:172.21.15.117:6826/3653875600,v1:172.21.15.117:6827/3653875600]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-19T03:28:39.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:39 smithi178 ceph-mon[67578]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-19T03:28:39.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:28:39.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:28:39.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:39.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:39 smithi178 ceph-mon[67578]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:39.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:39.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:39.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:28:39.413 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T03:28:39.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:39 smithi119 ceph-mon[66861]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-19T03:28:39.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:39 smithi119 ceph-mon[66861]: osdmap e65: 10 total, 9 up, 10 in 2024-03-19T03:28:39.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T03:28:39.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:39 smithi119 ceph-mon[66861]: from='osd.9 [v2:172.21.15.117:6826/3653875600,v1:172.21.15.117:6827/3653875600]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-19T03:28:39.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:39 smithi119 ceph-mon[66861]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-19T03:28:39.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:28:39.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:28:39.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:39.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:39 smithi119 ceph-mon[66861]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:39.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:39.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:39.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:28:39.490 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:39 smithi117 ceph-mon[31838]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-19T03:28:39.490 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:39 smithi117 ceph-mon[31838]: osdmap e65: 10 total, 9 up, 10 in 2024-03-19T03:28:39.490 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T03:28:39.490 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:39 smithi117 ceph-mon[31838]: from='osd.9 [v2:172.21.15.117:6826/3653875600,v1:172.21.15.117:6827/3653875600]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-19T03:28:39.490 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:39 smithi117 ceph-mon[31838]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-19T03:28:39.490 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:28:39.491 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:28:39.491 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:39.491 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:39 smithi117 ceph-mon[31838]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:39.491 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:39.491 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:39.491 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:28:39.793 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:28:39 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:28:39.677+0000 7fdfdee00700 -1 osd.9 0 waiting for initial osdmap 2024-03-19T03:28:39.793 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:28:39 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:28:39.708+0000 7fdfd4c00700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T03:28:40.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:40 smithi119 ceph-mon[66861]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-19T03:28:40.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:40 smithi119 ceph-mon[66861]: osdmap e66: 10 total, 9 up, 10 in 2024-03-19T03:28:40.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:40 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T03:28:40.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:40 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T03:28:40.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:40 smithi119 ceph-mon[66861]: from='osd.9 ' entity='osd.9' 2024-03-19T03:28:40.330 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:40 smithi117 ceph-mon[31838]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-19T03:28:40.330 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:40 smithi117 ceph-mon[31838]: osdmap e66: 10 total, 9 up, 10 in 2024-03-19T03:28:40.330 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:40 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T03:28:40.330 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:40 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T03:28:40.330 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:40 smithi117 ceph-mon[31838]: from='osd.9 ' entity='osd.9' 2024-03-19T03:28:40.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:40 smithi178 ceph-mon[67578]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-19T03:28:40.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:40 smithi178 ceph-mon[67578]: osdmap e66: 10 total, 9 up, 10 in 2024-03-19T03:28:40.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:40 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T03:28:40.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:40 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T03:28:40.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:40 smithi178 ceph-mon[67578]: from='osd.9 ' entity='osd.9' 2024-03-19T03:28:41.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:41 smithi119 ceph-mon[66861]: purged_snaps scrub starts 2024-03-19T03:28:41.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:41 smithi119 ceph-mon[66861]: purged_snaps scrub ok 2024-03-19T03:28:41.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T03:28:41.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:41 smithi119 ceph-mon[66861]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:41.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:41 smithi119 ceph-mon[66861]: osd.9 [v2:172.21.15.117:6826/3653875600,v1:172.21.15.117:6827/3653875600] boot 2024-03-19T03:28:41.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:41 smithi119 ceph-mon[66861]: osdmap e67: 10 total, 10 up, 10 in 2024-03-19T03:28:41.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T03:28:41.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:41 smithi117 ceph-mon[31838]: purged_snaps scrub starts 2024-03-19T03:28:41.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:41 smithi117 ceph-mon[31838]: purged_snaps scrub ok 2024-03-19T03:28:41.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T03:28:41.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:41 smithi117 ceph-mon[31838]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:41.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:41 smithi117 ceph-mon[31838]: osd.9 [v2:172.21.15.117:6826/3653875600,v1:172.21.15.117:6827/3653875600] boot 2024-03-19T03:28:41.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:41 smithi117 ceph-mon[31838]: osdmap e67: 10 total, 10 up, 10 in 2024-03-19T03:28:41.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T03:28:41.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:41 smithi178 ceph-mon[67578]: purged_snaps scrub starts 2024-03-19T03:28:41.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:41 smithi178 ceph-mon[67578]: purged_snaps scrub ok 2024-03-19T03:28:41.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T03:28:41.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:41 smithi178 ceph-mon[67578]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-19T03:28:41.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:41 smithi178 ceph-mon[67578]: osd.9 [v2:172.21.15.117:6826/3653875600,v1:172.21.15.117:6827/3653875600] boot 2024-03-19T03:28:41.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:41 smithi178 ceph-mon[67578]: osdmap e67: 10 total, 10 up, 10 in 2024-03-19T03:28:41.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T03:28:41.523 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-19T03:28:41.523 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-19T03:28:41.523 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-03-19T03:28:41.523 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-03-19T03:28:41.523 INFO:teuthology.orchestra.run.smithi119.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0120237 s, 872 MB/s 2024-03-19T03:28:41.524 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 2024-03-19T03:28:41.524 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-03-19T03:28:42.182 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi119:vg_nvme/lv_1 2024-03-19T03:28:42.425 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T03:28:43.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:43 smithi117 ceph-mon[31838]: osdmap e68: 10 total, 10 up, 10 in 2024-03-19T03:28:43.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:43 smithi117 ceph-mon[31838]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:43.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:43 smithi178 ceph-mon[67578]: osdmap e68: 10 total, 10 up, 10 in 2024-03-19T03:28:43.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:43 smithi178 ceph-mon[67578]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:43.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:43 smithi119 ceph-mon[66861]: osdmap e68: 10 total, 10 up, 10 in 2024-03-19T03:28:43.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:43 smithi119 ceph-mon[66861]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:44.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.054+0000 7fcbffe09700 1 -- 172.21.15.119:0/340989826 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbf8100620 msgr2=0x7fcbf8100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:28:44.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.054+0000 7fcbffe09700 1 --2- 172.21.15.119:0/340989826 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbf8100620 0x7fcbf8100a40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4009990 tx=0x7fcbf4009ca0 comp rx=0 tx=0).stop 2024-03-19T03:28:44.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.054+0000 7fcbffe09700 1 -- 172.21.15.119:0/340989826 shutdown_connections 2024-03-19T03:28:44.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.054+0000 7fcbffe09700 1 --2- 172.21.15.119:0/340989826 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbf8103fa0 0x7fcbf810a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:28:44.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.054+0000 7fcbffe09700 1 --2- 172.21.15.119:0/340989826 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbf8100620 0x7fcbf8100a40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:28:44.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.054+0000 7fcbffe09700 1 --2- 172.21.15.119:0/340989826 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcbf8103600 0x7fcbf8103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:28:44.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.054+0000 7fcbffe09700 1 -- 172.21.15.119:0/340989826 >> 172.21.15.119:0/340989826 conn(0x7fcbf80fbfa0 msgr2=0x7fcbf80fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:28:44.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.054+0000 7fcbffe09700 1 -- 172.21.15.119:0/340989826 shutdown_connections 2024-03-19T03:28:44.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.054+0000 7fcbffe09700 1 -- 172.21.15.119:0/340989826 wait complete. 2024-03-19T03:28:44.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.054+0000 7fcbffe09700 1 Processor -- start 2024-03-19T03:28:44.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.054+0000 7fcbffe09700 1 -- start start 2024-03-19T03:28:44.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.055+0000 7fcbffe09700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbf8100620 0x7fcbf81986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:28:44.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.055+0000 7fcbffe09700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbf8103600 0x7fcbf8198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:28:44.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.055+0000 7fcbffe09700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcbf8103fa0 0x7fcbf819fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:28:44.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.055+0000 7fcbffe09700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf810e010 con 0x7fcbf8100620 2024-03-19T03:28:44.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.055+0000 7fcbffe09700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf810de90 con 0x7fcbf8103600 2024-03-19T03:28:44.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.055+0000 7fcbffe09700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf810e190 con 0x7fcbf8103fa0 2024-03-19T03:28:44.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.055+0000 7fcbfd3a4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbf8103600 0x7fcbf8198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:28:44.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.055+0000 7fcbfd3a4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbf8103600 0x7fcbf8198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47028/0 (socket says 172.21.15.119:47028) 2024-03-19T03:28:44.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.055+0000 7fcbfd3a4700 1 -- 172.21.15.119:0/2697247835 learned_addr learned my addr 172.21.15.119:0/2697247835 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-03-19T03:28:44.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.055+0000 7fcbfd3a4700 1 -- 172.21.15.119:0/2697247835 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcbf8103fa0 msgr2=0x7fcbf819fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:28:44.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.055+0000 7fcbfd3a4700 1 --2- 172.21.15.119:0/2697247835 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcbf8103fa0 0x7fcbf819fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:28:44.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.055+0000 7fcbfd3a4700 1 -- 172.21.15.119:0/2697247835 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbf8100620 msgr2=0x7fcbf81986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:28:44.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.055+0000 7fcbfd3a4700 1 --2- 172.21.15.119:0/2697247835 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbf8100620 0x7fcbf81986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:28:44.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.055+0000 7fcbfd3a4700 1 -- 172.21.15.119:0/2697247835 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbf81a0330 con 0x7fcbf8103600 2024-03-19T03:28:44.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.055+0000 7fcbfd3a4700 1 --2- 172.21.15.119:0/2697247835 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbf8103600 0x7fcbf8198c10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcbe800b530 tx=0x7fcbe800b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:28:44.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.055+0000 7fcbeeffd700 1 -- 172.21.15.119:0/2697247835 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe8015020 con 0x7fcbf8103600 2024-03-19T03:28:44.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.055+0000 7fcbeeffd700 1 -- 172.21.15.119:0/2697247835 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbe8011d10 con 0x7fcbf8103600 2024-03-19T03:28:44.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.055+0000 7fcbffe09700 1 -- 172.21.15.119:0/2697247835 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbf81a0620 con 0x7fcbf8103600 2024-03-19T03:28:44.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.055+0000 7fcbeeffd700 1 -- 172.21.15.119:0/2697247835 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe800f2e0 con 0x7fcbf8103600 2024-03-19T03:28:44.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.055+0000 7fcbffe09700 1 -- 172.21.15.119:0/2697247835 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbf81a0c30 con 0x7fcbf8103600 2024-03-19T03:28:44.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.056+0000 7fcbffe09700 1 -- 172.21.15.119:0/2697247835 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbf8068fd0 con 0x7fcbf8103600 2024-03-19T03:28:44.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.057+0000 7fcbeeffd700 1 -- 172.21.15.119:0/2697247835 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fcbe8018020 con 0x7fcbf8103600 2024-03-19T03:28:44.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.058+0000 7fcbeeffd700 1 --2- 172.21.15.119:0/2697247835 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fcbe40a2910 0x7fcbe40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:28:44.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.058+0000 7fcbeeffd700 1 -- 172.21.15.119:0/2697247835 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7fcbe80cd9e0 con 0x7fcbf8103600 2024-03-19T03:28:44.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.058+0000 7fcbfdba5700 1 --2- 172.21.15.119:0/2697247835 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fcbe40a2910 0x7fcbe40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:28:44.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.059+0000 7fcbfdba5700 1 --2- 172.21.15.119:0/2697247835 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fcbe40a2910 0x7fcbe40a4dd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4005f50 tx=0x7fcbf4005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:28:44.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.062+0000 7fcbeeffd700 1 -- 172.21.15.119:0/2697247835 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcbe801f020 con 0x7fcbf8103600 2024-03-19T03:28:44.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:28:44.272+0000 7fcbffe09700 1 -- 172.21.15.119:0/2697247835 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fcbf8062750 con 0x7fcbe40a2910 2024-03-19T03:28:45.148 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:44 smithi119 ceph-mon[66861]: Detected new or changed devices on smithi117 2024-03-19T03:28:45.148 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:44 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:45.148 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:44 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:45.149 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:44 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:28:45.149 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:44 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:28:45.149 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:44 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:28:45.149 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:44 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:28:45.149 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:44 smithi119 ceph-mon[66861]: Adjusting osd_memory_target on smithi117 to 4272M 2024-03-19T03:28:45.149 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:44 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:45.149 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:44 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:28:45.149 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:44 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:28:45.149 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:44 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:45.149 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:44 smithi119 ceph-mon[66861]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:45.149 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:44 smithi119 ceph-mon[66861]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:28:45.149 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:44 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:28:45.149 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:44 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:28:45.149 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:44 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:28:45.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:44 smithi117 ceph-mon[31838]: Detected new or changed devices on smithi117 2024-03-19T03:28:45.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:44 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:45.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:44 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:45.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:44 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:28:45.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:44 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:28:45.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:44 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:28:45.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:44 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:28:45.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:44 smithi117 ceph-mon[31838]: Adjusting osd_memory_target on smithi117 to 4272M 2024-03-19T03:28:45.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:44 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:45.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:44 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:28:45.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:44 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:28:45.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:44 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:45.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:44 smithi117 ceph-mon[31838]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:45.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:44 smithi117 ceph-mon[31838]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:28:45.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:44 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:28:45.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:44 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:28:45.294 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:44 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:28:45.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:44 smithi178 ceph-mon[67578]: Detected new or changed devices on smithi117 2024-03-19T03:28:45.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:44 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:45.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:44 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:45.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:44 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:28:45.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:44 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:28:45.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:44 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:28:45.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:44 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:28:45.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:44 smithi178 ceph-mon[67578]: Adjusting osd_memory_target on smithi117 to 4272M 2024-03-19T03:28:45.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:44 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:45.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:44 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:28:45.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:44 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:28:45.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:44 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:28:45.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:44 smithi178 ceph-mon[67578]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:45.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:44 smithi178 ceph-mon[67578]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:28:45.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:44 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:28:45.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:44 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:28:45.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:44 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:28:47.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:47 smithi178 ceph-mon[67578]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:47.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:47 smithi178 ceph-mon[67578]: from='client.? 172.21.15.119:0/48830460' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de2049da-d19e-4386-afbf-fd6b5fddbcba"}]: dispatch 2024-03-19T03:28:47.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:47 smithi178 ceph-mon[67578]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de2049da-d19e-4386-afbf-fd6b5fddbcba"}]: dispatch 2024-03-19T03:28:47.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:47 smithi178 ceph-mon[67578]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "de2049da-d19e-4386-afbf-fd6b5fddbcba"}]': finished 2024-03-19T03:28:47.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:47 smithi178 ceph-mon[67578]: osdmap e69: 11 total, 10 up, 11 in 2024-03-19T03:28:47.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:47 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T03:28:47.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:47 smithi119 ceph-mon[66861]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:47.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:47 smithi119 ceph-mon[66861]: from='client.? 172.21.15.119:0/48830460' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de2049da-d19e-4386-afbf-fd6b5fddbcba"}]: dispatch 2024-03-19T03:28:47.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:47 smithi119 ceph-mon[66861]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de2049da-d19e-4386-afbf-fd6b5fddbcba"}]: dispatch 2024-03-19T03:28:47.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:47 smithi119 ceph-mon[66861]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "de2049da-d19e-4386-afbf-fd6b5fddbcba"}]': finished 2024-03-19T03:28:47.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:47 smithi119 ceph-mon[66861]: osdmap e69: 11 total, 10 up, 11 in 2024-03-19T03:28:47.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:47 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T03:28:47.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:47 smithi117 ceph-mon[31838]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:47.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:47 smithi117 ceph-mon[31838]: from='client.? 172.21.15.119:0/48830460' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de2049da-d19e-4386-afbf-fd6b5fddbcba"}]: dispatch 2024-03-19T03:28:47.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:47 smithi117 ceph-mon[31838]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de2049da-d19e-4386-afbf-fd6b5fddbcba"}]: dispatch 2024-03-19T03:28:47.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:47 smithi117 ceph-mon[31838]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "de2049da-d19e-4386-afbf-fd6b5fddbcba"}]': finished 2024-03-19T03:28:47.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:47 smithi117 ceph-mon[31838]: osdmap e69: 11 total, 10 up, 11 in 2024-03-19T03:28:47.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:47 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T03:28:48.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:48 smithi117 ceph-mon[31838]: from='client.? 172.21.15.119:0/851522873' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:28:48.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:48 smithi178 ceph-mon[67578]: from='client.? 172.21.15.119:0/851522873' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:28:48.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:48 smithi119 ceph-mon[66861]: from='client.? 172.21.15.119:0/851522873' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:28:49.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:49 smithi178 ceph-mon[67578]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:49.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:49 smithi119 ceph-mon[66861]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:49.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:49 smithi117 ceph-mon[31838]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:51.371 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:51 smithi178 ceph-mon[67578]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:51.417 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:51 smithi119 ceph-mon[66861]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:51.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:51 smithi117 ceph-mon[31838]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:53.592 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:53 smithi119 ceph-mon[66861]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:53.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:53 smithi178 ceph-mon[67578]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:53.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:53 smithi117 ceph-mon[31838]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:55.429 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:55 smithi119 ceph-mon[66861]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:55.429 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-19T03:28:55.429 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:28:55.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:55 smithi178 ceph-mon[67578]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:55.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-19T03:28:55.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:28:55.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:55 smithi117 ceph-mon[31838]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:55.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-19T03:28:55.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:28:56.471 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:56 smithi119 ceph-mon[66861]: Deploying daemon osd.10 on smithi119 2024-03-19T03:28:56.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:56 smithi117 ceph-mon[31838]: Deploying daemon osd.10 on smithi119 2024-03-19T03:28:56.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:56 smithi178 ceph-mon[67578]: Deploying daemon osd.10 on smithi119 2024-03-19T03:28:57.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:57 smithi119 ceph-mon[66861]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:57.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:57 smithi117 ceph-mon[31838]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:57.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:57 smithi178 ceph-mon[67578]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:59.712 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:28:59 smithi119 ceph-mon[66861]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:59.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:28:59 smithi117 ceph-mon[31838]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:28:59.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:28:59 smithi178 ceph-mon[67578]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:29:01.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:01 smithi119 ceph-mon[66861]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:29:01.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:01 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:01.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:01 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:01.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:01 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:29:01.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:01 smithi117 ceph-mon[31838]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:29:01.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:01 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:01.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:01 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:01.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:01 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:29:01.806 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:01 smithi178 ceph-mon[67578]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:29:01.806 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:01 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:01.806 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:01 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:01.806 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:01 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:29:03.652 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:03 smithi119 ceph-mon[66861]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:29:03.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:03 smithi117 ceph-mon[31838]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:29:03.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:03 smithi178 ceph-mon[67578]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:29:04.361 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 10 on host 'smithi119' 2024-03-19T03:29:04.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:29:04.359+0000 7fcbeeffd700 1 -- 172.21.15.119:0/2697247835 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fcbf8062750 con 0x7fcbe40a2910 2024-03-19T03:29:04.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:29:04.361+0000 7fcbffe09700 1 -- 172.21.15.119:0/2697247835 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fcbe40a2910 msgr2=0x7fcbe40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:04.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:29:04.361+0000 7fcbffe09700 1 --2- 172.21.15.119:0/2697247835 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fcbe40a2910 0x7fcbe40a4dd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4005f50 tx=0x7fcbf4005e80 comp rx=0 tx=0).stop 2024-03-19T03:29:04.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:29:04.361+0000 7fcbffe09700 1 -- 172.21.15.119:0/2697247835 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbf8103600 msgr2=0x7fcbf8198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:04.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:29:04.361+0000 7fcbffe09700 1 --2- 172.21.15.119:0/2697247835 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbf8103600 0x7fcbf8198c10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcbe800b530 tx=0x7fcbe800b8f0 comp rx=0 tx=0).stop 2024-03-19T03:29:04.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:29:04.361+0000 7fcbffe09700 1 -- 172.21.15.119:0/2697247835 shutdown_connections 2024-03-19T03:29:04.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:29:04.361+0000 7fcbffe09700 1 --2- 172.21.15.119:0/2697247835 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fcbe40a2910 0x7fcbe40a4dd0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:04.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:29:04.361+0000 7fcbffe09700 1 --2- 172.21.15.119:0/2697247835 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcbf8103fa0 0x7fcbf819fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:04.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:29:04.361+0000 7fcbffe09700 1 --2- 172.21.15.119:0/2697247835 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbf8100620 0x7fcbf81986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:04.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:29:04.361+0000 7fcbffe09700 1 --2- 172.21.15.119:0/2697247835 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbf8103600 0x7fcbf8198c10 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:04.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:29:04.361+0000 7fcbffe09700 1 -- 172.21.15.119:0/2697247835 >> 172.21.15.119:0/2697247835 conn(0x7fcbf80fbfa0 msgr2=0x7fcbf81069d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:29:04.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:29:04.361+0000 7fcbffe09700 1 -- 172.21.15.119:0/2697247835 shutdown_connections 2024-03-19T03:29:04.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-19T03:29:04.361+0000 7fcbffe09700 1 -- 172.21.15.119:0/2697247835 wait complete. 2024-03-19T03:29:04.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:04 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:04.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:04 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:04.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:04 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:04.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:04 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:29:04.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:04 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:04.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:04 smithi178 ceph-mon[67578]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:29:04.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:04 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:29:04.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:04 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:04.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:04 smithi178 ceph-mon[67578]: from='osd.10 [v2:172.21.15.119:6824/1050976194,v1:172.21.15.119:6825/1050976194]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-19T03:29:04.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:04 smithi178 ceph-mon[67578]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-19T03:29:04.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:04 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:04.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:04 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:04.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:04 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:04.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:04 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:04.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:04 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:29:04.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:04 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:04.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:04 smithi119 ceph-mon[66861]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:29:04.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:04 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:29:04.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:04 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:04.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:04 smithi119 ceph-mon[66861]: from='osd.10 [v2:172.21.15.119:6824/1050976194,v1:172.21.15.119:6825/1050976194]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-19T03:29:04.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:04 smithi119 ceph-mon[66861]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-19T03:29:04.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:04 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:05.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:04 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:05.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:04 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:05.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:04 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:05.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:04 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:29:05.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:04 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:05.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:04 smithi117 ceph-mon[31838]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:29:05.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:04 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:29:05.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:04 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:05.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:04 smithi117 ceph-mon[31838]: from='osd.10 [v2:172.21.15.119:6824/1050976194,v1:172.21.15.119:6825/1050976194]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-19T03:29:05.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:04 smithi117 ceph-mon[31838]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-19T03:29:05.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:04 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:05.050 DEBUG:teuthology.orchestra.run.smithi119:osd.10> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.10.service 2024-03-19T03:29:05.052 INFO:tasks.cephadm:Deploying osd.11 on smithi178 with /dev/vg_nvme/lv_1... 2024-03-19T03:29:05.053 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-19T03:29:05.299 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T03:29:06.498 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:06 smithi178 ceph-mon[67578]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-19T03:29:06.498 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:06 smithi178 ceph-mon[67578]: from='osd.10 [v2:172.21.15.119:6824/1050976194,v1:172.21.15.119:6825/1050976194]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-19T03:29:06.498 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:06 smithi178 ceph-mon[67578]: osdmap e70: 11 total, 10 up, 11 in 2024-03-19T03:29:06.498 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:06 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T03:29:06.498 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:06 smithi178 ceph-mon[67578]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-19T03:29:06.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:06 smithi119 ceph-mon[66861]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-19T03:29:06.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:06 smithi119 ceph-mon[66861]: from='osd.10 [v2:172.21.15.119:6824/1050976194,v1:172.21.15.119:6825/1050976194]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-19T03:29:06.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:06 smithi119 ceph-mon[66861]: osdmap e70: 11 total, 10 up, 11 in 2024-03-19T03:29:06.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:06 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T03:29:06.571 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:06 smithi119 ceph-mon[66861]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-19T03:29:06.571 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:29:06 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:29:06.313+0000 7faa3bb96700 -1 osd.10 0 waiting for initial osdmap 2024-03-19T03:29:06.572 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:29:06 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:29:06.327+0000 7faa37989700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T03:29:06.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:06 smithi117 ceph-mon[31838]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-19T03:29:06.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:06 smithi117 ceph-mon[31838]: from='osd.10 [v2:172.21.15.119:6824/1050976194,v1:172.21.15.119:6825/1050976194]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-19T03:29:06.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:06 smithi117 ceph-mon[31838]: osdmap e70: 11 total, 10 up, 11 in 2024-03-19T03:29:06.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:06 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T03:29:06.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:06 smithi117 ceph-mon[31838]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-03-19T03:29:07.391 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-19T03:29:07.391 INFO:teuthology.orchestra.run.smithi178.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-19T03:29:07.391 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10+0 records in 2024-03-19T03:29:07.391 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records out 2024-03-19T03:29:07.391 INFO:teuthology.orchestra.run.smithi178.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00831035 s, 1.3 GB/s 2024-03-19T03:29:07.391 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping successful for: 2024-03-19T03:29:07.392 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:07 smithi178 ceph-mon[67578]: purged_snaps scrub starts 2024-03-19T03:29:07.392 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:07 smithi178 ceph-mon[67578]: purged_snaps scrub ok 2024-03-19T03:29:07.392 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:07 smithi178 ceph-mon[67578]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:29:07.392 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:07 smithi178 ceph-mon[67578]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-19T03:29:07.392 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:07 smithi178 ceph-mon[67578]: osdmap e71: 11 total, 10 up, 11 in 2024-03-19T03:29:07.392 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:07 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T03:29:07.392 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:07 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T03:29:07.695 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:07 smithi117 ceph-mon[31838]: purged_snaps scrub starts 2024-03-19T03:29:07.695 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:07 smithi117 ceph-mon[31838]: purged_snaps scrub ok 2024-03-19T03:29:07.695 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:07 smithi117 ceph-mon[31838]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:29:07.695 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:07 smithi117 ceph-mon[31838]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-19T03:29:07.695 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:07 smithi117 ceph-mon[31838]: osdmap e71: 11 total, 10 up, 11 in 2024-03-19T03:29:07.695 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:07 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T03:29:07.695 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:07 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T03:29:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:07 smithi119 ceph-mon[66861]: purged_snaps scrub starts 2024-03-19T03:29:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:07 smithi119 ceph-mon[66861]: purged_snaps scrub ok 2024-03-19T03:29:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:07 smithi119 ceph-mon[66861]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:29:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:07 smithi119 ceph-mon[66861]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-03-19T03:29:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:07 smithi119 ceph-mon[66861]: osdmap e71: 11 total, 10 up, 11 in 2024-03-19T03:29:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:07 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T03:29:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:07 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T03:29:08.008 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi178:vg_nvme/lv_1 2024-03-19T03:29:08.255 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T03:29:08.526 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T03:29:08.526 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:08 smithi117 ceph-mon[31838]: osd.10 [v2:172.21.15.119:6824/1050976194,v1:172.21.15.119:6825/1050976194] boot 2024-03-19T03:29:08.526 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:08 smithi117 ceph-mon[31838]: osdmap e72: 11 total, 11 up, 11 in 2024-03-19T03:29:08.526 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T03:29:08.526 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:08.526 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:08.527 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:29:08.527 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:29:08.527 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:29:08.527 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:29:08.527 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:08.527 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:08.527 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:29:08.527 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:08 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:08.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T03:29:08.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:08 smithi178 ceph-mon[67578]: osd.10 [v2:172.21.15.119:6824/1050976194,v1:172.21.15.119:6825/1050976194] boot 2024-03-19T03:29:08.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:08 smithi178 ceph-mon[67578]: osdmap e72: 11 total, 11 up, 11 in 2024-03-19T03:29:08.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T03:29:08.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:08.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:08.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:29:08.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:29:08.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:29:08.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:29:08.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:08.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:08.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:29:08.597 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:08 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:08.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T03:29:08.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:08 smithi119 ceph-mon[66861]: osd.10 [v2:172.21.15.119:6824/1050976194,v1:172.21.15.119:6825/1050976194] boot 2024-03-19T03:29:08.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:08 smithi119 ceph-mon[66861]: osdmap e72: 11 total, 11 up, 11 in 2024-03-19T03:29:08.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T03:29:08.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:08.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:08.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:29:08.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:29:08.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:29:08.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:29:08.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:08.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:08.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:29:08.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:08 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:09.397 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:09 smithi178 ceph-mon[67578]: Detected new or changed devices on smithi119 2024-03-19T03:29:09.397 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:09 smithi178 ceph-mon[67578]: Adjusting osd_memory_target on smithi119 to 4270M 2024-03-19T03:29:09.397 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:09 smithi178 ceph-mon[67578]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:29:09.397 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:09 smithi178 ceph-mon[67578]: osdmap e73: 11 total, 11 up, 11 in 2024-03-19T03:29:09.710 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:09 smithi117 ceph-mon[31838]: Detected new or changed devices on smithi119 2024-03-19T03:29:09.711 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:09 smithi117 ceph-mon[31838]: Adjusting osd_memory_target on smithi119 to 4270M 2024-03-19T03:29:09.711 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:09 smithi117 ceph-mon[31838]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:29:09.711 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:09 smithi117 ceph-mon[31838]: osdmap e73: 11 total, 11 up, 11 in 2024-03-19T03:29:09.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:09 smithi119 ceph-mon[66861]: Detected new or changed devices on smithi119 2024-03-19T03:29:09.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:09 smithi119 ceph-mon[66861]: Adjusting osd_memory_target on smithi119 to 4270M 2024-03-19T03:29:09.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:09 smithi119 ceph-mon[66861]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-19T03:29:09.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:09 smithi119 ceph-mon[66861]: osdmap e73: 11 total, 11 up, 11 in 2024-03-19T03:29:09.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.953+0000 7fec51180700 1 -- 172.21.15.178:0/856520405 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fec4c102e80 msgr2=0x7fec4c109730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:09.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.953+0000 7fec51180700 1 --2- 172.21.15.178:0/856520405 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fec4c102e80 0x7fec4c109730 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fec4400b0a0 tx=0x7fec4400b3b0 comp rx=0 tx=0).stop 2024-03-19T03:29:09.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.953+0000 7fec51180700 1 -- 172.21.15.178:0/856520405 shutdown_connections 2024-03-19T03:29:09.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.953+0000 7fec51180700 1 --2- 172.21.15.178:0/856520405 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fec4c102e80 0x7fec4c109730 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:09.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.953+0000 7fec51180700 1 --2- 172.21.15.178:0/856520405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec4c1012e0 0x7fec4c101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:09.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.953+0000 7fec51180700 1 --2- 172.21.15.178:0/856520405 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec4c1024e0 0x7fec4c102940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:09.956 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.953+0000 7fec51180700 1 -- 172.21.15.178:0/856520405 >> 172.21.15.178:0/856520405 conn(0x7fec4c0fc8a0 msgr2=0x7fec4c0fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:29:09.956 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.953+0000 7fec51180700 1 -- 172.21.15.178:0/856520405 shutdown_connections 2024-03-19T03:29:09.956 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.953+0000 7fec51180700 1 -- 172.21.15.178:0/856520405 wait complete. 2024-03-19T03:29:09.956 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.954+0000 7fec51180700 1 Processor -- start 2024-03-19T03:29:09.956 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.954+0000 7fec51180700 1 -- start start 2024-03-19T03:29:09.956 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.954+0000 7fec51180700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fec4c1012e0 0x7fec4c06b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:09.956 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.954+0000 7fec51180700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec4c1024e0 0x7fec4c19d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:09.956 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.954+0000 7fec51180700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec4c102e80 0x7fec4c19f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:09.956 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.954+0000 7fec51180700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fec4c10ce50 con 0x7fec4c102e80 2024-03-19T03:29:09.956 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.954+0000 7fec51180700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fec4c10ccd0 con 0x7fec4c1024e0 2024-03-19T03:29:09.956 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.954+0000 7fec51180700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fec4c10cfd0 con 0x7fec4c1012e0 2024-03-19T03:29:09.956 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.954+0000 7fec4ad9d700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fec4c1012e0 0x7fec4c06b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:09.956 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.954+0000 7fec4ad9d700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fec4c1012e0 0x7fec4c06b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:48420/0 (socket says 172.21.15.178:48420) 2024-03-19T03:29:09.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.954+0000 7fec4ad9d700 1 -- 172.21.15.178:0/161994803 learned_addr learned my addr 172.21.15.178:0/161994803 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-19T03:29:09.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.954+0000 7fec43fff700 1 --2- 172.21.15.178:0/161994803 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec4c1024e0 0x7fec4c19d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:09.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.954+0000 7fec4ad9d700 1 -- 172.21.15.178:0/161994803 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec4c1024e0 msgr2=0x7fec4c19d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:09.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.954+0000 7fec4ad9d700 1 --2- 172.21.15.178:0/161994803 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec4c1024e0 0x7fec4c19d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:09.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.954+0000 7fec4ad9d700 1 -- 172.21.15.178:0/161994803 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec4c102e80 msgr2=0x7fec4c19f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:09.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.954+0000 7fec4ad9d700 1 --2- 172.21.15.178:0/161994803 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec4c102e80 0x7fec4c19f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:09.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.954+0000 7fec4ad9d700 1 -- 172.21.15.178:0/161994803 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec4c1a0160 con 0x7fec4c1012e0 2024-03-19T03:29:09.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.954+0000 7fec4ad9d700 1 --2- 172.21.15.178:0/161994803 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fec4c1012e0 0x7fec4c06b5d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00e950 tx=0x7fec3c00ec60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:29:09.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.955+0000 7fec48d99700 1 -- 172.21.15.178:0/161994803 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3c00eea0 con 0x7fec4c1012e0 2024-03-19T03:29:09.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.955+0000 7fec48d99700 1 -- 172.21.15.178:0/161994803 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec3c004510 con 0x7fec4c1012e0 2024-03-19T03:29:09.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.955+0000 7fec51180700 1 -- 172.21.15.178:0/161994803 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec4c1a0450 con 0x7fec4c1012e0 2024-03-19T03:29:09.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.955+0000 7fec48d99700 1 -- 172.21.15.178:0/161994803 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3c010430 con 0x7fec4c1012e0 2024-03-19T03:29:09.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.955+0000 7fec51180700 1 -- 172.21.15.178:0/161994803 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec4c1a0940 con 0x7fec4c1012e0 2024-03-19T03:29:09.958 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.956+0000 7fec51180700 1 -- 172.21.15.178:0/161994803 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec4c068fd0 con 0x7fec4c1012e0 2024-03-19T03:29:09.958 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.957+0000 7fec48d99700 1 -- 172.21.15.178:0/161994803 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fec3c003e90 con 0x7fec4c1012e0 2024-03-19T03:29:09.958 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.957+0000 7fec48d99700 1 --2- 172.21.15.178:0/161994803 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fec380a2890 0x7fec380a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:09.959 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.957+0000 7fec48d99700 1 -- 172.21.15.178:0/161994803 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6723+0+0 (secure 0 0 0) 0x7fec3c0ce7a0 con 0x7fec4c1012e0 2024-03-19T03:29:09.959 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.958+0000 7fec43fff700 1 --2- 172.21.15.178:0/161994803 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fec380a2890 0x7fec380a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:09.959 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.959+0000 7fec43fff700 1 --2- 172.21.15.178:0/161994803 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fec380a2890 0x7fec380a4d50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fec34005950 tx=0x7fec3400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:29:09.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:09.962+0000 7fec48d99700 1 -- 172.21.15.178:0/161994803 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fec3c09be70 con 0x7fec4c1012e0 2024-03-19T03:29:10.166 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:10.166+0000 7fec51180700 1 -- 172.21.15.178:0/161994803 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fec4c062750 con 0x7fec380a2890 2024-03-19T03:29:10.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:10 smithi178 ceph-mon[67578]: osdmap e74: 11 total, 11 up, 11 in 2024-03-19T03:29:10.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:10 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:29:10.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:10 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:29:10.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:10 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:10.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:10 smithi119 ceph-mon[66861]: osdmap e74: 11 total, 11 up, 11 in 2024-03-19T03:29:10.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:10 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:29:10.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:10 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:29:10.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:10 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:10.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:10 smithi117 ceph-mon[31838]: osdmap e74: 11 total, 11 up, 11 in 2024-03-19T03:29:10.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:10 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T03:29:10.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:10 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T03:29:10.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:10 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:11.724 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:11 smithi178 ceph-mon[67578]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:29:11.724 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:11 smithi178 ceph-mon[67578]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-03-19T03:29:11.724 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:11 smithi178 ceph-mon[67578]: osdmap e75: 11 total, 11 up, 11 in 2024-03-19T03:29:11.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:11 smithi119 ceph-mon[66861]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:29:11.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:11 smithi119 ceph-mon[66861]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-03-19T03:29:11.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:11 smithi119 ceph-mon[66861]: osdmap e75: 11 total, 11 up, 11 in 2024-03-19T03:29:11.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:11 smithi117 ceph-mon[31838]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:29:11.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:11 smithi117 ceph-mon[31838]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-03-19T03:29:11.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:11 smithi117 ceph-mon[31838]: osdmap e75: 11 total, 11 up, 11 in 2024-03-19T03:29:12.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:12 smithi119 ceph-mon[66861]: osdmap e76: 11 total, 11 up, 11 in 2024-03-19T03:29:12.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:12 smithi117 ceph-mon[31838]: osdmap e76: 11 total, 11 up, 11 in 2024-03-19T03:29:12.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:12 smithi178 ceph-mon[67578]: osdmap e76: 11 total, 11 up, 11 in 2024-03-19T03:29:13.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:13 smithi178 ceph-mon[67578]: pgmap v263: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-19T03:29:13.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:13 smithi178 ceph-mon[67578]: from='client.? 172.21.15.178:0/3660578012' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cccc0f9-d273-4b6b-9b8e-c926371472dd"}]: dispatch 2024-03-19T03:29:13.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:13 smithi178 ceph-mon[67578]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cccc0f9-d273-4b6b-9b8e-c926371472dd"}]: dispatch 2024-03-19T03:29:13.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:13 smithi178 ceph-mon[67578]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3cccc0f9-d273-4b6b-9b8e-c926371472dd"}]': finished 2024-03-19T03:29:13.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:13 smithi178 ceph-mon[67578]: osdmap e77: 12 total, 11 up, 12 in 2024-03-19T03:29:13.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:13 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T03:29:13.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:13 smithi119 ceph-mon[66861]: pgmap v263: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-19T03:29:13.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:13 smithi119 ceph-mon[66861]: from='client.? 172.21.15.178:0/3660578012' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cccc0f9-d273-4b6b-9b8e-c926371472dd"}]: dispatch 2024-03-19T03:29:13.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:13 smithi119 ceph-mon[66861]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cccc0f9-d273-4b6b-9b8e-c926371472dd"}]: dispatch 2024-03-19T03:29:13.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:13 smithi119 ceph-mon[66861]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3cccc0f9-d273-4b6b-9b8e-c926371472dd"}]': finished 2024-03-19T03:29:13.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:13 smithi119 ceph-mon[66861]: osdmap e77: 12 total, 11 up, 12 in 2024-03-19T03:29:13.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:13 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T03:29:13.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:13 smithi117 ceph-mon[31838]: pgmap v263: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-19T03:29:13.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:13 smithi117 ceph-mon[31838]: from='client.? 172.21.15.178:0/3660578012' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cccc0f9-d273-4b6b-9b8e-c926371472dd"}]: dispatch 2024-03-19T03:29:13.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:13 smithi117 ceph-mon[31838]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cccc0f9-d273-4b6b-9b8e-c926371472dd"}]: dispatch 2024-03-19T03:29:13.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:13 smithi117 ceph-mon[31838]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3cccc0f9-d273-4b6b-9b8e-c926371472dd"}]': finished 2024-03-19T03:29:13.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:13 smithi117 ceph-mon[31838]: osdmap e77: 12 total, 11 up, 12 in 2024-03-19T03:29:13.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:13 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T03:29:14.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:14 smithi119 ceph-mon[66861]: from='client.? 172.21.15.178:0/3374532632' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:29:14.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:14 smithi117 ceph-mon[31838]: from='client.? 172.21.15.178:0/3374532632' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:29:14.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:14 smithi178 ceph-mon[67578]: from='client.? 172.21.15.178:0/3374532632' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T03:29:15.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:15 smithi119 ceph-mon[66861]: pgmap v265: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-19T03:29:15.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:15 smithi117 ceph-mon[31838]: pgmap v265: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-19T03:29:15.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:15 smithi178 ceph-mon[67578]: pgmap v265: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-19T03:29:17.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:17 smithi178 ceph-mon[67578]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T03:29:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:17 smithi119 ceph-mon[66861]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T03:29:17.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:17 smithi117 ceph-mon[31838]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T03:29:19.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:19 smithi117 ceph-mon[31838]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-19T03:29:19.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:19 smithi178 ceph-mon[67578]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-19T03:29:19.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:19 smithi119 ceph-mon[66861]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-19T03:29:20.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:20 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-19T03:29:20.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:20 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:20.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:20 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-19T03:29:20.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:20 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:20.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:20 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-19T03:29:20.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:20 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:21.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:21 smithi178 ceph-mon[67578]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-19T03:29:21.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:21 smithi178 ceph-mon[67578]: Deploying daemon osd.11 on smithi178 2024-03-19T03:29:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:21 smithi119 ceph-mon[66861]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-19T03:29:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:21 smithi119 ceph-mon[66861]: Deploying daemon osd.11 on smithi178 2024-03-19T03:29:21.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:21 smithi117 ceph-mon[31838]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-19T03:29:21.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:21 smithi117 ceph-mon[31838]: Deploying daemon osd.11 on smithi178 2024-03-19T03:29:23.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:23 smithi119 ceph-mon[66861]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-19T03:29:23.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:23 smithi117 ceph-mon[31838]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-19T03:29:23.796 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:23 smithi178 ceph-mon[67578]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-19T03:29:25.724 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:25 smithi178 ceph-mon[67578]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-19T03:29:25.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:25 smithi119 ceph-mon[66861]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-19T03:29:25.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:25 smithi117 ceph-mon[31838]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-19T03:29:27.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:26 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:27.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:26 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:29:27.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:26 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:27.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:26 smithi119 ceph-mon[66861]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-19T03:29:27.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:26 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:27.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:26 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:29:27.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:26 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:27.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:26 smithi117 ceph-mon[31838]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-19T03:29:27.308 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:26 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:27.309 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:26 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:29:27.309 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:26 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:27.309 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:26 smithi178 ceph-mon[67578]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-19T03:29:29.356 INFO:teuthology.orchestra.run.smithi178.stdout:Created osd(s) 11 on host 'smithi178' 2024-03-19T03:29:29.356 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:29.355+0000 7fec48d99700 1 -- 172.21.15.178:0/161994803 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fec4c062750 con 0x7fec380a2890 2024-03-19T03:29:29.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:29.357+0000 7fec51180700 1 -- 172.21.15.178:0/161994803 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fec380a2890 msgr2=0x7fec380a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:29.359 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:29.357+0000 7fec51180700 1 --2- 172.21.15.178:0/161994803 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fec380a2890 0x7fec380a4d50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fec34005950 tx=0x7fec3400a300 comp rx=0 tx=0).stop 2024-03-19T03:29:29.359 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:29.357+0000 7fec51180700 1 -- 172.21.15.178:0/161994803 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fec4c1012e0 msgr2=0x7fec4c06b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:29.359 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:29.357+0000 7fec51180700 1 --2- 172.21.15.178:0/161994803 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fec4c1012e0 0x7fec4c06b5d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00e950 tx=0x7fec3c00ec60 comp rx=0 tx=0).stop 2024-03-19T03:29:29.359 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:29.357+0000 7fec51180700 1 -- 172.21.15.178:0/161994803 shutdown_connections 2024-03-19T03:29:29.359 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:29.357+0000 7fec51180700 1 --2- 172.21.15.178:0/161994803 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fec380a2890 0x7fec380a4d50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:29.359 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:29.357+0000 7fec51180700 1 --2- 172.21.15.178:0/161994803 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec4c102e80 0x7fec4c19f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:29.359 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:29.357+0000 7fec51180700 1 --2- 172.21.15.178:0/161994803 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fec4c1012e0 0x7fec4c06b5d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:29.359 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:29.357+0000 7fec51180700 1 --2- 172.21.15.178:0/161994803 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec4c1024e0 0x7fec4c19d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:29.359 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:29.357+0000 7fec51180700 1 -- 172.21.15.178:0/161994803 >> 172.21.15.178:0/161994803 conn(0x7fec4c0fc8a0 msgr2=0x7fec4c1057e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:29:29.360 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:29.357+0000 7fec51180700 1 -- 172.21.15.178:0/161994803 shutdown_connections 2024-03-19T03:29:29.360 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:29.357+0000 7fec51180700 1 -- 172.21.15.178:0/161994803 wait complete. 2024-03-19T03:29:29.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:29 smithi178 ceph-mon[67578]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-19T03:29:29.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:29 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:29.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:29 smithi119 ceph-mon[66861]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-19T03:29:29.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:29 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:29.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:29 smithi117 ceph-mon[31838]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-19T03:29:29.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:29 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:30.029 DEBUG:teuthology.orchestra.run.smithi178:osd.11> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.11.service 2024-03-19T03:29:30.031 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-19T03:29:30.031 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-19T03:29:30.479 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:29:30.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:30 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:30.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:30 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:30.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:30 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:30.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:30 smithi117 ceph-mon[31838]: from='osd.11 [v2:172.21.15.178:6824/2915080368,v1:172.21.15.178:6825/2915080368]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-19T03:29:30.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:30 smithi117 ceph-mon[31838]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-19T03:29:30.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:30 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:30.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:30 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:30.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:30 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:30.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:30 smithi178 ceph-mon[67578]: from='osd.11 [v2:172.21.15.178:6824/2915080368,v1:172.21.15.178:6825/2915080368]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-19T03:29:30.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:30 smithi178 ceph-mon[67578]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-19T03:29:30.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:30 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:30.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:30 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:30.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:30 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:30.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:30 smithi119 ceph-mon[66861]: from='osd.11 [v2:172.21.15.178:6824/2915080368,v1:172.21.15.178:6825/2915080368]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-19T03:29:30.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:30 smithi119 ceph-mon[66861]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-19T03:29:31.527 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:31 smithi178 ceph-mon[67578]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-19T03:29:31.528 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:31 smithi178 ceph-mon[67578]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-19T03:29:31.528 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:31 smithi178 ceph-mon[67578]: from='osd.11 [v2:172.21.15.178:6824/2915080368,v1:172.21.15.178:6825/2915080368]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-19T03:29:31.528 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:31 smithi178 ceph-mon[67578]: osdmap e78: 12 total, 11 up, 12 in 2024-03-19T03:29:31.528 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:31 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T03:29:31.528 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:31 smithi178 ceph-mon[67578]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-19T03:29:31.666 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:31 smithi117 ceph-mon[31838]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-19T03:29:31.666 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:31 smithi117 ceph-mon[31838]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-19T03:29:31.666 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:31 smithi117 ceph-mon[31838]: from='osd.11 [v2:172.21.15.178:6824/2915080368,v1:172.21.15.178:6825/2915080368]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-19T03:29:31.666 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:31 smithi117 ceph-mon[31838]: osdmap e78: 12 total, 11 up, 12 in 2024-03-19T03:29:31.666 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:31 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T03:29:31.666 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:31 smithi117 ceph-mon[31838]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-19T03:29:31.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:31 smithi119 ceph-mon[66861]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-19T03:29:31.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:31 smithi119 ceph-mon[66861]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-19T03:29:31.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:31 smithi119 ceph-mon[66861]: from='osd.11 [v2:172.21.15.178:6824/2915080368,v1:172.21.15.178:6825/2915080368]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-19T03:29:31.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:31 smithi119 ceph-mon[66861]: osdmap e78: 12 total, 11 up, 12 in 2024-03-19T03:29:31.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:31 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T03:29:31.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:31 smithi119 ceph-mon[66861]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2024-03-19T03:29:32.346 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:29:31 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:29:31.961+0000 7f93f7d44700 -1 osd.11 0 waiting for initial osdmap 2024-03-19T03:29:32.346 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:29:31 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:29:31.977+0000 7f93efb2f700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T03:29:32.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.675+0000 7f778ec00700 1 -- 172.21.15.117:0/3436369453 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7788076340 msgr2=0x7f77880747a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:32.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.675+0000 7f778ec00700 1 --2- 172.21.15.117:0/3436369453 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7788076340 0x7f77880747a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7778009990 tx=0x7f7778009ca0 comp rx=0 tx=0).stop 2024-03-19T03:29:32.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.675+0000 7f778ec00700 1 -- 172.21.15.117:0/3436369453 shutdown_connections 2024-03-19T03:29:32.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.675+0000 7f778ec00700 1 --2- 172.21.15.117:0/3436369453 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f778810a220 0x7f778810c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:32.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.675+0000 7f778ec00700 1 --2- 172.21.15.117:0/3436369453 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7788076340 0x7f77880747a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:32.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.675+0000 7f778ec00700 1 --2- 172.21.15.117:0/3436369453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7788074d70 0x7f77880751f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:32.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.675+0000 7f778ec00700 1 -- 172.21.15.117:0/3436369453 >> 172.21.15.117:0/3436369453 conn(0x7f77880fd8b0 msgr2=0x7f77880ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:29:32.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.675+0000 7f778ec00700 1 -- 172.21.15.117:0/3436369453 shutdown_connections 2024-03-19T03:29:32.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.675+0000 7f778ec00700 1 -- 172.21.15.117:0/3436369453 wait complete. 2024-03-19T03:29:32.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.676+0000 7f778ec00700 1 Processor -- start 2024-03-19T03:29:32.679 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.676+0000 7f778ec00700 1 -- start start 2024-03-19T03:29:32.679 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.677+0000 7f778ec00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7788074d70 0x7f778819a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:32.679 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.677+0000 7f778ec00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7788076340 0x7f778819ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:32.679 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.677+0000 7f778ec00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f778810a220 0x7f77881a1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:32.679 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.677+0000 7f778ec00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7788110190 con 0x7f778810a220 2024-03-19T03:29:32.679 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.677+0000 7f778ec00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7788110010 con 0x7f7788074d70 2024-03-19T03:29:32.679 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.677+0000 7f778ec00700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f7788110310 con 0x7f7788076340 2024-03-19T03:29:32.679 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.677+0000 7f7787400700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7788076340 0x7f778819ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:32.679 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.677+0000 7f7787400700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7788076340 0x7f778819ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:53134/0 (socket says 172.21.15.117:53134) 2024-03-19T03:29:32.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.677+0000 7f7787400700 1 -- 172.21.15.117:0/866674046 learned_addr learned my addr 172.21.15.117:0/866674046 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:29:32.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.677+0000 7f7787e00700 1 --2- 172.21.15.117:0/866674046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7788074d70 0x7f778819a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:32.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.677+0000 7f778cc00700 1 --2- 172.21.15.117:0/866674046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f778810a220 0x7f77881a1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:32.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.677+0000 7f7787400700 1 -- 172.21.15.117:0/866674046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7788074d70 msgr2=0x7f778819a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:32.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.677+0000 7f7787400700 1 --2- 172.21.15.117:0/866674046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7788074d70 0x7f778819a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:32.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.677+0000 7f7787400700 1 -- 172.21.15.117:0/866674046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f778810a220 msgr2=0x7f77881a1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:32.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.677+0000 7f7787400700 1 --2- 172.21.15.117:0/866674046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f778810a220 0x7f77881a1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:32.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.677+0000 7f7787400700 1 -- 172.21.15.117:0/866674046 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77881a2460 con 0x7f7788076340 2024-03-19T03:29:32.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.677+0000 7f7787e00700 1 --2- 172.21.15.117:0/866674046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7788074d70 0x7f778819a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:29:32.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.677+0000 7f778cc00700 1 --2- 172.21.15.117:0/866674046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f778810a220 0x7f77881a1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:29:32.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.678+0000 7f7787400700 1 --2- 172.21.15.117:0/866674046 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7788076340 0x7f778819ad40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f777400e9a0 tx=0x7f777400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:29:32.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.678+0000 7f7784c00700 1 -- 172.21.15.117:0/866674046 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f777400cbd0 con 0x7f7788076340 2024-03-19T03:29:32.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.678+0000 7f778ec00700 1 -- 172.21.15.117:0/866674046 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77881a2750 con 0x7f7788076340 2024-03-19T03:29:32.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.678+0000 7f7784c00700 1 -- 172.21.15.117:0/866674046 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7774004500 con 0x7f7788076340 2024-03-19T03:29:32.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.678+0000 7f7784c00700 1 -- 172.21.15.117:0/866674046 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7774003e00 con 0x7f7788076340 2024-03-19T03:29:32.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.678+0000 7f778ec00700 1 -- 172.21.15.117:0/866674046 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77881a2c90 con 0x7f7788076340 2024-03-19T03:29:32.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.680+0000 7f778ec00700 1 -- 172.21.15.117:0/866674046 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7788068fd0 con 0x7f7788076340 2024-03-19T03:29:32.682 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.681+0000 7f7784c00700 1 -- 172.21.15.117:0/866674046 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f7774003fe0 con 0x7f7788076340 2024-03-19T03:29:32.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.682+0000 7f7784c00700 1 --2- 172.21.15.117:0/866674046 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f776c0a2890 0x7f776c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:32.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.683+0000 7f7784c00700 1 -- 172.21.15.117:0/866674046 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6847+0+0 (secure 0 0 0) 0x7f77740ce820 con 0x7f7788076340 2024-03-19T03:29:32.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.683+0000 7f7787e00700 1 --2- 172.21.15.117:0/866674046 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f776c0a2890 0x7f776c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:32.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.683+0000 7f7787e00700 1 --2- 172.21.15.117:0/866674046 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f776c0a2890 0x7f776c0a4d50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7778000c00 tx=0x7f7778005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:29:32.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.686+0000 7f7784c00700 1 -- 172.21.15.117:0/866674046 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f777409bde0 con 0x7f7788076340 2024-03-19T03:29:32.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:32 smithi119 ceph-mon[66861]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-03-19T03:29:32.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:32 smithi119 ceph-mon[66861]: osdmap e79: 12 total, 11 up, 12 in 2024-03-19T03:29:32.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:32 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T03:29:32.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:32 smithi119 ceph-mon[66861]: from='osd.11 ' entity='osd.11' 2024-03-19T03:29:32.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:32 smithi119 ceph-mon[66861]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-19T03:29:32.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:32 smithi117 ceph-mon[31838]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-03-19T03:29:32.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:32 smithi117 ceph-mon[31838]: osdmap e79: 12 total, 11 up, 12 in 2024-03-19T03:29:32.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:32 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T03:29:32.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:32 smithi117 ceph-mon[31838]: from='osd.11 ' entity='osd.11' 2024-03-19T03:29:32.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:32 smithi117 ceph-mon[31838]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-19T03:29:32.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:32 smithi178 ceph-mon[67578]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2024-03-19T03:29:32.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:32 smithi178 ceph-mon[67578]: osdmap e79: 12 total, 11 up, 12 in 2024-03-19T03:29:32.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T03:29:32.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:32 smithi178 ceph-mon[67578]: from='osd.11 ' entity='osd.11' 2024-03-19T03:29:32.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:32 smithi178 ceph-mon[67578]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-19T03:29:32.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.929+0000 7f778ec00700 1 -- 172.21.15.117:0/866674046 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f7788062f20 con 0x7f7788076340 2024-03-19T03:29:32.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.930+0000 7f7784c00700 1 -- 172.21.15.117:0/866674046 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) ==== 74+0+133 (secure 0 0 0) 0x7f777409b6e0 con 0x7f7788076340 2024-03-19T03:29:32.931 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:29:32.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.933+0000 7f778ec00700 1 -- 172.21.15.117:0/866674046 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f776c0a2890 msgr2=0x7f776c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:32.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.933+0000 7f778ec00700 1 --2- 172.21.15.117:0/866674046 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f776c0a2890 0x7f776c0a4d50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7778000c00 tx=0x7f7778005fb0 comp rx=0 tx=0).stop 2024-03-19T03:29:32.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.933+0000 7f778ec00700 1 -- 172.21.15.117:0/866674046 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7788076340 msgr2=0x7f778819ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:32.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.933+0000 7f778ec00700 1 --2- 172.21.15.117:0/866674046 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7788076340 0x7f778819ad40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f777400e9a0 tx=0x7f777400ecb0 comp rx=0 tx=0).stop 2024-03-19T03:29:32.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.934+0000 7f778ec00700 1 -- 172.21.15.117:0/866674046 shutdown_connections 2024-03-19T03:29:32.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.934+0000 7f778ec00700 1 --2- 172.21.15.117:0/866674046 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f776c0a2890 0x7f776c0a4d50 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:32.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.934+0000 7f778ec00700 1 --2- 172.21.15.117:0/866674046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f778810a220 0x7f77881a1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:32.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.934+0000 7f778ec00700 1 --2- 172.21.15.117:0/866674046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7788074d70 0x7f778819a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:32.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.934+0000 7f778ec00700 1 --2- 172.21.15.117:0/866674046 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7788076340 0x7f778819ad40 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:32.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.934+0000 7f778ec00700 1 -- 172.21.15.117:0/866674046 >> 172.21.15.117:0/866674046 conn(0x7f77880fd8b0 msgr2=0x7f7788106120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:29:32.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.934+0000 7f778ec00700 1 -- 172.21.15.117:0/866674046 shutdown_connections 2024-03-19T03:29:32.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:32.934+0000 7f778ec00700 1 -- 172.21.15.117:0/866674046 wait complete. 2024-03-19T03:29:33.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:33 smithi119 ceph-mon[66861]: purged_snaps scrub starts 2024-03-19T03:29:33.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:33 smithi119 ceph-mon[66861]: purged_snaps scrub ok 2024-03-19T03:29:33.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T03:29:33.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:33 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/866674046' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-19T03:29:33.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:33 smithi119 ceph-mon[66861]: osd.11 [v2:172.21.15.178:6824/2915080368,v1:172.21.15.178:6825/2915080368] boot 2024-03-19T03:29:33.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:33 smithi119 ceph-mon[66861]: osdmap e80: 12 total, 12 up, 12 in 2024-03-19T03:29:33.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T03:29:33.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:33 smithi119 ceph-mon[66861]: Detected new or changed devices on smithi178 2024-03-19T03:29:33.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:33.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:33.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:29:33.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:29:33.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:29:33.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:29:33.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:33 smithi119 ceph-mon[66861]: Adjusting osd_memory_target on smithi178 to 4270M 2024-03-19T03:29:33.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:33.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:33.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:29:33.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:33.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:33 smithi117 ceph-mon[31838]: purged_snaps scrub starts 2024-03-19T03:29:33.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:33 smithi117 ceph-mon[31838]: purged_snaps scrub ok 2024-03-19T03:29:33.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T03:29:33.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:33 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/866674046' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-19T03:29:33.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:33 smithi117 ceph-mon[31838]: osd.11 [v2:172.21.15.178:6824/2915080368,v1:172.21.15.178:6825/2915080368] boot 2024-03-19T03:29:33.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:33 smithi117 ceph-mon[31838]: osdmap e80: 12 total, 12 up, 12 in 2024-03-19T03:29:33.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T03:29:33.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:33 smithi117 ceph-mon[31838]: Detected new or changed devices on smithi178 2024-03-19T03:29:33.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:33.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:33.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:29:33.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:29:33.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:29:33.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:29:33.794 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:33 smithi117 ceph-mon[31838]: Adjusting osd_memory_target on smithi178 to 4270M 2024-03-19T03:29:33.794 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:33.794 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:33.794 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:29:33.794 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:33 smithi178 ceph-mon[67578]: purged_snaps scrub starts 2024-03-19T03:29:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:33 smithi178 ceph-mon[67578]: purged_snaps scrub ok 2024-03-19T03:29:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T03:29:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:33 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/866674046' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-19T03:29:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:33 smithi178 ceph-mon[67578]: osd.11 [v2:172.21.15.178:6824/2915080368,v1:172.21.15.178:6825/2915080368] boot 2024-03-19T03:29:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:33 smithi178 ceph-mon[67578]: osdmap e80: 12 total, 12 up, 12 in 2024-03-19T03:29:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T03:29:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:33 smithi178 ceph-mon[67578]: Detected new or changed devices on smithi178 2024-03-19T03:29:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:33.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:29:33.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:29:33.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:29:33.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:29:33.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:33 smithi178 ceph-mon[67578]: Adjusting osd_memory_target on smithi178 to 4270M 2024-03-19T03:29:33.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:33.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:33.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:29:33.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:33.896 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":79,"num_osds":12,"num_up_osds":11,"osd_up_since":1710818947,"num_in_osds":12,"osd_in_since":1710818952,"num_remapped_pgs":0} 2024-03-19T03:29:34.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:34 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:29:34.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:34 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:34.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:34 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:29:34.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:34 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:34.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:34 smithi178 ceph-mon[67578]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:34.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:34 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:29:34.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:34 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:34.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:34 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:29:34.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:34 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:34.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:34 smithi119 ceph-mon[66861]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:34.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:34 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:29:34.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:34 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:34.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:34 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:29:34.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:34 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:34.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:34 smithi117 ceph-mon[31838]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:34.897 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-19T03:29:35.343 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:29:35.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:35 smithi117 ceph-mon[31838]: osdmap e81: 12 total, 12 up, 12 in 2024-03-19T03:29:35.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:35 smithi178 ceph-mon[67578]: osdmap e81: 12 total, 12 up, 12 in 2024-03-19T03:29:35.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:35 smithi119 ceph-mon[66861]: osdmap e81: 12 total, 12 up, 12 in 2024-03-19T03:29:36.740 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:36 smithi117 ceph-mon[31838]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:36.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:36 smithi178 ceph-mon[67578]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:36.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:36 smithi119 ceph-mon[66861]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:37.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.586+0000 7fca14200700 1 -- 172.21.15.117:0/3815093711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca0c105790 msgr2=0x7fca0c105bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:37.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.586+0000 7fca14200700 1 --2- 172.21.15.117:0/3815093711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca0c105790 0x7fca0c105bf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc9f80099c0 tx=0x7fc9f8009cd0 comp rx=0 tx=0).stop 2024-03-19T03:29:37.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.586+0000 7fca14200700 1 -- 172.21.15.117:0/3815093711 shutdown_connections 2024-03-19T03:29:37.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.586+0000 7fca14200700 1 --2- 172.21.15.117:0/3815093711 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca0c106130 0x7fca0c10c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:37.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.586+0000 7fca14200700 1 --2- 172.21.15.117:0/3815093711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca0c104590 0x7fca0c1049b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:37.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.586+0000 7fca14200700 1 --2- 172.21.15.117:0/3815093711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca0c105790 0x7fca0c105bf0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:37.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.586+0000 7fca14200700 1 -- 172.21.15.117:0/3815093711 >> 172.21.15.117:0/3815093711 conn(0x7fca0c0ffb30 msgr2=0x7fca0c101f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:29:37.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.586+0000 7fca14200700 1 -- 172.21.15.117:0/3815093711 shutdown_connections 2024-03-19T03:29:37.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.586+0000 7fca14200700 1 -- 172.21.15.117:0/3815093711 wait complete. 2024-03-19T03:29:37.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.587+0000 7fca14200700 1 Processor -- start 2024-03-19T03:29:37.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.587+0000 7fca14200700 1 -- start start 2024-03-19T03:29:37.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.587+0000 7fca14200700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca0c104590 0x7fca0c19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:37.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.587+0000 7fca14200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca0c105790 0x7fca0c19ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:37.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.588+0000 7fca14200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca0c106130 0x7fca0c1a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:37.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.588+0000 7fca14200700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c110150 con 0x7fca0c105790 2024-03-19T03:29:37.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.588+0000 7fca14200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c10ffd0 con 0x7fca0c106130 2024-03-19T03:29:37.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.588+0000 7fca14200700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c1102d0 con 0x7fca0c104590 2024-03-19T03:29:37.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.588+0000 7fca10e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca0c105790 0x7fca0c19ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:37.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.588+0000 7fca12200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca0c106130 0x7fca0c1a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:37.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.588+0000 7fca10e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca0c105790 0x7fca0c19ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40380/0 (socket says 172.21.15.117:40380) 2024-03-19T03:29:37.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.588+0000 7fca10e00700 1 -- 172.21.15.117:0/759639939 learned_addr learned my addr 172.21.15.117:0/759639939 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:29:37.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.588+0000 7fca11800700 1 --2- 172.21.15.117:0/759639939 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca0c104590 0x7fca0c19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:37.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.588+0000 7fca12200700 1 -- 172.21.15.117:0/759639939 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca0c104590 msgr2=0x7fca0c19a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:37.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.588+0000 7fca12200700 1 --2- 172.21.15.117:0/759639939 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca0c104590 0x7fca0c19a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:37.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.588+0000 7fca12200700 1 -- 172.21.15.117:0/759639939 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca0c105790 msgr2=0x7fca0c19ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:37.591 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.588+0000 7fca12200700 1 --2- 172.21.15.117:0/759639939 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca0c105790 0x7fca0c19ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:37.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.588+0000 7fca12200700 1 -- 172.21.15.117:0/759639939 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca0c1a2390 con 0x7fca0c106130 2024-03-19T03:29:37.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.588+0000 7fca10e00700 1 --2- 172.21.15.117:0/759639939 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca0c105790 0x7fca0c19ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:29:37.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.589+0000 7fca12200700 1 --2- 172.21.15.117:0/759639939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca0c106130 0x7fca0c1a1cf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fca04007f10 tx=0x7fca0400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:29:37.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.589+0000 7fca0a000700 1 -- 172.21.15.117:0/759639939 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca04004db0 con 0x7fca0c106130 2024-03-19T03:29:37.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.589+0000 7fca0a000700 1 -- 172.21.15.117:0/759639939 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca04009e20 con 0x7fca0c106130 2024-03-19T03:29:37.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.589+0000 7fca0a000700 1 -- 172.21.15.117:0/759639939 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca04015400 con 0x7fca0c106130 2024-03-19T03:29:37.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.589+0000 7fca14200700 1 -- 172.21.15.117:0/759639939 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca0c1a2680 con 0x7fca0c106130 2024-03-19T03:29:37.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.589+0000 7fca14200700 1 -- 172.21.15.117:0/759639939 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca0c1a2b70 con 0x7fca0c106130 2024-03-19T03:29:37.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.590+0000 7fca14200700 1 -- 172.21.15.117:0/759639939 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca0c068fd0 con 0x7fca0c106130 2024-03-19T03:29:37.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.592+0000 7fca0a000700 1 -- 172.21.15.117:0/759639939 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fca0400db20 con 0x7fca0c106130 2024-03-19T03:29:37.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.593+0000 7fca0a000700 1 --2- 172.21.15.117:0/759639939 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc9f40a2890 0x7fc9f40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:37.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.594+0000 7fca0a000700 1 -- 172.21.15.117:0/759639939 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fca040cdf40 con 0x7fca0c106130 2024-03-19T03:29:37.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.594+0000 7fca11800700 1 --2- 172.21.15.117:0/759639939 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc9f40a2890 0x7fc9f40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:37.595 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.594+0000 7fca11800700 1 --2- 172.21.15.117:0/759639939 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc9f40a2890 0x7fc9f40a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fca00009680 tx=0x7fca00009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:29:37.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.598+0000 7fca0a000700 1 -- 172.21.15.117:0/759639939 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fca04010070 con 0x7fca0c106130 2024-03-19T03:29:37.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.822+0000 7fca14200700 1 -- 172.21.15.117:0/759639939 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fca0c062f20 con 0x7fca0c106130 2024-03-19T03:29:37.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.822+0000 7fca0a000700 1 -- 172.21.15.117:0/759639939 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7fca0409b3e0 con 0x7fca0c106130 2024-03-19T03:29:37.824 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:29:37.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.826+0000 7fca14200700 1 -- 172.21.15.117:0/759639939 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc9f40a2890 msgr2=0x7fc9f40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:37.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.826+0000 7fca14200700 1 --2- 172.21.15.117:0/759639939 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc9f40a2890 0x7fc9f40a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fca00009680 tx=0x7fca00009260 comp rx=0 tx=0).stop 2024-03-19T03:29:37.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.826+0000 7fca14200700 1 -- 172.21.15.117:0/759639939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca0c106130 msgr2=0x7fca0c1a1cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:37.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.826+0000 7fca14200700 1 --2- 172.21.15.117:0/759639939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca0c106130 0x7fca0c1a1cf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fca04007f10 tx=0x7fca0400d3b0 comp rx=0 tx=0).stop 2024-03-19T03:29:37.828 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.827+0000 7fca14200700 1 -- 172.21.15.117:0/759639939 shutdown_connections 2024-03-19T03:29:37.828 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.827+0000 7fca14200700 1 --2- 172.21.15.117:0/759639939 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc9f40a2890 0x7fc9f40a4d50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:37.828 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.827+0000 7fca14200700 1 --2- 172.21.15.117:0/759639939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca0c106130 0x7fca0c1a1cf0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:37.828 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.827+0000 7fca14200700 1 --2- 172.21.15.117:0/759639939 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca0c104590 0x7fca0c19a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:37.828 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.827+0000 7fca14200700 1 --2- 172.21.15.117:0/759639939 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca0c105790 0x7fca0c19ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:37.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.827+0000 7fca14200700 1 -- 172.21.15.117:0/759639939 >> 172.21.15.117:0/759639939 conn(0x7fca0c0ffb30 msgr2=0x7fca0c108a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:29:37.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.827+0000 7fca14200700 1 -- 172.21.15.117:0/759639939 shutdown_connections 2024-03-19T03:29:37.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:37.827+0000 7fca14200700 1 -- 172.21.15.117:0/759639939 wait complete. 2024-03-19T03:29:38.211 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:37 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/759639939' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-19T03:29:38.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:37 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/759639939' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-19T03:29:38.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:37 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/759639939' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-19T03:29:38.648 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1710818972,"num_in_osds":12,"osd_in_since":1710818952,"num_remapped_pgs":0} 2024-03-19T03:29:38.649 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-19T03:29:39.087 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:29:39.233 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:38 smithi117 ceph-mon[31838]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:39.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:38 smithi119 ceph-mon[66861]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:39.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:38 smithi178 ceph-mon[67578]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:41.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.222+0000 7efd86200700 1 -- 172.21.15.117:0/3957369464 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd801025c0 msgr2=0x7efd801029a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:41.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.222+0000 7efd86200700 1 --2- 172.21.15.117:0/3957369464 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd801025c0 0x7efd801029a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efd700098f0 tx=0x7efd70009c00 comp rx=0 tx=0).stop 2024-03-19T03:29:41.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.222+0000 7efd86200700 1 -- 172.21.15.117:0/3957369464 shutdown_connections 2024-03-19T03:29:41.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.222+0000 7efd86200700 1 --2- 172.21.15.117:0/3957369464 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd80107690 0x7efd8010f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:41.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.222+0000 7efd86200700 1 --2- 172.21.15.117:0/3957369464 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd801025c0 0x7efd801029a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:41.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.222+0000 7efd86200700 1 --2- 172.21.15.117:0/3957369464 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efd80102f70 0x7efd80106fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:41.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.222+0000 7efd86200700 1 -- 172.21.15.117:0/3957369464 >> 172.21.15.117:0/3957369464 conn(0x7efd80079470 msgr2=0x7efd80079880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:29:41.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.222+0000 7efd86200700 1 -- 172.21.15.117:0/3957369464 shutdown_connections 2024-03-19T03:29:41.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.222+0000 7efd86200700 1 -- 172.21.15.117:0/3957369464 wait complete. 2024-03-19T03:29:41.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.223+0000 7efd86200700 1 Processor -- start 2024-03-19T03:29:41.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.224+0000 7efd86200700 1 -- start start 2024-03-19T03:29:41.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.224+0000 7efd86200700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efd801025c0 0x7efd80074610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:41.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.224+0000 7efd86200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd80102f70 0x7efd80074b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:41.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.224+0000 7efd86200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd80107690 0x7efd80075090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:41.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.224+0000 7efd86200700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7efd801126c0 con 0x7efd80102f70 2024-03-19T03:29:41.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.224+0000 7efd86200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7efd80112540 con 0x7efd80107690 2024-03-19T03:29:41.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.224+0000 7efd86200700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7efd80112840 con 0x7efd801025c0 2024-03-19T03:29:41.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.225+0000 7efd7f400700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efd801025c0 0x7efd80074610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:41.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.225+0000 7efd77e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd80102f70 0x7efd80074b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:41.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.225+0000 7efd7f400700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efd801025c0 0x7efd80074610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:51946/0 (socket says 172.21.15.117:51946) 2024-03-19T03:29:41.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.225+0000 7efd7f400700 1 -- 172.21.15.117:0/3963531917 learned_addr learned my addr 172.21.15.117:0/3963531917 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:29:41.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.225+0000 7efd7fe00700 1 --2- 172.21.15.117:0/3963531917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd80107690 0x7efd80075090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:41.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.225+0000 7efd7f400700 1 -- 172.21.15.117:0/3963531917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd80107690 msgr2=0x7efd80075090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:41.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.225+0000 7efd7f400700 1 --2- 172.21.15.117:0/3963531917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd80107690 0x7efd80075090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:41.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.225+0000 7efd7f400700 1 -- 172.21.15.117:0/3963531917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd80102f70 msgr2=0x7efd80074b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:41.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.225+0000 7efd7f400700 1 --2- 172.21.15.117:0/3963531917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd80102f70 0x7efd80074b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:41.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.225+0000 7efd7f400700 1 -- 172.21.15.117:0/3963531917 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd801a71c0 con 0x7efd801025c0 2024-03-19T03:29:41.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.225+0000 7efd7f400700 1 --2- 172.21.15.117:0/3963531917 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efd801025c0 0x7efd80074610 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efd70009fd0 tx=0x7efd700094b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:29:41.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.226+0000 7efd7cc00700 1 -- 172.21.15.117:0/3963531917 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd70025070 con 0x7efd801025c0 2024-03-19T03:29:41.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.226+0000 7efd7cc00700 1 -- 172.21.15.117:0/3963531917 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd7000b560 con 0x7efd801025c0 2024-03-19T03:29:41.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.226+0000 7efd7cc00700 1 -- 172.21.15.117:0/3963531917 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd70004c30 con 0x7efd801025c0 2024-03-19T03:29:41.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.226+0000 7efd86200700 1 -- 172.21.15.117:0/3963531917 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd801a7450 con 0x7efd801025c0 2024-03-19T03:29:41.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.226+0000 7efd86200700 1 -- 172.21.15.117:0/3963531917 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd801a78a0 con 0x7efd801025c0 2024-03-19T03:29:41.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.227+0000 7efd86200700 1 -- 172.21.15.117:0/3963531917 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd80068fd0 con 0x7efd801025c0 2024-03-19T03:29:41.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.229+0000 7efd7cc00700 1 -- 172.21.15.117:0/3963531917 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7efd70028070 con 0x7efd801025c0 2024-03-19T03:29:41.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.230+0000 7efd7cc00700 1 --2- 172.21.15.117:0/3963531917 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7efd640a2890 0x7efd640a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:41.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.231+0000 7efd7cc00700 1 -- 172.21.15.117:0/3963531917 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7efd700cef70 con 0x7efd801025c0 2024-03-19T03:29:41.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.231+0000 7efd77e00700 1 --2- 172.21.15.117:0/3963531917 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7efd640a2890 0x7efd640a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:41.232 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.231+0000 7efd77e00700 1 --2- 172.21.15.117:0/3963531917 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7efd640a2890 0x7efd640a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7efd80102030 tx=0x7efd68009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:29:41.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.232+0000 7efd7cc00700 1 -- 172.21.15.117:0/3963531917 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efd70013030 con 0x7efd801025c0 2024-03-19T03:29:41.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.457+0000 7efd86200700 1 -- 172.21.15.117:0/3963531917 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efd8010a900 con 0x7efd801025c0 2024-03-19T03:29:41.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.459+0000 7efd7cc00700 1 -- 172.21.15.117:0/3963531917 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18538 (secure 0 0 0) 0x7efd7000bdb0 con 0x7efd801025c0 2024-03-19T03:29:41.460 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:29:41.460 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":81,"fsid":"a3a82456-e59f-11ee-95c9-87774f69a715","created":"2024-03-19T03:21:30.427136+0000","modified":"2024-03-19T03:29:34.539579+0000","last_up_change":"2024-03-19T03:29:32.967505+0000","last_in_change":"2024-03-19T03:29:12.848740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:25:18.824270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b32eb0ee-9d4b-4748-87ec-eda02c907415","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6803","nonce":3014934900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6805","nonce":3014934900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6809","nonce":3014934900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6807","nonce":3014934900}]},"public_addr":"172.21.15.117:6803/3014934900","cluster_addr":"172.21.15.117:6805/3014934900","heartbeat_back_addr":"172.21.15.117:6809/3014934900","heartbeat_front_addr":"172.21.15.117:6807/3014934900","state":["exists","up"]},{"osd":1,"uuid":"e07585be-c016-4466-b07c-abbc22a58de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6801","nonce":2425927341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6803","nonce":2425927341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6807","nonce":2425927341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6805","nonce":2425927341}]},"public_addr":"172.21.15.119:6801/2425927341","cluster_addr":"172.21.15.119:6803/2425927341","heartbeat_back_addr":"172.21.15.119:6807/2425927341","heartbeat_front_addr":"172.21.15.119:6805/2425927341","state":["exists","up"]},{"osd":2,"uuid":"ede9edda-3b9f-41ca-ad04-5dcb738ab757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6801","nonce":3733812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6803","nonce":3733812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6807","nonce":3733812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6805","nonce":3733812989}]},"public_addr":"172.21.15.178:6801/3733812989","cluster_addr":"172.21.15.178:6803/3733812989","heartbeat_back_addr":"172.21.15.178:6807/3733812989","heartbeat_front_addr":"172.21.15.178:6805/3733812989","state":["exists","up"]},{"osd":3,"uuid":"f81c41e1-3a69-40db-a90f-58151286b081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6811","nonce":2178310135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6813","nonce":2178310135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6817","nonce":2178310135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6815","nonce":2178310135}]},"public_addr":"172.21.15.117:6811/2178310135","cluster_addr":"172.21.15.117:6813/2178310135","heartbeat_back_addr":"172.21.15.117:6817/2178310135","heartbeat_front_addr":"172.21.15.117:6815/2178310135","state":["exists","up"]},{"osd":4,"uuid":"caf7ce86-caf0-4eaf-bbab-bd2d93e63c33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6809","nonce":3455398086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6811","nonce":3455398086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6815","nonce":3455398086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6813","nonce":3455398086}]},"public_addr":"172.21.15.119:6809/3455398086","cluster_addr":"172.21.15.119:6811/3455398086","heartbeat_back_addr":"172.21.15.119:6815/3455398086","heartbeat_front_addr":"172.21.15.119:6813/3455398086","state":["exists","up"]},{"osd":5,"uuid":"20ba0b8a-2e0f-4ea7-b435-430e44766200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6809","nonce":1259631912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6811","nonce":1259631912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6815","nonce":1259631912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6813","nonce":1259631912}]},"public_addr":"172.21.15.178:6809/1259631912","cluster_addr":"172.21.15.178:6811/1259631912","heartbeat_back_addr":"172.21.15.178:6815/1259631912","heartbeat_front_addr":"172.21.15.178:6813/1259631912","state":["exists","up"]},{"osd":6,"uuid":"75c2ddd7-f837-42e0-ac47-8cfbcf97a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6819","nonce":864146757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6821","nonce":864146757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6825","nonce":864146757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6823","nonce":864146757}]},"public_addr":"172.21.15.117:6819/864146757","cluster_addr":"172.21.15.117:6821/864146757","heartbeat_back_addr":"172.21.15.117:6825/864146757","heartbeat_front_addr":"172.21.15.117:6823/864146757","state":["exists","up"]},{"osd":7,"uuid":"929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6817","nonce":1273365545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6819","nonce":1273365545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6823","nonce":1273365545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6821","nonce":1273365545}]},"public_addr":"172.21.15.119:6817/1273365545","cluster_addr":"172.21.15.119:6819/1273365545","heartbeat_back_addr":"172.21.15.119:6823/1273365545","heartbeat_front_addr":"172.21.15.119:6821/1273365545","state":["exists","up"]},{"osd":8,"uuid":"fd371461-a538-4781-92e1-5eead4ca3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6817","nonce":1059587501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6819","nonce":1059587501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6823","nonce":1059587501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6821","nonce":1059587501}]},"public_addr":"172.21.15.178:6817/1059587501","cluster_addr":"172.21.15.178:6819/1059587501","heartbeat_back_addr":"172.21.15.178:6823/1059587501","heartbeat_front_addr":"172.21.15.178:6821/1059587501","state":["exists","up"]},{"osd":9,"uuid":"0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6827","nonce":3653875600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6829","nonce":3653875600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6833","nonce":3653875600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6831","nonce":3653875600}]},"public_addr":"172.21.15.117:6827/3653875600","cluster_addr":"172.21.15.117:6829/3653875600","heartbeat_back_addr":"172.21.15.117:6833/3653875600","heartbeat_front_addr":"172.21.15.117:6831/3653875600","state":["exists","up"]},{"osd":10,"uuid":"de2049da-d19e-4386-afbf-fd6b5fddbcba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6825","nonce":1050976194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6827","nonce":1050976194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6831","nonce":1050976194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6829","nonce":1050976194}]},"public_addr":"172.21.15.119:6825/1050976194","cluster_addr":"172.21.15.119:6827/1050976194","heartbeat_back_addr":"172.21.15.119:6831/1050976194","heartbeat_front_addr":"172.21.15.119:6829/1050976194","state":["exists","up"]},{"osd":11,"uuid":"3cccc0f9-d273-4b6b-9b8e-c926371472dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6825","nonce":2915080368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6827","nonce":2915080368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6831","nonce":2915080368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6829","nonce":2915080368}]},"public_addr":"172.21.15.178:6825/2915080368","cluster_addr":"172.21.15.178:6827/2915080368","heartbeat_back_addr":"172.21.15.178:6831/2915080368","heartbeat_front_addr":"172.21.15.178:6829/2915080368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:27.876274+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:53.094706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:16.598813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:48.160419+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:14.048925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:38.814304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:12.542328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:38.942747+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:04.488963+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:38.196053+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:05.254728+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:30.874915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.117:0/185684447":"2024-03-20T03:22:38.134578+0000","172.21.15.117:0/361367101":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/2788067603":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3506200675":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6800/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3283471541":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3750920169":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/2780813298":"2024-03-20T03:22:03.905461+0000","172.21.15.117:6800/2788067603":"2024-03-20T03:22:38.134578+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:29:41.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.461+0000 7efd86200700 1 -- 172.21.15.117:0/3963531917 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7efd640a2890 msgr2=0x7efd640a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:41.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.462+0000 7efd86200700 1 --2- 172.21.15.117:0/3963531917 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7efd640a2890 0x7efd640a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7efd80102030 tx=0x7efd68009260 comp rx=0 tx=0).stop 2024-03-19T03:29:41.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.462+0000 7efd86200700 1 -- 172.21.15.117:0/3963531917 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efd801025c0 msgr2=0x7efd80074610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:41.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.462+0000 7efd86200700 1 --2- 172.21.15.117:0/3963531917 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efd801025c0 0x7efd80074610 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efd70009fd0 tx=0x7efd700094b0 comp rx=0 tx=0).stop 2024-03-19T03:29:41.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.462+0000 7efd86200700 1 -- 172.21.15.117:0/3963531917 shutdown_connections 2024-03-19T03:29:41.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.462+0000 7efd86200700 1 --2- 172.21.15.117:0/3963531917 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7efd640a2890 0x7efd640a4d50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:41.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.462+0000 7efd86200700 1 --2- 172.21.15.117:0/3963531917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd80107690 0x7efd80075090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:41.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.462+0000 7efd86200700 1 --2- 172.21.15.117:0/3963531917 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efd801025c0 0x7efd80074610 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:41.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.462+0000 7efd86200700 1 --2- 172.21.15.117:0/3963531917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd80102f70 0x7efd80074b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:41.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.462+0000 7efd86200700 1 -- 172.21.15.117:0/3963531917 >> 172.21.15.117:0/3963531917 conn(0x7efd80079470 msgr2=0x7efd80100720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:29:41.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.462+0000 7efd86200700 1 -- 172.21.15.117:0/3963531917 shutdown_connections 2024-03-19T03:29:41.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:41.462+0000 7efd86200700 1 -- 172.21.15.117:0/3963531917 wait complete. 2024-03-19T03:29:41.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:41 smithi178 ceph-mon[67578]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:41.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:41 smithi119 ceph-mon[66861]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:41.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:41 smithi117 ceph-mon[31838]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:42.316 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T03:25:18.824270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T03:29:42.317 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-19T03:29:42.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:42 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3963531917' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:29:42.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:42 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3963531917' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:29:42.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:42 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3963531917' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:29:42.750 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:29:43.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:43 smithi117 ceph-mon[31838]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:43.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:43 smithi178 ceph-mon[67578]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:43.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:43 smithi119 ceph-mon[66861]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:44.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.868+0000 7fc099600700 1 -- 172.21.15.117:0/2341290972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc094108920 msgr2=0x7fc094108da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:44.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.868+0000 7fc099600700 1 --2- 172.21.15.117:0/2341290972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc094108920 0x7fc094108da0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc0800098f0 tx=0x7fc080009c00 comp rx=0 tx=0).stop 2024-03-19T03:29:44.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.868+0000 7fc099600700 1 -- 172.21.15.117:0/2341290972 shutdown_connections 2024-03-19T03:29:44.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.868+0000 7fc099600700 1 --2- 172.21.15.117:0/2341290972 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0941092e0 0x7fc0941157f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:44.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.868+0000 7fc099600700 1 --2- 172.21.15.117:0/2341290972 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc09410e920 0x7fc09410ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:44.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.868+0000 7fc099600700 1 --2- 172.21.15.117:0/2341290972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc094108920 0x7fc094108da0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:44.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.868+0000 7fc099600700 1 -- 172.21.15.117:0/2341290972 >> 172.21.15.117:0/2341290972 conn(0x7fc094075ae0 msgr2=0x7fc094077f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:29:44.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.868+0000 7fc099600700 1 -- 172.21.15.117:0/2341290972 shutdown_connections 2024-03-19T03:29:44.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.868+0000 7fc099600700 1 -- 172.21.15.117:0/2341290972 wait complete. 2024-03-19T03:29:44.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.869+0000 7fc099600700 1 Processor -- start 2024-03-19T03:29:44.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.870+0000 7fc099600700 1 -- start start 2024-03-19T03:29:44.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.870+0000 7fc099600700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc094108920 0x7fc09419f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:44.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.870+0000 7fc099600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0941092e0 0x7fc09419f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:44.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.870+0000 7fc099600700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc09410e920 0x7fc0941a39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:44.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.870+0000 7fc099600700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc094118cd0 con 0x7fc09410e920 2024-03-19T03:29:44.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.870+0000 7fc099600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc094118b50 con 0x7fc0941092e0 2024-03-19T03:29:44.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.870+0000 7fc099600700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fc094118e50 con 0x7fc094108920 2024-03-19T03:29:44.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.870+0000 7fc092a00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc094108920 0x7fc09419f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:44.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.870+0000 7fc093400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc09410e920 0x7fc0941a39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:44.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.870+0000 7fc092a00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc094108920 0x7fc09419f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:51962/0 (socket says 172.21.15.117:51962) 2024-03-19T03:29:44.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.870+0000 7fc092a00700 1 -- 172.21.15.117:0/3590395120 learned_addr learned my addr 172.21.15.117:0/3590395120 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:29:44.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.870+0000 7fc092000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0941092e0 0x7fc09419f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:44.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.870+0000 7fc093400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc09410e920 0x7fc0941a39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40406/0 (socket says 172.21.15.117:40406) 2024-03-19T03:29:44.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.871+0000 7fc093400700 1 -- 172.21.15.117:0/3590395120 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc094108920 msgr2=0x7fc09419f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:44.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.871+0000 7fc093400700 1 --2- 172.21.15.117:0/3590395120 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc094108920 0x7fc09419f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:44.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.871+0000 7fc093400700 1 -- 172.21.15.117:0/3590395120 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0941092e0 msgr2=0x7fc09419f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:44.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.871+0000 7fc093400700 1 --2- 172.21.15.117:0/3590395120 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0941092e0 0x7fc09419f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:44.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.871+0000 7fc093400700 1 -- 172.21.15.117:0/3590395120 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0941a4130 con 0x7fc09410e920 2024-03-19T03:29:44.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.871+0000 7fc092000700 1 --2- 172.21.15.117:0/3590395120 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0941092e0 0x7fc09419f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:29:44.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.871+0000 7fc093400700 1 --2- 172.21.15.117:0/3590395120 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc09410e920 0x7fc0941a39f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc08c00efc0 tx=0x7fc08c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:29:44.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.871+0000 7fc08b400700 1 -- 172.21.15.117:0/3590395120 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc08c0092e0 con 0x7fc09410e920 2024-03-19T03:29:44.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.872+0000 7fc099600700 1 -- 172.21.15.117:0/3590395120 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0941a43c0 con 0x7fc09410e920 2024-03-19T03:29:44.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.872+0000 7fc08b400700 1 -- 172.21.15.117:0/3590395120 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc08c004bb0 con 0x7fc09410e920 2024-03-19T03:29:44.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.872+0000 7fc08b400700 1 -- 172.21.15.117:0/3590395120 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc08c004280 con 0x7fc09410e920 2024-03-19T03:29:44.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.872+0000 7fc099600700 1 -- 172.21.15.117:0/3590395120 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0941abca0 con 0x7fc09410e920 2024-03-19T03:29:44.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.873+0000 7fc099600700 1 -- 172.21.15.117:0/3590395120 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc094068fd0 con 0x7fc09410e920 2024-03-19T03:29:44.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.875+0000 7fc08b400700 1 -- 172.21.15.117:0/3590395120 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fc08c01b030 con 0x7fc09410e920 2024-03-19T03:29:44.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.876+0000 7fc08b400700 1 --2- 172.21.15.117:0/3590395120 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc0780a2890 0x7fc0780a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:44.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.877+0000 7fc08b400700 1 -- 172.21.15.117:0/3590395120 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc08c0ceb20 con 0x7fc09410e920 2024-03-19T03:29:44.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.877+0000 7fc092a00700 1 --2- 172.21.15.117:0/3590395120 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc0780a2890 0x7fc0780a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:44.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.877+0000 7fc092a00700 1 --2- 172.21.15.117:0/3590395120 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc0780a2890 0x7fc0780a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc084009750 tx=0x7fc084009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:29:44.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:44.880+0000 7fc08b400700 1 -- 172.21.15.117:0/3590395120 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc08c022020 con 0x7fc09410e920 2024-03-19T03:29:45.101 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:45.100+0000 7fc099600700 1 -- 172.21.15.117:0/3590395120 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc09402dbb0 con 0x7fc09410e920 2024-03-19T03:29:45.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:45.101+0000 7fc08b400700 1 -- 172.21.15.117:0/3590395120 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fc08c010040 con 0x7fc09410e920 2024-03-19T03:29:45.102 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 1 2024-03-19T03:29:45.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:45.105+0000 7fc099600700 1 -- 172.21.15.117:0/3590395120 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc0780a2890 msgr2=0x7fc0780a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:45.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:45.105+0000 7fc099600700 1 --2- 172.21.15.117:0/3590395120 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc0780a2890 0x7fc0780a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc084009750 tx=0x7fc084009310 comp rx=0 tx=0).stop 2024-03-19T03:29:45.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:45.105+0000 7fc099600700 1 -- 172.21.15.117:0/3590395120 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc09410e920 msgr2=0x7fc0941a39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:45.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:45.105+0000 7fc099600700 1 --2- 172.21.15.117:0/3590395120 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc09410e920 0x7fc0941a39f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc08c00efc0 tx=0x7fc08c00c3a0 comp rx=0 tx=0).stop 2024-03-19T03:29:45.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:45.105+0000 7fc099600700 1 -- 172.21.15.117:0/3590395120 shutdown_connections 2024-03-19T03:29:45.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:45.105+0000 7fc099600700 1 --2- 172.21.15.117:0/3590395120 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc0780a2890 0x7fc0780a4d50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:45.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:45.105+0000 7fc099600700 1 --2- 172.21.15.117:0/3590395120 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc09410e920 0x7fc0941a39f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:45.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:45.105+0000 7fc099600700 1 --2- 172.21.15.117:0/3590395120 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc094108920 0x7fc09419f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:45.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:45.105+0000 7fc099600700 1 --2- 172.21.15.117:0/3590395120 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0941092e0 0x7fc09419f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:45.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:45.105+0000 7fc099600700 1 -- 172.21.15.117:0/3590395120 >> 172.21.15.117:0/3590395120 conn(0x7fc094075ae0 msgr2=0x7fc094076020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:29:45.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:45.105+0000 7fc099600700 1 -- 172.21.15.117:0/3590395120 shutdown_connections 2024-03-19T03:29:45.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:45.105+0000 7fc099600700 1 -- 172.21.15.117:0/3590395120 wait complete. 2024-03-19T03:29:45.524 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:45 smithi117 ceph-mon[31838]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:45.524 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:45 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3590395120' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:29:45.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:45 smithi178 ceph-mon[67578]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:45.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:45 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3590395120' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:29:45.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:45 smithi119 ceph-mon[66861]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:45.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:45 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3590395120' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:29:45.943 INFO:tasks.cephadm:Adding mds.a on smithi117 2024-03-19T03:29:45.943 INFO:tasks.cephadm:Adding mds.d on smithi117 2024-03-19T03:29:45.943 INFO:tasks.cephadm:Adding mds.g on smithi117 2024-03-19T03:29:45.943 INFO:tasks.cephadm:Adding mds.j on smithi117 2024-03-19T03:29:45.943 INFO:tasks.cephadm:Adding mds.b on smithi119 2024-03-19T03:29:45.943 INFO:tasks.cephadm:Adding mds.e on smithi119 2024-03-19T03:29:45.943 INFO:tasks.cephadm:Adding mds.h on smithi119 2024-03-19T03:29:45.943 INFO:tasks.cephadm:Adding mds.k on smithi119 2024-03-19T03:29:45.943 INFO:tasks.cephadm:Adding mds.c on smithi178 2024-03-19T03:29:45.943 INFO:tasks.cephadm:Adding mds.f on smithi178 2024-03-19T03:29:45.943 INFO:tasks.cephadm:Adding mds.i on smithi178 2024-03-19T03:29:45.943 INFO:tasks.cephadm:Adding mds.l on smithi178 2024-03-19T03:29:45.944 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph orch apply mds all '12;smithi117=a;smithi117=d;smithi117=g;smithi117=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l' 2024-03-19T03:29:46.191 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T03:29:47.593 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:47 smithi178 ceph-mon[67578]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:47.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:47 smithi119 ceph-mon[66861]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:47.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:47 smithi117 ceph-mon[31838]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:47.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.832+0000 7fd4b413b700 1 -- 172.21.15.178:0/1008498612 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4ac102440 msgr2=0x7fd4ac1028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:47.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.832+0000 7fd4b413b700 1 --2- 172.21.15.178:0/1008498612 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4ac102440 0x7fd4ac1028a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd49c009990 tx=0x7fd49c009ca0 comp rx=0 tx=0).stop 2024-03-19T03:29:47.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.832+0000 7fd4b413b700 1 -- 172.21.15.178:0/1008498612 shutdown_connections 2024-03-19T03:29:47.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.832+0000 7fd4b413b700 1 --2- 172.21.15.178:0/1008498612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4ac102de0 0x7fd4ac109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:47.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.832+0000 7fd4b413b700 1 --2- 172.21.15.178:0/1008498612 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4ac101240 0x7fd4ac101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:47.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.832+0000 7fd4b413b700 1 --2- 172.21.15.178:0/1008498612 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4ac102440 0x7fd4ac1028a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:47.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.832+0000 7fd4b413b700 1 -- 172.21.15.178:0/1008498612 >> 172.21.15.178:0/1008498612 conn(0x7fd4ac0fc800 msgr2=0x7fd4ac0fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:29:47.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.832+0000 7fd4b413b700 1 -- 172.21.15.178:0/1008498612 shutdown_connections 2024-03-19T03:29:47.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.832+0000 7fd4b413b700 1 -- 172.21.15.178:0/1008498612 wait complete. 2024-03-19T03:29:47.834 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.832+0000 7fd4b413b700 1 Processor -- start 2024-03-19T03:29:47.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.832+0000 7fd4b413b700 1 -- start start 2024-03-19T03:29:47.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.832+0000 7fd4b413b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4ac101240 0x7fd4ac06b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:47.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.832+0000 7fd4b413b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4ac102440 0x7fd4ac19d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:47.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.832+0000 7fd4b413b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4ac102de0 0x7fd4ac19f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:47.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.832+0000 7fd4b413b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ac10ceb0 con 0x7fd4ac102440 2024-03-19T03:29:47.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.832+0000 7fd4b413b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ac10cd30 con 0x7fd4ac102de0 2024-03-19T03:29:47.835 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.832+0000 7fd4b413b700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ac10d030 con 0x7fd4ac101240 2024-03-19T03:29:47.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.833+0000 7fd4b1ed7700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4ac101240 0x7fd4ac06b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:47.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.833+0000 7fd4b1ed7700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4ac101240 0x7fd4ac06b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:36704/0 (socket says 172.21.15.178:36704) 2024-03-19T03:29:47.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.833+0000 7fd4b1ed7700 1 -- 172.21.15.178:0/3130469751 learned_addr learned my addr 172.21.15.178:0/3130469751 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-19T03:29:47.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.833+0000 7fd4b26d8700 1 --2- 172.21.15.178:0/3130469751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4ac102de0 0x7fd4ac19f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:47.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.833+0000 7fd4b1ed7700 1 -- 172.21.15.178:0/3130469751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4ac102de0 msgr2=0x7fd4ac19f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:47.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.833+0000 7fd4b1ed7700 1 --2- 172.21.15.178:0/3130469751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4ac102de0 0x7fd4ac19f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:47.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.833+0000 7fd4b1ed7700 1 -- 172.21.15.178:0/3130469751 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4ac102440 msgr2=0x7fd4ac19d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:47.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.833+0000 7fd4b1ed7700 1 --2- 172.21.15.178:0/3130469751 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4ac102440 0x7fd4ac19d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:47.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.833+0000 7fd4b1ed7700 1 -- 172.21.15.178:0/3130469751 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4ac1a01f0 con 0x7fd4ac101240 2024-03-19T03:29:47.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.833+0000 7fd4b26d8700 1 --2- 172.21.15.178:0/3130469751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4ac102de0 0x7fd4ac19f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T03:29:47.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.833+0000 7fd4b1ed7700 1 --2- 172.21.15.178:0/3130469751 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4ac101240 0x7fd4ac06b5b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd4a800d980 tx=0x7fd4a800dd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:29:47.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.833+0000 7fd4a2ffd700 1 -- 172.21.15.178:0/3130469751 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a8017070 con 0x7fd4ac101240 2024-03-19T03:29:47.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.833+0000 7fd4a2ffd700 1 -- 172.21.15.178:0/3130469751 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4a8009b40 con 0x7fd4ac101240 2024-03-19T03:29:47.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.833+0000 7fd4b413b700 1 -- 172.21.15.178:0/3130469751 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4ac1a04e0 con 0x7fd4ac101240 2024-03-19T03:29:47.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.833+0000 7fd4a2ffd700 1 -- 172.21.15.178:0/3130469751 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a800f620 con 0x7fd4ac101240 2024-03-19T03:29:47.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.833+0000 7fd4b413b700 1 -- 172.21.15.178:0/3130469751 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4ac1a0aa0 con 0x7fd4ac101240 2024-03-19T03:29:47.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.834+0000 7fd4b413b700 1 -- 172.21.15.178:0/3130469751 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4ac068fd0 con 0x7fd4ac101240 2024-03-19T03:29:47.838 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.835+0000 7fd4a2ffd700 1 -- 172.21.15.178:0/3130469751 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fd4a8010460 con 0x7fd4ac101240 2024-03-19T03:29:47.838 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.836+0000 7fd4a2ffd700 1 --2- 172.21.15.178:0/3130469751 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd4980a2910 0x7fd4980a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:47.838 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.836+0000 7fd4a2ffd700 1 -- 172.21.15.178:0/3130469751 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd4a80ce230 con 0x7fd4ac101240 2024-03-19T03:29:47.838 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.837+0000 7fd4b16d6700 1 --2- 172.21.15.178:0/3130469751 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd4980a2910 0x7fd4980a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:47.838 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.837+0000 7fd4b16d6700 1 --2- 172.21.15.178:0/3130469751 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd4980a2910 0x7fd4980a4dd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd49c0098e0 tx=0x7fd49c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:29:47.841 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:47.841+0000 7fd4a2ffd700 1 -- 172.21.15.178:0/3130469751 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd4a809b770 con 0x7fd4ac101240 2024-03-19T03:29:48.049 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:48.048+0000 7fd4b413b700 1 -- 172.21.15.178:0/3130469751 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi117=a;smithi117=d;smithi117=g;smithi117=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l", "target": ["mon-mgr", ""]}) -- 0x7fd4ac062a10 con 0x7fd4980a2910 2024-03-19T03:29:48.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:48.065+0000 7fd4a2ffd700 1 -- 172.21.15.178:0/3130469751 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fd4ac062a10 con 0x7fd4980a2910 2024-03-19T03:29:48.066 INFO:teuthology.orchestra.run.smithi178.stdout:Scheduled mds.all update... 2024-03-19T03:29:48.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:48.067+0000 7fd4b413b700 1 -- 172.21.15.178:0/3130469751 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd4980a2910 msgr2=0x7fd4980a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:48.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:48.067+0000 7fd4b413b700 1 --2- 172.21.15.178:0/3130469751 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd4980a2910 0x7fd4980a4dd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd49c0098e0 tx=0x7fd49c005fb0 comp rx=0 tx=0).stop 2024-03-19T03:29:48.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:48.067+0000 7fd4b413b700 1 -- 172.21.15.178:0/3130469751 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4ac101240 msgr2=0x7fd4ac06b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:48.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:48.067+0000 7fd4b413b700 1 --2- 172.21.15.178:0/3130469751 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4ac101240 0x7fd4ac06b5b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd4a800d980 tx=0x7fd4a800dd40 comp rx=0 tx=0).stop 2024-03-19T03:29:48.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:48.067+0000 7fd4b413b700 1 -- 172.21.15.178:0/3130469751 shutdown_connections 2024-03-19T03:29:48.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:48.067+0000 7fd4b413b700 1 --2- 172.21.15.178:0/3130469751 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd4980a2910 0x7fd4980a4dd0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:48.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:48.068+0000 7fd4b413b700 1 --2- 172.21.15.178:0/3130469751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4ac102de0 0x7fd4ac19f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:48.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:48.068+0000 7fd4b413b700 1 --2- 172.21.15.178:0/3130469751 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4ac101240 0x7fd4ac06b5b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:48.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:48.068+0000 7fd4b413b700 1 --2- 172.21.15.178:0/3130469751 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4ac102440 0x7fd4ac19d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:48.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:48.068+0000 7fd4b413b700 1 -- 172.21.15.178:0/3130469751 >> 172.21.15.178:0/3130469751 conn(0x7fd4ac0fc800 msgr2=0x7fd4ac105740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:29:48.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:48.068+0000 7fd4b413b700 1 -- 172.21.15.178:0/3130469751 shutdown_connections 2024-03-19T03:29:48.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-19T03:29:48.068+0000 7fd4b413b700 1 -- 172.21.15.178:0/3130469751 wait complete. 2024-03-19T03:29:48.735 DEBUG:teuthology.orchestra.run.smithi117:mds.a> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.a.service 2024-03-19T03:29:48.746 DEBUG:teuthology.orchestra.run.smithi117:mds.d> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.d.service 2024-03-19T03:29:48.793 DEBUG:teuthology.orchestra.run.smithi117:mds.g> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.g.service 2024-03-19T03:29:48.840 DEBUG:teuthology.orchestra.run.smithi117:mds.j> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.j.service 2024-03-19T03:29:48.886 DEBUG:teuthology.orchestra.run.smithi119:mds.b> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.b.service 2024-03-19T03:29:48.888 DEBUG:teuthology.orchestra.run.smithi119:mds.e> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.e.service 2024-03-19T03:29:48.933 DEBUG:teuthology.orchestra.run.smithi119:mds.h> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.h.service 2024-03-19T03:29:48.975 DEBUG:teuthology.orchestra.run.smithi119:mds.k> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.k.service 2024-03-19T03:29:49.018 DEBUG:teuthology.orchestra.run.smithi178:mds.c> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.c.service 2024-03-19T03:29:49.020 DEBUG:teuthology.orchestra.run.smithi178:mds.f> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.f.service 2024-03-19T03:29:49.063 DEBUG:teuthology.orchestra.run.smithi178:mds.i> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.i.service 2024-03-19T03:29:49.106 DEBUG:teuthology.orchestra.run.smithi178:mds.l> sudo journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.l.service 2024-03-19T03:29:49.149 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-19T03:29:49.150 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:29:49.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:49 smithi119 ceph-mon[66861]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi117=a;smithi117=d;smithi117=g;smithi117=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:29:49.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:49 smithi119 ceph-mon[66861]: Saving service mds.all spec with placement smithi117=a;smithi117=d;smithi117=g;smithi117=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l;count:12 2024-03-19T03:29:49.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:49 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:49.236 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:49 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:29:49.236 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:49 smithi119 ceph-mon[66861]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:49.332 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:49 smithi178 ceph-mon[67578]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi117=a;smithi117=d;smithi117=g;smithi117=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:29:49.333 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:49 smithi178 ceph-mon[67578]: Saving service mds.all spec with placement smithi117=a;smithi117=d;smithi117=g;smithi117=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l;count:12 2024-03-19T03:29:49.333 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:49 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:49.333 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:49 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:29:49.333 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:49 smithi178 ceph-mon[67578]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:49.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.371+0000 7fc192a00640 1 -- 172.21.15.117:0/3738217882 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc18c104a60 msgr2=0x7fc18c104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:49.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.371+0000 7fc192a00640 1 --2- 172.21.15.117:0/3738217882 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc18c104a60 0x7fc18c104ee0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc178009900 tx=0x7fc17802bc50 comp rx=0 tx=0).stop 2024-03-19T03:29:49.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.371+0000 7fc192a00640 1 -- 172.21.15.117:0/3738217882 shutdown_connections 2024-03-19T03:29:49.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.371+0000 7fc192a00640 1 --2- 172.21.15.117:0/3738217882 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc18c105420 0x7fc18c10bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:49.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.371+0000 7fc192a00640 1 --2- 172.21.15.117:0/3738217882 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc18c104a60 0x7fc18c104ee0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:49.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.371+0000 7fc192a00640 1 --2- 172.21.15.117:0/3738217882 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc18c103870 0x7fc18c103c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:49.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.371+0000 7fc192a00640 1 -- 172.21.15.117:0/3738217882 >> 172.21.15.117:0/3738217882 conn(0x7fc18c0fee00 msgr2=0x7fc18c101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:29:49.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.371+0000 7fc192a00640 1 -- 172.21.15.117:0/3738217882 shutdown_connections 2024-03-19T03:29:49.373 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.371+0000 7fc192a00640 1 -- 172.21.15.117:0/3738217882 wait complete. 2024-03-19T03:29:49.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.372+0000 7fc192a00640 1 Processor -- start 2024-03-19T03:29:49.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.372+0000 7fc192a00640 1 -- start start 2024-03-19T03:29:49.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.372+0000 7fc192a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc18c103870 0x7fc18c199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:49.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.372+0000 7fc192a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc18c104a60 0x7fc18c19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:49.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.372+0000 7fc192a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc18c105420 0x7fc18c1a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:49.375 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.372+0000 7fc192a00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc18c10f4f0 con 0x7fc18c104a60 2024-03-19T03:29:49.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.372+0000 7fc192a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc18c10f370 con 0x7fc18c105420 2024-03-19T03:29:49.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.372+0000 7fc192a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fc18c10f670 con 0x7fc18c103870 2024-03-19T03:29:49.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.373+0000 7fc191600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc18c103870 0x7fc18c199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:49.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.373+0000 7fc191600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc18c103870 0x7fc18c199c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:46762/0 (socket says 172.21.15.117:46762) 2024-03-19T03:29:49.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.373+0000 7fc191600640 1 -- 172.21.15.117:0/207405770 learned_addr learned my addr 172.21.15.117:0/207405770 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:29:49.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.373+0000 7fc190c00640 1 --2- 172.21.15.117:0/207405770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc18c104a60 0x7fc18c19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:49.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.373+0000 7fc192000640 1 --2- 172.21.15.117:0/207405770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc18c105420 0x7fc18c1a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:49.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.373+0000 7fc191600640 1 -- 172.21.15.117:0/207405770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc18c105420 msgr2=0x7fc18c1a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:49.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.373+0000 7fc191600640 1 --2- 172.21.15.117:0/207405770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc18c105420 0x7fc18c1a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:49.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.373+0000 7fc191600640 1 -- 172.21.15.117:0/207405770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc18c104a60 msgr2=0x7fc18c19a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:49.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.373+0000 7fc191600640 1 --2- 172.21.15.117:0/207405770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc18c104a60 0x7fc18c19a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:49.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.373+0000 7fc191600640 1 -- 172.21.15.117:0/207405770 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc18c1a17e0 con 0x7fc18c103870 2024-03-19T03:29:49.376 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.373+0000 7fc192000640 1 --2- 172.21.15.117:0/207405770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc18c105420 0x7fc18c1a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:29:49.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.373+0000 7fc191600640 1 --2- 172.21.15.117:0/207405770 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc18c103870 0x7fc18c199c60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc18000e9c0 tx=0x7fc18000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:29:49.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.374+0000 7fc18a000640 1 -- 172.21.15.117:0/207405770 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18000cd80 con 0x7fc18c103870 2024-03-19T03:29:49.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.374+0000 7fc18a000640 1 -- 172.21.15.117:0/207405770 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1800035f0 con 0x7fc18c103870 2024-03-19T03:29:49.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.374+0000 7fc192a00640 1 -- 172.21.15.117:0/207405770 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc18c1a1ad0 con 0x7fc18c103870 2024-03-19T03:29:49.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.374+0000 7fc18a000640 1 -- 172.21.15.117:0/207405770 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc180010690 con 0x7fc18c103870 2024-03-19T03:29:49.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.374+0000 7fc192a00640 1 -- 172.21.15.117:0/207405770 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc18c1a1ff0 con 0x7fc18c103870 2024-03-19T03:29:49.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.375+0000 7fc192a00640 1 -- 172.21.15.117:0/207405770 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc18c10ab90 con 0x7fc18c103870 2024-03-19T03:29:49.378 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.377+0000 7fc18a000640 1 -- 172.21.15.117:0/207405770 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fc1800037a0 con 0x7fc18c103870 2024-03-19T03:29:49.379 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.378+0000 7fc18a000640 1 --2- 172.21.15.117:0/207405770 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc1740a2880 0x7fc1740a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:49.379 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.379+0000 7fc18a000640 1 -- 172.21.15.117:0/207405770 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc1800ce9c0 con 0x7fc18c103870 2024-03-19T03:29:49.379 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.379+0000 7fc190c00640 1 --2- 172.21.15.117:0/207405770 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc1740a2880 0x7fc1740a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:49.380 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.379+0000 7fc190c00640 1 --2- 172.21.15.117:0/207405770 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc1740a2880 0x7fc1740a4d40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc17802c7d0 tx=0x7fc178005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:29:49.381 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.380+0000 7fc18a000640 1 -- 172.21.15.117:0/207405770 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc18009be60 con 0x7fc18c103870 2024-03-19T03:29:49.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:49 smithi117 ceph-mon[31838]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi117=a;smithi117=d;smithi117=g;smithi117=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:29:49.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:49 smithi117 ceph-mon[31838]: Saving service mds.all spec with placement smithi117=a;smithi117=d;smithi117=g;smithi117=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l;count:12 2024-03-19T03:29:49.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:49 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:49.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:49 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:29:49.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:49 smithi117 ceph-mon[31838]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:49.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.582+0000 7fc192a00640 1 -- 172.21.15.117:0/207405770 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc18c062420 con 0x7fc18c103870 2024-03-19T03:29:49.585 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.584+0000 7fc18a000640 1 -- 172.21.15.117:0/207405770 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18538 (secure 0 0 0) 0x7fc18009b760 con 0x7fc18c103870 2024-03-19T03:29:49.585 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:29:49.585 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":81,"fsid":"a3a82456-e59f-11ee-95c9-87774f69a715","created":"2024-03-19T03:21:30.427136+0000","modified":"2024-03-19T03:29:34.539579+0000","last_up_change":"2024-03-19T03:29:32.967505+0000","last_in_change":"2024-03-19T03:29:12.848740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:25:18.824270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b32eb0ee-9d4b-4748-87ec-eda02c907415","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6803","nonce":3014934900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6805","nonce":3014934900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6809","nonce":3014934900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6807","nonce":3014934900}]},"public_addr":"172.21.15.117:6803/3014934900","cluster_addr":"172.21.15.117:6805/3014934900","heartbeat_back_addr":"172.21.15.117:6809/3014934900","heartbeat_front_addr":"172.21.15.117:6807/3014934900","state":["exists","up"]},{"osd":1,"uuid":"e07585be-c016-4466-b07c-abbc22a58de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6801","nonce":2425927341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6803","nonce":2425927341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6807","nonce":2425927341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6805","nonce":2425927341}]},"public_addr":"172.21.15.119:6801/2425927341","cluster_addr":"172.21.15.119:6803/2425927341","heartbeat_back_addr":"172.21.15.119:6807/2425927341","heartbeat_front_addr":"172.21.15.119:6805/2425927341","state":["exists","up"]},{"osd":2,"uuid":"ede9edda-3b9f-41ca-ad04-5dcb738ab757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6801","nonce":3733812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6803","nonce":3733812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6807","nonce":3733812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6805","nonce":3733812989}]},"public_addr":"172.21.15.178:6801/3733812989","cluster_addr":"172.21.15.178:6803/3733812989","heartbeat_back_addr":"172.21.15.178:6807/3733812989","heartbeat_front_addr":"172.21.15.178:6805/3733812989","state":["exists","up"]},{"osd":3,"uuid":"f81c41e1-3a69-40db-a90f-58151286b081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6811","nonce":2178310135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6813","nonce":2178310135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6817","nonce":2178310135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6815","nonce":2178310135}]},"public_addr":"172.21.15.117:6811/2178310135","cluster_addr":"172.21.15.117:6813/2178310135","heartbeat_back_addr":"172.21.15.117:6817/2178310135","heartbeat_front_addr":"172.21.15.117:6815/2178310135","state":["exists","up"]},{"osd":4,"uuid":"caf7ce86-caf0-4eaf-bbab-bd2d93e63c33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6809","nonce":3455398086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6811","nonce":3455398086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6815","nonce":3455398086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6813","nonce":3455398086}]},"public_addr":"172.21.15.119:6809/3455398086","cluster_addr":"172.21.15.119:6811/3455398086","heartbeat_back_addr":"172.21.15.119:6815/3455398086","heartbeat_front_addr":"172.21.15.119:6813/3455398086","state":["exists","up"]},{"osd":5,"uuid":"20ba0b8a-2e0f-4ea7-b435-430e44766200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6809","nonce":1259631912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6811","nonce":1259631912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6815","nonce":1259631912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6813","nonce":1259631912}]},"public_addr":"172.21.15.178:6809/1259631912","cluster_addr":"172.21.15.178:6811/1259631912","heartbeat_back_addr":"172.21.15.178:6815/1259631912","heartbeat_front_addr":"172.21.15.178:6813/1259631912","state":["exists","up"]},{"osd":6,"uuid":"75c2ddd7-f837-42e0-ac47-8cfbcf97a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6819","nonce":864146757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6821","nonce":864146757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6825","nonce":864146757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6823","nonce":864146757}]},"public_addr":"172.21.15.117:6819/864146757","cluster_addr":"172.21.15.117:6821/864146757","heartbeat_back_addr":"172.21.15.117:6825/864146757","heartbeat_front_addr":"172.21.15.117:6823/864146757","state":["exists","up"]},{"osd":7,"uuid":"929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6817","nonce":1273365545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6819","nonce":1273365545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6823","nonce":1273365545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6821","nonce":1273365545}]},"public_addr":"172.21.15.119:6817/1273365545","cluster_addr":"172.21.15.119:6819/1273365545","heartbeat_back_addr":"172.21.15.119:6823/1273365545","heartbeat_front_addr":"172.21.15.119:6821/1273365545","state":["exists","up"]},{"osd":8,"uuid":"fd371461-a538-4781-92e1-5eead4ca3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6817","nonce":1059587501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6819","nonce":1059587501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6823","nonce":1059587501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6821","nonce":1059587501}]},"public_addr":"172.21.15.178:6817/1059587501","cluster_addr":"172.21.15.178:6819/1059587501","heartbeat_back_addr":"172.21.15.178:6823/1059587501","heartbeat_front_addr":"172.21.15.178:6821/1059587501","state":["exists","up"]},{"osd":9,"uuid":"0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6827","nonce":3653875600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6829","nonce":3653875600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6833","nonce":3653875600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6831","nonce":3653875600}]},"public_addr":"172.21.15.117:6827/3653875600","cluster_addr":"172.21.15.117:6829/3653875600","heartbeat_back_addr":"172.21.15.117:6833/3653875600","heartbeat_front_addr":"172.21.15.117:6831/3653875600","state":["exists","up"]},{"osd":10,"uuid":"de2049da-d19e-4386-afbf-fd6b5fddbcba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6825","nonce":1050976194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6827","nonce":1050976194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6831","nonce":1050976194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6829","nonce":1050976194}]},"public_addr":"172.21.15.119:6825/1050976194","cluster_addr":"172.21.15.119:6827/1050976194","heartbeat_back_addr":"172.21.15.119:6831/1050976194","heartbeat_front_addr":"172.21.15.119:6829/1050976194","state":["exists","up"]},{"osd":11,"uuid":"3cccc0f9-d273-4b6b-9b8e-c926371472dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6825","nonce":2915080368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6827","nonce":2915080368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6831","nonce":2915080368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6829","nonce":2915080368}]},"public_addr":"172.21.15.178:6825/2915080368","cluster_addr":"172.21.15.178:6827/2915080368","heartbeat_back_addr":"172.21.15.178:6831/2915080368","heartbeat_front_addr":"172.21.15.178:6829/2915080368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:27.876274+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:53.094706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:16.598813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:48.160419+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:14.048925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:38.814304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:12.542328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:38.942747+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:04.488963+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:38.196053+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:05.254728+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:30.874915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.117:0/185684447":"2024-03-20T03:22:38.134578+0000","172.21.15.117:0/361367101":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/2788067603":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3506200675":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6800/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3283471541":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3750920169":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/2780813298":"2024-03-20T03:22:03.905461+0000","172.21.15.117:6800/2788067603":"2024-03-20T03:22:38.134578+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:29:49.587 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.586+0000 7fc192a00640 1 -- 172.21.15.117:0/207405770 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc1740a2880 msgr2=0x7fc1740a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:49.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.586+0000 7fc192a00640 1 --2- 172.21.15.117:0/207405770 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc1740a2880 0x7fc1740a4d40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc17802c7d0 tx=0x7fc178005fd0 comp rx=0 tx=0).stop 2024-03-19T03:29:49.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.586+0000 7fc192a00640 1 -- 172.21.15.117:0/207405770 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc18c103870 msgr2=0x7fc18c199c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:49.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.586+0000 7fc192a00640 1 --2- 172.21.15.117:0/207405770 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc18c103870 0x7fc18c199c60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc18000e9c0 tx=0x7fc18000ee80 comp rx=0 tx=0).stop 2024-03-19T03:29:49.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.586+0000 7fc192a00640 1 -- 172.21.15.117:0/207405770 shutdown_connections 2024-03-19T03:29:49.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.586+0000 7fc192a00640 1 --2- 172.21.15.117:0/207405770 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc1740a2880 0x7fc1740a4d40 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:49.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.586+0000 7fc192a00640 1 --2- 172.21.15.117:0/207405770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc18c105420 0x7fc18c1a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:49.588 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.586+0000 7fc192a00640 1 --2- 172.21.15.117:0/207405770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc18c104a60 0x7fc18c19a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:49.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.586+0000 7fc192a00640 1 --2- 172.21.15.117:0/207405770 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc18c103870 0x7fc18c199c60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:49.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.586+0000 7fc192a00640 1 -- 172.21.15.117:0/207405770 >> 172.21.15.117:0/207405770 conn(0x7fc18c0fee00 msgr2=0x7fc18c107d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:29:49.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.586+0000 7fc192a00640 1 -- 172.21.15.117:0/207405770 shutdown_connections 2024-03-19T03:29:49.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.587+0000 7fc192a00640 1 -- 172.21.15.117:0/207405770 wait complete. 2024-03-19T03:29:49.608 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T03:25:18.824270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T03:29:49.609 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T03:29:49.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.825+0000 7f55e2200640 1 -- 172.21.15.117:0/4235339797 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55dc103a40 msgr2=0x7f55dc103e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:49.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.825+0000 7f55e2200640 1 --2- 172.21.15.117:0/4235339797 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55dc103a40 0x7f55dc103e60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f55cc009900 tx=0x7f55cc02bc40 comp rx=0 tx=0).stop 2024-03-19T03:29:49.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.825+0000 7f55e2200640 1 -- 172.21.15.117:0/4235339797 shutdown_connections 2024-03-19T03:29:49.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.825+0000 7f55e2200640 1 --2- 172.21.15.117:0/4235339797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55dc1055f0 0x7f55dc10be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:49.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.825+0000 7f55e2200640 1 --2- 172.21.15.117:0/4235339797 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55dc104c30 0x7f55dc1050b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:49.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.825+0000 7f55e2200640 1 --2- 172.21.15.117:0/4235339797 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55dc103a40 0x7f55dc103e60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:49.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.825+0000 7f55e2200640 1 -- 172.21.15.117:0/4235339797 >> 172.21.15.117:0/4235339797 conn(0x7f55dc0fefd0 msgr2=0x7f55dc1013f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:29:49.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.825+0000 7f55e2200640 1 -- 172.21.15.117:0/4235339797 shutdown_connections 2024-03-19T03:29:49.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.825+0000 7f55e2200640 1 -- 172.21.15.117:0/4235339797 wait complete. 2024-03-19T03:29:49.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.826+0000 7f55e2200640 1 Processor -- start 2024-03-19T03:29:49.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.826+0000 7f55e2200640 1 -- start start 2024-03-19T03:29:49.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.826+0000 7f55e2200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55dc103a40 0x7f55dc199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:49.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.827+0000 7f55e2200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55dc104c30 0x7f55dc19a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:49.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.827+0000 7f55e2200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55dc1055f0 0x7f55dc1a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:49.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.827+0000 7f55e2200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f55dc10f690 con 0x7f55dc1055f0 2024-03-19T03:29:49.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.827+0000 7f55e2200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f55dc10f510 con 0x7f55dc104c30 2024-03-19T03:29:49.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.827+0000 7f55e2200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f55dc10f810 con 0x7f55dc103a40 2024-03-19T03:29:49.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.827+0000 7f55db400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55dc103a40 0x7f55dc199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:49.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.827+0000 7f55db400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55dc103a40 0x7f55dc199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:46786/0 (socket says 172.21.15.117:46786) 2024-03-19T03:29:49.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.827+0000 7f55db400640 1 -- 172.21.15.117:0/4069933216 learned_addr learned my addr 172.21.15.117:0/4069933216 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:29:49.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.827+0000 7f55d3e00640 1 --2- 172.21.15.117:0/4069933216 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55dc104c30 0x7f55dc19a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:49.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.827+0000 7f55dbe00640 1 --2- 172.21.15.117:0/4069933216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55dc1055f0 0x7f55dc1a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:49.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.827+0000 7f55db400640 1 -- 172.21.15.117:0/4069933216 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55dc104c30 msgr2=0x7f55dc19a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:49.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.827+0000 7f55db400640 1 --2- 172.21.15.117:0/4069933216 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55dc104c30 0x7f55dc19a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:49.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.827+0000 7f55db400640 1 -- 172.21.15.117:0/4069933216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55dc1055f0 msgr2=0x7f55dc1a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:49.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.827+0000 7f55db400640 1 --2- 172.21.15.117:0/4069933216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55dc1055f0 0x7f55dc1a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:49.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.827+0000 7f55db400640 1 -- 172.21.15.117:0/4069933216 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55dc1a19b0 con 0x7f55dc103a40 2024-03-19T03:29:49.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.827+0000 7f55d3e00640 1 --2- 172.21.15.117:0/4069933216 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55dc104c30 0x7f55dc19a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:29:49.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.828+0000 7f55db400640 1 --2- 172.21.15.117:0/4069933216 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55dc103a40 0x7f55dc199da0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f55cc000c00 tx=0x7f55cc02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:29:49.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.828+0000 7f55d8c00640 1 -- 172.21.15.117:0/4069933216 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55cc041070 con 0x7f55dc103a40 2024-03-19T03:29:49.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.828+0000 7f55d8c00640 1 -- 172.21.15.117:0/4069933216 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55cc004450 con 0x7f55dc103a40 2024-03-19T03:29:49.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.828+0000 7f55d8c00640 1 -- 172.21.15.117:0/4069933216 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55cc003e20 con 0x7f55dc103a40 2024-03-19T03:29:49.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.828+0000 7f55e2200640 1 -- 172.21.15.117:0/4069933216 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55dc1a1c40 con 0x7f55dc103a40 2024-03-19T03:29:49.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.828+0000 7f55e2200640 1 -- 172.21.15.117:0/4069933216 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55dc1a20e0 con 0x7f55dc103a40 2024-03-19T03:29:49.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.829+0000 7f55e2200640 1 -- 172.21.15.117:0/4069933216 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55dc10ae30 con 0x7f55dc103a40 2024-03-19T03:29:49.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.831+0000 7f55d8c00640 1 -- 172.21.15.117:0/4069933216 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f55cc044070 con 0x7f55dc103a40 2024-03-19T03:29:49.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.832+0000 7f55d8c00640 1 --2- 172.21.15.117:0/4069933216 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f55c00a2800 0x7f55c00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:49.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.832+0000 7f55d8c00640 1 -- 172.21.15.117:0/4069933216 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f55cc0f01d0 con 0x7f55dc103a40 2024-03-19T03:29:49.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.832+0000 7f55d3e00640 1 --2- 172.21.15.117:0/4069933216 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f55c00a2800 0x7f55c00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:49.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.833+0000 7f55d3e00640 1 --2- 172.21.15.117:0/4069933216 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f55c00a2800 0x7f55c00a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009fc0 tx=0x7f55c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:29:49.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:49.836+0000 7f55d8c00640 1 -- 172.21.15.117:0/4069933216 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f55cc0bd670 con 0x7f55dc103a40 2024-03-19T03:29:50.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.033+0000 7f55e2200640 1 -- 172.21.15.117:0/4069933216 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f55dc062450 con 0x7f55dc103a40 2024-03-19T03:29:50.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.035+0000 7f55d8c00640 1 -- 172.21.15.117:0/4069933216 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f55cc0bcf70 con 0x7f55dc103a40 2024-03-19T03:29:50.035 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 1 2024-03-19T03:29:50.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.037+0000 7f55e2200640 1 -- 172.21.15.117:0/4069933216 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f55c00a2800 msgr2=0x7f55c00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:50.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.037+0000 7f55e2200640 1 --2- 172.21.15.117:0/4069933216 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f55c00a2800 0x7f55c00a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009fc0 tx=0x7f55c4005f90 comp rx=0 tx=0).stop 2024-03-19T03:29:50.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.037+0000 7f55e2200640 1 -- 172.21.15.117:0/4069933216 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55dc103a40 msgr2=0x7f55dc199da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:50.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.037+0000 7f55e2200640 1 --2- 172.21.15.117:0/4069933216 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55dc103a40 0x7f55dc199da0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f55cc000c00 tx=0x7f55cc02c650 comp rx=0 tx=0).stop 2024-03-19T03:29:50.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.037+0000 7f55e2200640 1 -- 172.21.15.117:0/4069933216 shutdown_connections 2024-03-19T03:29:50.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.037+0000 7f55e2200640 1 --2- 172.21.15.117:0/4069933216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55dc1055f0 0x7f55dc1a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:50.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.037+0000 7f55e2200640 1 --2- 172.21.15.117:0/4069933216 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f55c00a2800 0x7f55c00a4cc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:50.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.037+0000 7f55e2200640 1 --2- 172.21.15.117:0/4069933216 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55dc104c30 0x7f55dc19a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:50.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.037+0000 7f55e2200640 1 --2- 172.21.15.117:0/4069933216 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55dc103a40 0x7f55dc199da0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:50.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.037+0000 7f55e2200640 1 -- 172.21.15.117:0/4069933216 >> 172.21.15.117:0/4069933216 conn(0x7f55dc0fefd0 msgr2=0x7f55dc108000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:29:50.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.037+0000 7f55e2200640 1 -- 172.21.15.117:0/4069933216 shutdown_connections 2024-03-19T03:29:50.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.037+0000 7f55e2200640 1 -- 172.21.15.117:0/4069933216 wait complete. 2024-03-19T03:29:50.058 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:29:50.252 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:50 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:50.252 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:50 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:29:50.252 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:50 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:50.252 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:50 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:29:50.253 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:50 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:29:50.253 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:50 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:50.253 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:50 smithi178 ceph-mon[67578]: Deploying daemon mds.l on smithi178 2024-03-19T03:29:50.253 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:50 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/207405770' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:29:50.253 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:50 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/4069933216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:29:50.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.269+0000 7f29cca00640 1 -- 172.21.15.117:0/3023111618 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f29c8101660 msgr2=0x7f29c8101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:50.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.269+0000 7f29cca00640 1 --2- 172.21.15.117:0/3023111618 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f29c8101660 0x7f29c8101a80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009900 tx=0x7f29b802bc50 comp rx=0 tx=0).stop 2024-03-19T03:29:50.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.269+0000 7f29cca00640 1 -- 172.21.15.117:0/3023111618 shutdown_connections 2024-03-19T03:29:50.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.269+0000 7f29cca00640 1 --2- 172.21.15.117:0/3023111618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29c8103210 0x7f29c8109aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:50.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.269+0000 7f29cca00640 1 --2- 172.21.15.117:0/3023111618 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29c8102850 0x7f29c8102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:50.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.269+0000 7f29cca00640 1 --2- 172.21.15.117:0/3023111618 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f29c8101660 0x7f29c8101a80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:50.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.269+0000 7f29cca00640 1 -- 172.21.15.117:0/3023111618 >> 172.21.15.117:0/3023111618 conn(0x7f29c80fcbb0 msgr2=0x7f29c80ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:29:50.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.270+0000 7f29cca00640 1 -- 172.21.15.117:0/3023111618 shutdown_connections 2024-03-19T03:29:50.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.270+0000 7f29cca00640 1 -- 172.21.15.117:0/3023111618 wait complete. 2024-03-19T03:29:50.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.271+0000 7f29cca00640 1 Processor -- start 2024-03-19T03:29:50.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.271+0000 7f29cca00640 1 -- start start 2024-03-19T03:29:50.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.271+0000 7f29cca00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f29c8101660 0x7f29c8197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:50.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.271+0000 7f29cca00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29c8102850 0x7f29c8197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:50.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.271+0000 7f29cca00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29c8103210 0x7f29c819f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:50.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.271+0000 7f29cca00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f29c810d2d0 con 0x7f29c8102850 2024-03-19T03:29:50.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.271+0000 7f29cca00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f29c810d150 con 0x7f29c8103210 2024-03-19T03:29:50.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.271+0000 7f29cca00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f29c810d450 con 0x7f29c8101660 2024-03-19T03:29:50.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.272+0000 7f29bde00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29c8102850 0x7f29c8197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:50.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.272+0000 7f29c7200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29c8103210 0x7f29c819f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:50.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.272+0000 7f29c6800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f29c8101660 0x7f29c8197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:50.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.272+0000 7f29c7200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29c8103210 0x7f29c819f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:38200/0 (socket says 172.21.15.117:38200) 2024-03-19T03:29:50.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.272+0000 7f29c7200640 1 -- 172.21.15.117:0/1228282706 learned_addr learned my addr 172.21.15.117:0/1228282706 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:29:50.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.272+0000 7f29bde00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29c8102850 0x7f29c8197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42052/0 (socket says 172.21.15.117:42052) 2024-03-19T03:29:50.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.272+0000 7f29c7200640 1 -- 172.21.15.117:0/1228282706 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f29c8101660 msgr2=0x7f29c8197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:50.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.272+0000 7f29c7200640 1 --2- 172.21.15.117:0/1228282706 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f29c8101660 0x7f29c8197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:50.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.272+0000 7f29c7200640 1 -- 172.21.15.117:0/1228282706 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29c8102850 msgr2=0x7f29c8197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:50.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.272+0000 7f29c7200640 1 --2- 172.21.15.117:0/1228282706 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29c8102850 0x7f29c8197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:50.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.272+0000 7f29c7200640 1 -- 172.21.15.117:0/1228282706 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c819f670 con 0x7f29c8103210 2024-03-19T03:29:50.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.272+0000 7f29c7200640 1 --2- 172.21.15.117:0/1228282706 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29c8103210 0x7f29c819f000 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f29c000efd0 tx=0x7f29c000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:29:50.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.273+0000 7f29bfe00640 1 -- 172.21.15.117:0/1228282706 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c0003b70 con 0x7f29c8103210 2024-03-19T03:29:50.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.273+0000 7f29bfe00640 1 -- 172.21.15.117:0/1228282706 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29c0003d10 con 0x7f29c8103210 2024-03-19T03:29:50.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.273+0000 7f29bfe00640 1 -- 172.21.15.117:0/1228282706 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c00109f0 con 0x7f29c8103210 2024-03-19T03:29:50.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.273+0000 7f29cca00640 1 -- 172.21.15.117:0/1228282706 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29c819f960 con 0x7f29c8103210 2024-03-19T03:29:50.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.273+0000 7f29cca00640 1 -- 172.21.15.117:0/1228282706 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29c819feb0 con 0x7f29c8103210 2024-03-19T03:29:50.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.274+0000 7f29cca00640 1 -- 172.21.15.117:0/1228282706 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29c8108a00 con 0x7f29c8103210 2024-03-19T03:29:50.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.276+0000 7f29bfe00640 1 -- 172.21.15.117:0/1228282706 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f29c0004020 con 0x7f29c8103210 2024-03-19T03:29:50.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.277+0000 7f29bfe00640 1 --2- 172.21.15.117:0/1228282706 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f29ac0a2880 0x7f29ac0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:50.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.277+0000 7f29bfe00640 1 -- 172.21.15.117:0/1228282706 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f29c00cea50 con 0x7f29c8103210 2024-03-19T03:29:50.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.277+0000 7f29c6800640 1 --2- 172.21.15.117:0/1228282706 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f29ac0a2880 0x7f29ac0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:50.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.278+0000 7f29c6800640 1 --2- 172.21.15.117:0/1228282706 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f29ac0a2880 0x7f29ac0a4d40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f29b8000c00 tx=0x7f29b8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:29:50.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.281+0000 7f29bfe00640 1 -- 172.21.15.117:0/1228282706 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f29c009bef0 con 0x7f29c8103210 2024-03-19T03:29:50.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:50 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:50.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:50 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:29:50.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:50 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:50.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:50 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:29:50.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:50 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:29:50.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:50 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:50.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:50 smithi119 ceph-mon[66861]: Deploying daemon mds.l on smithi178 2024-03-19T03:29:50.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:50 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/207405770' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:29:50.485 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:50 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/4069933216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:29:50.522 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.521+0000 7f29cca00640 1 -- 172.21.15.117:0/1228282706 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29c8062420 con 0x7f29c8103210 2024-03-19T03:29:50.522 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.522+0000 7f29bfe00640 1 -- 172.21.15.117:0/1228282706 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f29c009b7f0 con 0x7f29c8103210 2024-03-19T03:29:50.523 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:29:50.523 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-19T03:29:50.523 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 1 2024-03-19T03:29:50.526 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.525+0000 7f29cca00640 1 -- 172.21.15.117:0/1228282706 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f29ac0a2880 msgr2=0x7f29ac0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:50.526 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.525+0000 7f29cca00640 1 --2- 172.21.15.117:0/1228282706 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f29ac0a2880 0x7f29ac0a4d40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f29b8000c00 tx=0x7f29b8005fd0 comp rx=0 tx=0).stop 2024-03-19T03:29:50.526 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.525+0000 7f29cca00640 1 -- 172.21.15.117:0/1228282706 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29c8103210 msgr2=0x7f29c819f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:50.526 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.525+0000 7f29cca00640 1 --2- 172.21.15.117:0/1228282706 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29c8103210 0x7f29c819f000 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f29c000efd0 tx=0x7f29c000c6a0 comp rx=0 tx=0).stop 2024-03-19T03:29:50.526 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.525+0000 7f29cca00640 1 -- 172.21.15.117:0/1228282706 shutdown_connections 2024-03-19T03:29:50.526 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.525+0000 7f29cca00640 1 --2- 172.21.15.117:0/1228282706 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f29ac0a2880 0x7f29ac0a4d40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:50.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.525+0000 7f29cca00640 1 --2- 172.21.15.117:0/1228282706 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29c8103210 0x7f29c819f000 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:50.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.525+0000 7f29cca00640 1 --2- 172.21.15.117:0/1228282706 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29c8102850 0x7f29c8197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:50.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.525+0000 7f29cca00640 1 --2- 172.21.15.117:0/1228282706 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f29c8101660 0x7f29c8197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:50.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.525+0000 7f29cca00640 1 -- 172.21.15.117:0/1228282706 >> 172.21.15.117:0/1228282706 conn(0x7f29c80fcbb0 msgr2=0x7f29c8105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:29:50.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.525+0000 7f29cca00640 1 -- 172.21.15.117:0/1228282706 shutdown_connections 2024-03-19T03:29:50.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:50.525+0000 7f29cca00640 1 -- 172.21.15.117:0/1228282706 wait complete. 2024-03-19T03:29:50.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:50 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:50.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:50 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:29:50.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:50 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:50.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:50 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:29:50.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:50 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:29:50.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:50 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:50.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:50 smithi117 ceph-mon[31838]: Deploying daemon mds.l on smithi178 2024-03-19T03:29:50.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:50 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/207405770' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:29:50.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:50 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/4069933216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:29:51.205 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:51 smithi178 ceph-mon[67578]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:51.205 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:51 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1228282706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:29:51.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:51 smithi119 ceph-mon[66861]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:51.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:51 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1228282706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:29:51.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:51 smithi117 ceph-mon[31838]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:51.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:51 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1228282706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:29:51.596 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 19 03:29:51 smithi178 systemd[1]: Starting Ceph mds.l for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:29:51.862 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 19 03:29:51 smithi178 podman[97809]: 2024-03-19 03:29:51.501129971 +0000 UTC m=+0.017461891 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:29:51.862 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 19 03:29:51 smithi178 podman[97809]: 2024-03-19 03:29:51.601125026 +0000 UTC m=+0.117456943 container create 503c6609fe6d22ac2b098397c1cdaa9f979700f1a6467d2bf89979353f8f43be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-l, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T03:29:51.862 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 19 03:29:51 smithi178 podman[97809]: 2024-03-19 03:29:51.862417969 +0000 UTC m=+0.378749888 container init 503c6609fe6d22ac2b098397c1cdaa9f979700f1a6467d2bf89979353f8f43be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-l, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-19T03:29:52.346 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 19 03:29:51 smithi178 podman[97809]: 2024-03-19 03:29:51.865124457 +0000 UTC m=+0.381456374 container start 503c6609fe6d22ac2b098397c1cdaa9f979700f1a6467d2bf89979353f8f43be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0) 2024-03-19T03:29:52.346 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 19 03:29:51 smithi178 ceph-mds[97827]: -- 172.21.15.178:0/2068392575 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564a7f735a40 con 0x564a7f708800 2024-03-19T03:29:52.346 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 19 03:29:51 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-l[97823]: starting mds.l at 2024-03-19T03:29:52.346 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 19 03:29:51 smithi178 bash[97809]: 503c6609fe6d22ac2b098397c1cdaa9f979700f1a6467d2bf89979353f8f43be 2024-03-19T03:29:52.346 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 19 03:29:51 smithi178 systemd[1]: Started Ceph mds.l for a3a82456-e59f-11ee-95c9-87774f69a715. 2024-03-19T03:29:52.546 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:29:52.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.765+0000 7f60acc00640 1 -- 172.21.15.117:0/836315274 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60a8077420 msgr2=0x7f60a8077880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:52.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.765+0000 7f60acc00640 1 --2- 172.21.15.117:0/836315274 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60a8077420 0x7f60a8077880 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6094009900 tx=0x7f609402bc50 comp rx=0 tx=0).stop 2024-03-19T03:29:52.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.765+0000 7f60acc00640 1 -- 172.21.15.117:0/836315274 shutdown_connections 2024-03-19T03:29:52.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.765+0000 7f60acc00640 1 --2- 172.21.15.117:0/836315274 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60a8077dc0 0x7f60a8078660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:52.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.765+0000 7f60acc00640 1 --2- 172.21.15.117:0/836315274 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60a8077420 0x7f60a8077880 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:52.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.765+0000 7f60acc00640 1 --2- 172.21.15.117:0/836315274 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f60a80761e0 0x7f60a8076600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:52.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.765+0000 7f60acc00640 1 -- 172.21.15.117:0/836315274 >> 172.21.15.117:0/836315274 conn(0x7f60a80fefd0 msgr2=0x7f60a8101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:29:52.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.765+0000 7f60acc00640 1 -- 172.21.15.117:0/836315274 shutdown_connections 2024-03-19T03:29:52.767 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.765+0000 7f60acc00640 1 -- 172.21.15.117:0/836315274 wait complete. 2024-03-19T03:29:52.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.766+0000 7f60acc00640 1 Processor -- start 2024-03-19T03:29:52.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.766+0000 7f60acc00640 1 -- start start 2024-03-19T03:29:52.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.767+0000 7f60acc00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60a80761e0 0x7f60a819e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:52.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.767+0000 7f60acc00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60a8077420 0x7f60a819e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:52.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.767+0000 7f60acc00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f60a8077dc0 0x7f60a81a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:52.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.767+0000 7f60acc00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f60a8113af0 con 0x7f60a8077420 2024-03-19T03:29:52.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.767+0000 7f60acc00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f60a8113970 con 0x7f60a80761e0 2024-03-19T03:29:52.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.767+0000 7f60acc00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f60a8113c70 con 0x7f60a8077dc0 2024-03-19T03:29:52.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.767+0000 7f60a5e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60a80761e0 0x7f60a819e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:52.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.767+0000 7f60a6800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f60a8077dc0 0x7f60a81a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:52.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.767+0000 7f60a5e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60a80761e0 0x7f60a819e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:38222/0 (socket says 172.21.15.117:38222) 2024-03-19T03:29:52.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.767+0000 7f60a5e00640 1 -- 172.21.15.117:0/1098103466 learned_addr learned my addr 172.21.15.117:0/1098103466 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:29:52.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.767+0000 7f60a5400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60a8077420 0x7f60a819e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:52.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.767+0000 7f60a5e00640 1 -- 172.21.15.117:0/1098103466 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f60a8077dc0 msgr2=0x7f60a81a57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:52.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.767+0000 7f60a5e00640 1 --2- 172.21.15.117:0/1098103466 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f60a8077dc0 0x7f60a81a57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:52.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.767+0000 7f60a5e00640 1 -- 172.21.15.117:0/1098103466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60a8077420 msgr2=0x7f60a819e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:52.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.767+0000 7f60a5e00640 1 --2- 172.21.15.117:0/1098103466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60a8077420 0x7f60a819e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:52.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.767+0000 7f60a5e00640 1 -- 172.21.15.117:0/1098103466 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60a81a5e20 con 0x7f60a80761e0 2024-03-19T03:29:52.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.768+0000 7f60a5e00640 1 --2- 172.21.15.117:0/1098103466 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60a80761e0 0x7f60a819e200 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f609800eab0 tx=0x7f609800ef70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:29:52.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.768+0000 7f609ea00640 1 -- 172.21.15.117:0/1098103466 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609800ce40 con 0x7f60a80761e0 2024-03-19T03:29:52.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.768+0000 7f609ea00640 1 -- 172.21.15.117:0/1098103466 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6098011950 con 0x7f60a80761e0 2024-03-19T03:29:52.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.768+0000 7f60acc00640 1 -- 172.21.15.117:0/1098103466 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60a81a6110 con 0x7f60a80761e0 2024-03-19T03:29:52.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.768+0000 7f609ea00640 1 -- 172.21.15.117:0/1098103466 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6098010ae0 con 0x7f60a80761e0 2024-03-19T03:29:52.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.768+0000 7f60acc00640 1 -- 172.21.15.117:0/1098103466 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60a81a6680 con 0x7f60a80761e0 2024-03-19T03:29:52.771 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.769+0000 7f60acc00640 1 -- 172.21.15.117:0/1098103466 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60a8110550 con 0x7f60a80761e0 2024-03-19T03:29:52.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.771+0000 7f609ea00640 1 -- 172.21.15.117:0/1098103466 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f6098011b00 con 0x7f60a80761e0 2024-03-19T03:29:52.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.772+0000 7f609ea00640 1 --2- 172.21.15.117:0/1098103466 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f608c0a2880 0x7f608c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:52.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.773+0000 7f609ea00640 1 -- 172.21.15.117:0/1098103466 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f60980cec10 con 0x7f60a80761e0 2024-03-19T03:29:52.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.773+0000 7f60a5400640 1 --2- 172.21.15.117:0/1098103466 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f608c0a2880 0x7f608c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:52.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.773+0000 7f60a5400640 1 --2- 172.21.15.117:0/1098103466 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f608c0a2880 0x7f608c0a4d40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f609402c7d0 tx=0x7f6094005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:29:52.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:52.777+0000 7f609ea00640 1 -- 172.21.15.117:0/1098103466 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6098016030 con 0x7f60a80761e0 2024-03-19T03:29:52.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:52 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.178:6832/3751720259,v1:172.21.15.178:6833/3751720259] up:boot 2024-03-19T03:29:52.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:52 smithi119 ceph-mon[66861]: fsmap 1 up:standby 2024-03-19T03:29:52.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:52 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T03:29:52.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:52 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:52.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:52 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:52.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:52 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:52.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:52 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:29:52.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:52 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:29:52.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:52 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:52.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:52 smithi119 ceph-mon[66861]: Deploying daemon mds.k on smithi119 2024-03-19T03:29:52.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:52 smithi119 ceph-mon[66861]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:53.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:53.017+0000 7f60acc00640 1 -- 172.21.15.117:0/1098103466 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60a8062420 con 0x7f60a80761e0 2024-03-19T03:29:53.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:53.018+0000 7f609ea00640 1 -- 172.21.15.117:0/1098103466 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f609809c0b0 con 0x7f60a80761e0 2024-03-19T03:29:53.019 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:29:53.020 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-19T03:29:53.020 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 2 2024-03-19T03:29:53.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:53.021+0000 7f60acc00640 1 -- 172.21.15.117:0/1098103466 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f608c0a2880 msgr2=0x7f608c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:53.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:53.021+0000 7f60acc00640 1 --2- 172.21.15.117:0/1098103466 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f608c0a2880 0x7f608c0a4d40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f609402c7d0 tx=0x7f6094005fd0 comp rx=0 tx=0).stop 2024-03-19T03:29:53.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:53.021+0000 7f60acc00640 1 -- 172.21.15.117:0/1098103466 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60a80761e0 msgr2=0x7f60a819e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:53.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:53.021+0000 7f60acc00640 1 --2- 172.21.15.117:0/1098103466 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60a80761e0 0x7f60a819e200 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f609800eab0 tx=0x7f609800ef70 comp rx=0 tx=0).stop 2024-03-19T03:29:53.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:53.021+0000 7f60acc00640 1 -- 172.21.15.117:0/1098103466 shutdown_connections 2024-03-19T03:29:53.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:53.021+0000 7f60acc00640 1 --2- 172.21.15.117:0/1098103466 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f60a8077dc0 0x7f60a81a57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:53.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:53.021+0000 7f60acc00640 1 --2- 172.21.15.117:0/1098103466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60a8077420 0x7f60a819e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:53.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:53.021+0000 7f60acc00640 1 --2- 172.21.15.117:0/1098103466 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f608c0a2880 0x7f608c0a4d40 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:53.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:53.021+0000 7f60acc00640 1 --2- 172.21.15.117:0/1098103466 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60a80761e0 0x7f60a819e200 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:53.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:53.021+0000 7f60acc00640 1 -- 172.21.15.117:0/1098103466 >> 172.21.15.117:0/1098103466 conn(0x7f60a80fefd0 msgr2=0x7f60a8100730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:29:53.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:53.022+0000 7f60acc00640 1 -- 172.21.15.117:0/1098103466 shutdown_connections 2024-03-19T03:29:53.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:53.022+0000 7f60acc00640 1 -- 172.21.15.117:0/1098103466 wait complete. 2024-03-19T03:29:53.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:52 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.178:6832/3751720259,v1:172.21.15.178:6833/3751720259] up:boot 2024-03-19T03:29:53.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:52 smithi117 ceph-mon[31838]: fsmap 1 up:standby 2024-03-19T03:29:53.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:52 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T03:29:53.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:52 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:53.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:52 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:53.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:52 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:53.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:52 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:29:53.293 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:52 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:29:53.294 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:52 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:53.294 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:52 smithi117 ceph-mon[31838]: Deploying daemon mds.k on smithi119 2024-03-19T03:29:53.294 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:52 smithi117 ceph-mon[31838]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:53.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:52 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.178:6832/3751720259,v1:172.21.15.178:6833/3751720259] up:boot 2024-03-19T03:29:53.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:52 smithi178 ceph-mon[67578]: fsmap 1 up:standby 2024-03-19T03:29:53.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:52 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T03:29:53.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:52 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:53.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:52 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:53.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:52 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:53.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:52 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:29:53.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:52 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:29:53.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:52 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:53.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:52 smithi178 ceph-mon[67578]: Deploying daemon mds.k on smithi119 2024-03-19T03:29:53.347 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:52 smithi178 ceph-mon[67578]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:53.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:53 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1098103466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:29:54.291 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:53 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1098103466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:29:54.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:53 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1098103466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:29:54.484 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 19 03:29:54 smithi119 systemd[1]: Starting Ceph mds.k for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:29:54.803 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 19 03:29:54 smithi119 podman[96065]: 2024-03-19 03:29:54.397284095 +0000 UTC m=+0.017537905 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:29:54.804 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 19 03:29:54 smithi119 podman[96065]: 2024-03-19 03:29:54.500774185 +0000 UTC m=+0.121027997 container create 0ad918646f67f8c935f922f1c21663e460f4417478d6e8849188312df1a702d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-19T03:29:54.804 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 19 03:29:54 smithi119 podman[96065]: 2024-03-19 03:29:54.711749399 +0000 UTC m=+0.332003197 container init 0ad918646f67f8c935f922f1c21663e460f4417478d6e8849188312df1a702d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-k, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-19T03:29:54.804 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 19 03:29:54 smithi119 podman[96065]: 2024-03-19 03:29:54.714381576 +0000 UTC m=+0.334635376 container start 0ad918646f67f8c935f922f1c21663e460f4417478d6e8849188312df1a702d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-19T03:29:54.804 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 19 03:29:54 smithi119 ceph-mds[96084]: -- 172.21.15.119:0/3677223297 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556eead71a40 con 0x556eead44800 2024-03-19T03:29:54.804 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 19 03:29:54 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-k[96080]: starting mds.k at 2024-03-19T03:29:54.804 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 19 03:29:54 smithi119 bash[96065]: 0ad918646f67f8c935f922f1c21663e460f4417478d6e8849188312df1a702d4 2024-03-19T03:29:55.043 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:29:55.234 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 19 03:29:54 smithi119 systemd[1]: Started Ceph mds.k for a3a82456-e59f-11ee-95c9-87774f69a715. 2024-03-19T03:29:55.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:54 smithi119 ceph-mon[66861]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:55.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:54 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:55.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:54 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:55.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:54 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:55.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:54 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:29:55.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:54 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:29:55.235 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:54 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:55.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.235+0000 7fd6e1000640 1 -- 172.21.15.117:0/1901466052 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6dc0761e0 msgr2=0x7fd6dc076600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:55.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.235+0000 7fd6e1000640 1 --2- 172.21.15.117:0/1901466052 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6dc0761e0 0x7fd6dc076600 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc009860 tx=0x7fd6cc02bb70 comp rx=0 tx=0).stop 2024-03-19T03:29:55.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.235+0000 7fd6e1000640 1 -- 172.21.15.117:0/1901466052 shutdown_connections 2024-03-19T03:29:55.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.235+0000 7fd6e1000640 1 --2- 172.21.15.117:0/1901466052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6dc077dc0 0x7fd6dc078660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:55.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.235+0000 7fd6e1000640 1 --2- 172.21.15.117:0/1901466052 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6dc077420 0x7fd6dc077880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:55.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.235+0000 7fd6e1000640 1 --2- 172.21.15.117:0/1901466052 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6dc0761e0 0x7fd6dc076600 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:55.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.235+0000 7fd6e1000640 1 -- 172.21.15.117:0/1901466052 >> 172.21.15.117:0/1901466052 conn(0x7fd6dc0ff010 msgr2=0x7fd6dc101430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:29:55.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.235+0000 7fd6e1000640 1 -- 172.21.15.117:0/1901466052 shutdown_connections 2024-03-19T03:29:55.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.235+0000 7fd6e1000640 1 -- 172.21.15.117:0/1901466052 wait complete. 2024-03-19T03:29:55.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.236+0000 7fd6e1000640 1 Processor -- start 2024-03-19T03:29:55.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.236+0000 7fd6e1000640 1 -- start start 2024-03-19T03:29:55.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.237+0000 7fd6e1000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6dc0761e0 0x7fd6dc19e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:55.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.237+0000 7fd6e1000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6dc077420 0x7fd6dc19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:55.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.237+0000 7fd6e1000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6dc077dc0 0x7fd6dc1a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:55.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.237+0000 7fd6e1000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd6dc113af0 con 0x7fd6dc0761e0 2024-03-19T03:29:55.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.237+0000 7fd6e1000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd6dc113970 con 0x7fd6dc077420 2024-03-19T03:29:55.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.237+0000 7fd6e1000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fd6dc113c70 con 0x7fd6dc077dc0 2024-03-19T03:29:55.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.237+0000 7fd6d9e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6dc0761e0 0x7fd6dc19e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:55.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.237+0000 7fd6d9e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6dc0761e0 0x7fd6dc19e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42094/0 (socket says 172.21.15.117:42094) 2024-03-19T03:29:55.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.237+0000 7fd6d9e00640 1 -- 172.21.15.117:0/2809605656 learned_addr learned my addr 172.21.15.117:0/2809605656 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:29:55.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.237+0000 7fd6d9400640 1 --2- 172.21.15.117:0/2809605656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6dc077420 0x7fd6dc19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:55.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.237+0000 7fd6da800640 1 --2- 172.21.15.117:0/2809605656 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6dc077dc0 0x7fd6dc1a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:55.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.237+0000 7fd6d9400640 1 -- 172.21.15.117:0/2809605656 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6dc077dc0 msgr2=0x7fd6dc1a56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:55.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.237+0000 7fd6d9400640 1 --2- 172.21.15.117:0/2809605656 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6dc077dc0 0x7fd6dc1a56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:55.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.237+0000 7fd6d9400640 1 -- 172.21.15.117:0/2809605656 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6dc0761e0 msgr2=0x7fd6dc19e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:55.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.237+0000 7fd6d9400640 1 --2- 172.21.15.117:0/2809605656 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6dc0761e0 0x7fd6dc19e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:55.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.237+0000 7fd6d9400640 1 -- 172.21.15.117:0/2809605656 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6dc1a5d50 con 0x7fd6dc077420 2024-03-19T03:29:55.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.237+0000 7fd6d9e00640 1 --2- 172.21.15.117:0/2809605656 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6dc0761e0 0x7fd6dc19e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:29:55.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.238+0000 7fd6d9400640 1 --2- 172.21.15.117:0/2809605656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6dc077420 0x7fd6dc19e670 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8005fd0 tx=0x7fd6c800ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:29:55.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.238+0000 7fd6d2a00640 1 -- 172.21.15.117:0/2809605656 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c800cd00 con 0x7fd6dc077420 2024-03-19T03:29:55.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.238+0000 7fd6d2a00640 1 -- 172.21.15.117:0/2809605656 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6c80035f0 con 0x7fd6dc077420 2024-03-19T03:29:55.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.238+0000 7fd6e1000640 1 -- 172.21.15.117:0/2809605656 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6dc1a5fe0 con 0x7fd6dc077420 2024-03-19T03:29:55.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.238+0000 7fd6d2a00640 1 -- 172.21.15.117:0/2809605656 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8010690 con 0x7fd6dc077420 2024-03-19T03:29:55.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.238+0000 7fd6e1000640 1 -- 172.21.15.117:0/2809605656 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6dc1a6500 con 0x7fd6dc077420 2024-03-19T03:29:55.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.239+0000 7fd6e1000640 1 -- 172.21.15.117:0/2809605656 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6dc110580 con 0x7fd6dc077420 2024-03-19T03:29:55.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.241+0000 7fd6d2a00640 1 -- 172.21.15.117:0/2809605656 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fd6c80037a0 con 0x7fd6dc077420 2024-03-19T03:29:55.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.243+0000 7fd6d2a00640 1 --2- 172.21.15.117:0/2809605656 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd6c00a2800 0x7fd6c00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:55.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.243+0000 7fd6d2a00640 1 -- 172.21.15.117:0/2809605656 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd6c80ce850 con 0x7fd6dc077420 2024-03-19T03:29:55.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.243+0000 7fd6d9e00640 1 --2- 172.21.15.117:0/2809605656 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd6c00a2800 0x7fd6c00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:55.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.244+0000 7fd6d9e00640 1 --2- 172.21.15.117:0/2809605656 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd6c00a2800 0x7fd6c00a4cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc0095f0 tx=0x7fd6cc02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:29:55.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.246+0000 7fd6d2a00640 1 -- 172.21.15.117:0/2809605656 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd6c8009510 con 0x7fd6dc077420 2024-03-19T03:29:55.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:54 smithi117 ceph-mon[31838]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:55.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:54 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:55.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:54 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:55.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:54 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:55.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:54 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:29:55.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:54 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:29:55.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:54 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:55.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:54 smithi178 ceph-mon[67578]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:55.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:54 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:55.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:54 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:55.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:54 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:55.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:54 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:29:55.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:54 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:29:55.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:54 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:55.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.485+0000 7fd6e1000640 1 -- 172.21.15.117:0/2809605656 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6dc02d1f0 con 0x7fd6dc077420 2024-03-19T03:29:55.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.486+0000 7fd6d2a00640 1 -- 172.21.15.117:0/2809605656 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7fd6c809bcf0 con 0x7fd6dc077420 2024-03-19T03:29:55.486 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:29:55.486 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/1291833214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1291833214},{"type":"v1","addr":"172.21.15.119:6833","nonce":1291833214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-19T03:29:55.487 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 3 2024-03-19T03:29:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.489+0000 7fd6e1000640 1 -- 172.21.15.117:0/2809605656 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd6c00a2800 msgr2=0x7fd6c00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.489+0000 7fd6e1000640 1 --2- 172.21.15.117:0/2809605656 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd6c00a2800 0x7fd6c00a4cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc0095f0 tx=0x7fd6cc02e040 comp rx=0 tx=0).stop 2024-03-19T03:29:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.489+0000 7fd6e1000640 1 -- 172.21.15.117:0/2809605656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6dc077420 msgr2=0x7fd6dc19e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.489+0000 7fd6e1000640 1 --2- 172.21.15.117:0/2809605656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6dc077420 0x7fd6dc19e670 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8005fd0 tx=0x7fd6c800ee50 comp rx=0 tx=0).stop 2024-03-19T03:29:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.489+0000 7fd6e1000640 1 -- 172.21.15.117:0/2809605656 shutdown_connections 2024-03-19T03:29:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.489+0000 7fd6e1000640 1 --2- 172.21.15.117:0/2809605656 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6dc077dc0 0x7fd6dc1a56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.489+0000 7fd6e1000640 1 --2- 172.21.15.117:0/2809605656 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd6c00a2800 0x7fd6c00a4cc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.489+0000 7fd6e1000640 1 --2- 172.21.15.117:0/2809605656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6dc077420 0x7fd6dc19e670 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:55.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.489+0000 7fd6e1000640 1 --2- 172.21.15.117:0/2809605656 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6dc0761e0 0x7fd6dc19e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:55.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.489+0000 7fd6e1000640 1 -- 172.21.15.117:0/2809605656 >> 172.21.15.117:0/2809605656 conn(0x7fd6dc0ff010 msgr2=0x7fd6dc100730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:29:55.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.489+0000 7fd6e1000640 1 -- 172.21.15.117:0/2809605656 shutdown_connections 2024-03-19T03:29:55.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:55.489+0000 7fd6e1000640 1 -- 172.21.15.117:0/2809605656 wait complete. 2024-03-19T03:29:56.190 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:55 smithi117 ceph-mon[31838]: Deploying daemon mds.j on smithi117 2024-03-19T03:29:56.190 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:55 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] up:boot 2024-03-19T03:29:56.191 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:55 smithi117 ceph-mon[31838]: fsmap 2 up:standby 2024-03-19T03:29:56.191 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T03:29:56.191 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:55 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2809605656' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:29:56.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:55 smithi119 ceph-mon[66861]: Deploying daemon mds.j on smithi117 2024-03-19T03:29:56.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:55 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] up:boot 2024-03-19T03:29:56.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:55 smithi119 ceph-mon[66861]: fsmap 2 up:standby 2024-03-19T03:29:56.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T03:29:56.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:55 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2809605656' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:29:56.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:55 smithi178 ceph-mon[67578]: Deploying daemon mds.j on smithi117 2024-03-19T03:29:56.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:55 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] up:boot 2024-03-19T03:29:56.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:55 smithi178 ceph-mon[67578]: fsmap 2 up:standby 2024-03-19T03:29:56.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T03:29:56.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:55 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2809605656' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:29:57.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:56 smithi117 ceph-mon[31838]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:57.123 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:56 smithi119 ceph-mon[66861]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:57.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:56 smithi178 ceph-mon[67578]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:57.509 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:29:57.957 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.957+0000 7ff5a1c00640 1 -- 172.21.15.117:0/989663213 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff59c101690 msgr2=0x7ff59c101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:57.958 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.957+0000 7ff5a1c00640 1 --2- 172.21.15.117:0/989663213 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff59c101690 0x7ff59c101ab0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff58c009900 tx=0x7ff58c02bc30 comp rx=0 tx=0).stop 2024-03-19T03:29:57.958 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.957+0000 7ff5a1c00640 1 -- 172.21.15.117:0/989663213 shutdown_connections 2024-03-19T03:29:57.958 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.957+0000 7ff5a1c00640 1 --2- 172.21.15.117:0/989663213 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff59c103240 0x7ff59c109ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:57.958 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.957+0000 7ff5a1c00640 1 --2- 172.21.15.117:0/989663213 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff59c102880 0x7ff59c102d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:57.958 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.957+0000 7ff5a1c00640 1 --2- 172.21.15.117:0/989663213 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff59c101690 0x7ff59c101ab0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:57.958 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.957+0000 7ff5a1c00640 1 -- 172.21.15.117:0/989663213 >> 172.21.15.117:0/989663213 conn(0x7ff59c0fcc00 msgr2=0x7ff59c0ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:29:57.958 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.958+0000 7ff5a1c00640 1 -- 172.21.15.117:0/989663213 shutdown_connections 2024-03-19T03:29:57.958 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.958+0000 7ff5a1c00640 1 -- 172.21.15.117:0/989663213 wait complete. 2024-03-19T03:29:57.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.959+0000 7ff5a1c00640 1 Processor -- start 2024-03-19T03:29:57.959 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.959+0000 7ff5a1c00640 1 -- start start 2024-03-19T03:29:57.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.959+0000 7ff5a1c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff59c101690 0x7ff59c199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:57.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.959+0000 7ff5a1c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff59c102880 0x7ff59c19a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:57.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.959+0000 7ff5a1c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff59c103240 0x7ff59c1a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:57.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.960+0000 7ff5a1c00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff59c10d350 con 0x7ff59c103240 2024-03-19T03:29:57.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.960+0000 7ff5a1c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff59c10d1d0 con 0x7ff59c102880 2024-03-19T03:29:57.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.960+0000 7ff5a1c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7ff59c10d4d0 con 0x7ff59c101690 2024-03-19T03:29:57.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.960+0000 7ff59b400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff59c102880 0x7ff59c19a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:57.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.960+0000 7ff59b400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff59c102880 0x7ff59c19a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:49694/0 (socket says 172.21.15.117:49694) 2024-03-19T03:29:57.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.960+0000 7ff59b400640 1 -- 172.21.15.117:0/3032732581 learned_addr learned my addr 172.21.15.117:0/3032732581 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:29:57.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.960+0000 7ff5a1200640 1 --2- 172.21.15.117:0/3032732581 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff59c103240 0x7ff59c1a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:57.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.960+0000 7ff59be00640 1 --2- 172.21.15.117:0/3032732581 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff59c101690 0x7ff59c199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:57.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.960+0000 7ff59b400640 1 -- 172.21.15.117:0/3032732581 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff59c101690 msgr2=0x7ff59c199b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:57.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.960+0000 7ff59b400640 1 --2- 172.21.15.117:0/3032732581 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff59c101690 0x7ff59c199b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:57.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.960+0000 7ff59b400640 1 -- 172.21.15.117:0/3032732581 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff59c103240 msgr2=0x7ff59c1a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:57.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.960+0000 7ff59b400640 1 --2- 172.21.15.117:0/3032732581 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff59c103240 0x7ff59c1a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:57.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.960+0000 7ff59b400640 1 -- 172.21.15.117:0/3032732581 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff59c1a17b0 con 0x7ff59c102880 2024-03-19T03:29:57.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.960+0000 7ff5a1200640 1 --2- 172.21.15.117:0/3032732581 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff59c103240 0x7ff59c1a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:29:57.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.961+0000 7ff59b400640 1 --2- 172.21.15.117:0/3032732581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff59c102880 0x7ff59c19a0d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff58800e9f0 tx=0x7ff58800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:29:57.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.961+0000 7ff598c00640 1 -- 172.21.15.117:0/3032732581 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58800cd80 con 0x7ff59c102880 2024-03-19T03:29:57.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.961+0000 7ff598c00640 1 -- 172.21.15.117:0/3032732581 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff588005500 con 0x7ff59c102880 2024-03-19T03:29:57.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.961+0000 7ff5a1c00640 1 -- 172.21.15.117:0/3032732581 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff59c1a1aa0 con 0x7ff59c102880 2024-03-19T03:29:57.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.961+0000 7ff598c00640 1 -- 172.21.15.117:0/3032732581 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff588010690 con 0x7ff59c102880 2024-03-19T03:29:57.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.961+0000 7ff5a1c00640 1 -- 172.21.15.117:0/3032732581 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff59c1a1ff0 con 0x7ff59c102880 2024-03-19T03:29:57.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.962+0000 7ff5a1c00640 1 -- 172.21.15.117:0/3032732581 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff59c1a2280 con 0x7ff59c102880 2024-03-19T03:29:57.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.965+0000 7ff598c00640 1 -- 172.21.15.117:0/3032732581 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7ff588004f60 con 0x7ff59c102880 2024-03-19T03:29:57.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.966+0000 7ff598c00640 1 --2- 172.21.15.117:0/3032732581 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7ff5800a2880 0x7ff5800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:29:57.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.966+0000 7ff598c00640 1 -- 172.21.15.117:0/3032732581 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff5880ce9e0 con 0x7ff59c102880 2024-03-19T03:29:57.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.966+0000 7ff59be00640 1 --2- 172.21.15.117:0/3032732581 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7ff5800a2880 0x7ff5800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:29:57.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.967+0000 7ff59be00640 1 --2- 172.21.15.117:0/3032732581 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7ff5800a2880 0x7ff5800a4d40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff58c02c170 tx=0x7ff58c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:29:57.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:57.970+0000 7ff598c00640 1 -- 172.21.15.117:0/3032732581 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff58809be80 con 0x7ff59c102880 2024-03-19T03:29:58.229 INFO:journalctl@ceph.mds.j.smithi117.stdout:Mar 19 03:29:57 smithi117 systemd[1]: Starting Ceph mds.j for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:29:58.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:58.242+0000 7ff5a1c00640 1 -- 172.21.15.117:0/3032732581 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff59c1a2450 con 0x7ff59c102880 2024-03-19T03:29:58.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:58.243+0000 7ff598c00640 1 -- 172.21.15.117:0/3032732581 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7ff58809b780 con 0x7ff59c102880 2024-03-19T03:29:58.243 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:29:58.243 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/1291833214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1291833214},{"type":"v1","addr":"172.21.15.119:6833","nonce":1291833214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-19T03:29:58.243 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 3 2024-03-19T03:29:58.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:58.245+0000 7ff5a1c00640 1 -- 172.21.15.117:0/3032732581 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7ff5800a2880 msgr2=0x7ff5800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:58.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:58.246+0000 7ff5a1c00640 1 --2- 172.21.15.117:0/3032732581 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7ff5800a2880 0x7ff5800a4d40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff58c02c170 tx=0x7ff58c02df90 comp rx=0 tx=0).stop 2024-03-19T03:29:58.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:58.246+0000 7ff5a1c00640 1 -- 172.21.15.117:0/3032732581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff59c102880 msgr2=0x7ff59c19a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:29:58.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:58.246+0000 7ff5a1c00640 1 --2- 172.21.15.117:0/3032732581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff59c102880 0x7ff59c19a0d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff58800e9f0 tx=0x7ff58800edf0 comp rx=0 tx=0).stop 2024-03-19T03:29:58.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:58.246+0000 7ff5a1c00640 1 -- 172.21.15.117:0/3032732581 shutdown_connections 2024-03-19T03:29:58.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:58.246+0000 7ff5a1c00640 1 --2- 172.21.15.117:0/3032732581 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff59c103240 0x7ff59c1a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:58.247 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:58.246+0000 7ff5a1c00640 1 --2- 172.21.15.117:0/3032732581 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7ff5800a2880 0x7ff5800a4d40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:58.247 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:58.246+0000 7ff5a1c00640 1 --2- 172.21.15.117:0/3032732581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff59c102880 0x7ff59c19a0d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:58.247 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:58.246+0000 7ff5a1c00640 1 --2- 172.21.15.117:0/3032732581 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff59c101690 0x7ff59c199b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:29:58.247 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:58.246+0000 7ff5a1c00640 1 -- 172.21.15.117:0/3032732581 >> 172.21.15.117:0/3032732581 conn(0x7ff59c0fcc00 msgr2=0x7ff59c073a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:29:58.247 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:58.246+0000 7ff5a1c00640 1 -- 172.21.15.117:0/3032732581 shutdown_connections 2024-03-19T03:29:58.247 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:29:58.246+0000 7ff5a1c00640 1 -- 172.21.15.117:0/3032732581 wait complete. 2024-03-19T03:29:58.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:58 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3032732581' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:29:58.534 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:58 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3032732581' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:29:58.534 INFO:journalctl@ceph.mds.j.smithi117.stdout:Mar 19 03:29:58 smithi117 podman[72649]: 2024-03-19 03:29:58.133765122 +0000 UTC m=+0.032506292 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:29:58.534 INFO:journalctl@ceph.mds.j.smithi117.stdout:Mar 19 03:29:58 smithi117 podman[72649]: 2024-03-19 03:29:58.263624689 +0000 UTC m=+0.162365857 container create 657e4491ff59e38c824874bf92b170d9ba9f7752c1f8e1fc453461ba0c5a4f2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-j, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-19T03:29:58.535 INFO:journalctl@ceph.mds.j.smithi117.stdout:Mar 19 03:29:58 smithi117 podman[72649]: 2024-03-19 03:29:58.533867606 +0000 UTC m=+0.432608820 container init 657e4491ff59e38c824874bf92b170d9ba9f7752c1f8e1fc453461ba0c5a4f2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-j, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:29:58.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:58 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3032732581' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:29:58.792 INFO:journalctl@ceph.mds.j.smithi117.stdout:Mar 19 03:29:58 smithi117 podman[72649]: 2024-03-19 03:29:58.544127511 +0000 UTC m=+0.442868693 container start 657e4491ff59e38c824874bf92b170d9ba9f7752c1f8e1fc453461ba0c5a4f2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-j, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-19T03:29:58.792 INFO:journalctl@ceph.mds.j.smithi117.stdout:Mar 19 03:29:58 smithi117 ceph-mds[72669]: -- 172.21.15.117:0/1844782511 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563be7ed1a40 con 0x563be7ea4c00 2024-03-19T03:29:58.792 INFO:journalctl@ceph.mds.j.smithi117.stdout:Mar 19 03:29:58 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-j[72665]: starting mds.j at 2024-03-19T03:29:58.792 INFO:journalctl@ceph.mds.j.smithi117.stdout:Mar 19 03:29:58 smithi117 bash[72649]: 657e4491ff59e38c824874bf92b170d9ba9f7752c1f8e1fc453461ba0c5a4f2a 2024-03-19T03:29:58.792 INFO:journalctl@ceph.mds.j.smithi117.stdout:Mar 19 03:29:58 smithi117 systemd[1]: Started Ceph mds.j for a3a82456-e59f-11ee-95c9-87774f69a715. 2024-03-19T03:29:59.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:59 smithi178 ceph-mon[67578]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:59.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:59 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:59.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:59 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:59.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:59 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:59.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:59 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:29:59.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:59 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:29:59.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:29:59 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:59.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:59 smithi119 ceph-mon[66861]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:59.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:59 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:59.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:59 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:59.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:59 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:59.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:59 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:29:59.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:59 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:29:59.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:29:59 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:29:59.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:59 smithi117 ceph-mon[31838]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:29:59.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:59 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:59.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:59 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:59.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:59 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:29:59.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:59 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:29:59.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:59 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:29:59.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:29:59 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:00.270 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:30:00.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.488+0000 7fc130e00640 1 -- 172.21.15.117:0/4187662053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc12c104c50 msgr2=0x7fc12c1050d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:00.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.488+0000 7fc130e00640 1 --2- 172.21.15.117:0/4187662053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc12c104c50 0x7fc12c1050d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc118009900 tx=0x7fc11802bc30 comp rx=0 tx=0).stop 2024-03-19T03:30:00.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.489+0000 7fc130e00640 1 -- 172.21.15.117:0/4187662053 shutdown_connections 2024-03-19T03:30:00.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.489+0000 7fc130e00640 1 --2- 172.21.15.117:0/4187662053 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc12c105610 0x7fc12c10bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:00.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.489+0000 7fc130e00640 1 --2- 172.21.15.117:0/4187662053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc12c104c50 0x7fc12c1050d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:00.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.489+0000 7fc130e00640 1 --2- 172.21.15.117:0/4187662053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc12c103a60 0x7fc12c103e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:00.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.489+0000 7fc130e00640 1 -- 172.21.15.117:0/4187662053 >> 172.21.15.117:0/4187662053 conn(0x7fc12c0ff010 msgr2=0x7fc12c101430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:00.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.489+0000 7fc130e00640 1 -- 172.21.15.117:0/4187662053 shutdown_connections 2024-03-19T03:30:00.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.489+0000 7fc130e00640 1 -- 172.21.15.117:0/4187662053 wait complete. 2024-03-19T03:30:00.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.490+0000 7fc130e00640 1 Processor -- start 2024-03-19T03:30:00.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.490+0000 7fc130e00640 1 -- start start 2024-03-19T03:30:00.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.490+0000 7fc130e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc12c103a60 0x7fc12c199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:00.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.490+0000 7fc130e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc12c104c50 0x7fc12c19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:00.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.490+0000 7fc130e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc12c105610 0x7fc12c1a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:00.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.490+0000 7fc130e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc12c10f730 con 0x7fc12c104c50 2024-03-19T03:30:00.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.490+0000 7fc130e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc12c10f5b0 con 0x7fc12c103a60 2024-03-19T03:30:00.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.490+0000 7fc130e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fc12c10f8b0 con 0x7fc12c105610 2024-03-19T03:30:00.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.491+0000 7fc129400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc12c104c50 0x7fc12c19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:00.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.491+0000 7fc129e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc12c103a60 0x7fc12c199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:00.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.491+0000 7fc129e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc12c103a60 0x7fc12c199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:49728/0 (socket says 172.21.15.117:49728) 2024-03-19T03:30:00.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.491+0000 7fc129e00640 1 -- 172.21.15.117:0/3294590696 learned_addr learned my addr 172.21.15.117:0/3294590696 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:00.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.491+0000 7fc12a800640 1 --2- 172.21.15.117:0/3294590696 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc12c105610 0x7fc12c1a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:00.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.491+0000 7fc129e00640 1 -- 172.21.15.117:0/3294590696 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc12c105610 msgr2=0x7fc12c1a1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:00.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.491+0000 7fc129e00640 1 --2- 172.21.15.117:0/3294590696 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc12c105610 0x7fc12c1a1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:00.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.491+0000 7fc129e00640 1 -- 172.21.15.117:0/3294590696 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc12c104c50 msgr2=0x7fc12c19a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:00.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.491+0000 7fc129e00640 1 --2- 172.21.15.117:0/3294590696 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc12c104c50 0x7fc12c19a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:00.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.491+0000 7fc129e00640 1 -- 172.21.15.117:0/3294590696 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc12c1a1970 con 0x7fc12c103a60 2024-03-19T03:30:00.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.491+0000 7fc129e00640 1 --2- 172.21.15.117:0/3294590696 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc12c103a60 0x7fc12c199d50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc11c00e9c0 tx=0x7fc11c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:00.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.491+0000 7fc122a00640 1 -- 172.21.15.117:0/3294590696 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11c00cde0 con 0x7fc12c103a60 2024-03-19T03:30:00.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.492+0000 7fc122a00640 1 -- 172.21.15.117:0/3294590696 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc11c010910 con 0x7fc12c103a60 2024-03-19T03:30:00.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.492+0000 7fc130e00640 1 -- 172.21.15.117:0/3294590696 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc12c1a1c60 con 0x7fc12c103a60 2024-03-19T03:30:00.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.492+0000 7fc122a00640 1 -- 172.21.15.117:0/3294590696 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11c018a00 con 0x7fc12c103a60 2024-03-19T03:30:00.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.492+0000 7fc130e00640 1 -- 172.21.15.117:0/3294590696 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc12c1a2180 con 0x7fc12c103a60 2024-03-19T03:30:00.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.493+0000 7fc130e00640 1 -- 172.21.15.117:0/3294590696 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc12c10ad80 con 0x7fc12c103a60 2024-03-19T03:30:00.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.495+0000 7fc122a00640 1 -- 172.21.15.117:0/3294590696 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fc11c01f020 con 0x7fc12c103a60 2024-03-19T03:30:00.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.496+0000 7fc122a00640 1 --2- 172.21.15.117:0/3294590696 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc1100a2800 0x7fc1100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:00.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.496+0000 7fc122a00640 1 -- 172.21.15.117:0/3294590696 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc11c0ce7e0 con 0x7fc12c103a60 2024-03-19T03:30:00.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.496+0000 7fc129400640 1 --2- 172.21.15.117:0/3294590696 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc1100a2800 0x7fc1100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:00.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.497+0000 7fc129400640 1 --2- 172.21.15.117:0/3294590696 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc1100a2800 0x7fc1100a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc11802c240 tx=0x7fc11802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:00.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.500+0000 7fc122a00640 1 -- 172.21.15.117:0/3294590696 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc11c09bc80 con 0x7fc12c103a60 2024-03-19T03:30:00.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:00 smithi117 ceph-mon[31838]: Deploying daemon mds.d on smithi117 2024-03-19T03:30:00.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:00 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.117:6834/54638259,v1:172.21.15.117:6835/54638259] up:boot 2024-03-19T03:30:00.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:00 smithi117 ceph-mon[31838]: fsmap 3 up:standby 2024-03-19T03:30:00.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:00 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-19T03:30:00.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:00 smithi117 ceph-mon[31838]: overall HEALTH_OK 2024-03-19T03:30:00.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:00 smithi178 ceph-mon[67578]: Deploying daemon mds.d on smithi117 2024-03-19T03:30:00.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:00 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.117:6834/54638259,v1:172.21.15.117:6835/54638259] up:boot 2024-03-19T03:30:00.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:00 smithi178 ceph-mon[67578]: fsmap 3 up:standby 2024-03-19T03:30:00.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:00 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-19T03:30:00.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:00 smithi178 ceph-mon[67578]: overall HEALTH_OK 2024-03-19T03:30:00.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:00 smithi119 ceph-mon[66861]: Deploying daemon mds.d on smithi117 2024-03-19T03:30:00.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:00 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.117:6834/54638259,v1:172.21.15.117:6835/54638259] up:boot 2024-03-19T03:30:00.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:00 smithi119 ceph-mon[66861]: fsmap 3 up:standby 2024-03-19T03:30:00.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:00 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-19T03:30:00.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:00 smithi119 ceph-mon[66861]: overall HEALTH_OK 2024-03-19T03:30:00.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.764+0000 7fc130e00640 1 -- 172.21.15.117:0/3294590696 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc12c0624c0 con 0x7fc12c103a60 2024-03-19T03:30:00.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.765+0000 7fc122a00640 1 -- 172.21.15.117:0/3294590696 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7fc11c09b580 con 0x7fc12c103a60 2024-03-19T03:30:00.765 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:30:00.765 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/54638259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":54638259},{"type":"v1","addr":"172.21.15.117:6835","nonce":54638259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/1291833214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1291833214},{"type":"v1","addr":"172.21.15.119:6833","nonce":1291833214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-19T03:30:00.765 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 4 2024-03-19T03:30:00.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.768+0000 7fc130e00640 1 -- 172.21.15.117:0/3294590696 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc1100a2800 msgr2=0x7fc1100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:00.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.768+0000 7fc130e00640 1 --2- 172.21.15.117:0/3294590696 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc1100a2800 0x7fc1100a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc11802c240 tx=0x7fc11802df90 comp rx=0 tx=0).stop 2024-03-19T03:30:00.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.768+0000 7fc130e00640 1 -- 172.21.15.117:0/3294590696 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc12c103a60 msgr2=0x7fc12c199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:00.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.768+0000 7fc130e00640 1 --2- 172.21.15.117:0/3294590696 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc12c103a60 0x7fc12c199d50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc11c00e9c0 tx=0x7fc11c00ee80 comp rx=0 tx=0).stop 2024-03-19T03:30:00.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.768+0000 7fc130e00640 1 -- 172.21.15.117:0/3294590696 shutdown_connections 2024-03-19T03:30:00.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.768+0000 7fc130e00640 1 --2- 172.21.15.117:0/3294590696 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc12c105610 0x7fc12c1a1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:00.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.768+0000 7fc130e00640 1 --2- 172.21.15.117:0/3294590696 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc12c104c50 0x7fc12c19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:00.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.768+0000 7fc130e00640 1 --2- 172.21.15.117:0/3294590696 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc1100a2800 0x7fc1100a4cc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:00.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.768+0000 7fc130e00640 1 --2- 172.21.15.117:0/3294590696 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc12c103a60 0x7fc12c199d50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:00.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.768+0000 7fc130e00640 1 -- 172.21.15.117:0/3294590696 >> 172.21.15.117:0/3294590696 conn(0x7fc12c0ff010 msgr2=0x7fc12c107f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:00.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.768+0000 7fc130e00640 1 -- 172.21.15.117:0/3294590696 shutdown_connections 2024-03-19T03:30:00.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:00.768+0000 7fc130e00640 1 -- 172.21.15.117:0/3294590696 wait complete. 2024-03-19T03:30:01.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:01 smithi178 ceph-mon[67578]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:01.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:01 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3294590696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:01.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:01 smithi119 ceph-mon[66861]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:01.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:01 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3294590696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:01.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:01 smithi117 ceph-mon[31838]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:01.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:01 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3294590696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:02.225 INFO:journalctl@ceph.mds.d.smithi117.stdout:Mar 19 03:30:01 smithi117 systemd[1]: Starting Ceph mds.d for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:30:02.542 INFO:journalctl@ceph.mds.d.smithi117.stdout:Mar 19 03:30:02 smithi117 podman[72996]: 2024-03-19 03:30:02.129014941 +0000 UTC m=+0.031989960 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:30:02.542 INFO:journalctl@ceph.mds.d.smithi117.stdout:Mar 19 03:30:02 smithi117 podman[72996]: 2024-03-19 03:30:02.252964871 +0000 UTC m=+0.155939919 container create 4dd4662cf2814d8597370f913f3d6ecd84044e393a4dbb0ad7de07b3650f7701 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-19T03:30:02.792 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:30:02.837 INFO:journalctl@ceph.mds.d.smithi117.stdout:Mar 19 03:30:02 smithi117 podman[72996]: 2024-03-19 03:30:02.565566497 +0000 UTC m=+0.468541577 container init 4dd4662cf2814d8597370f913f3d6ecd84044e393a4dbb0ad7de07b3650f7701 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-19T03:30:02.837 INFO:journalctl@ceph.mds.d.smithi117.stdout:Mar 19 03:30:02 smithi117 podman[72996]: 2024-03-19 03:30:02.57514621 +0000 UTC m=+0.478121247 container start 4dd4662cf2814d8597370f913f3d6ecd84044e393a4dbb0ad7de07b3650f7701 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-d, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:30:02.837 INFO:journalctl@ceph.mds.d.smithi117.stdout:Mar 19 03:30:02 smithi117 ceph-mds[73015]: -- 172.21.15.117:0/3101305494 <== mon.0 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561e8d053a40 con 0x561e8d026800 2024-03-19T03:30:02.837 INFO:journalctl@ceph.mds.d.smithi117.stdout:Mar 19 03:30:02 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-d[73010]: starting mds.d at 2024-03-19T03:30:02.837 INFO:journalctl@ceph.mds.d.smithi117.stdout:Mar 19 03:30:02 smithi117 bash[72996]: 4dd4662cf2814d8597370f913f3d6ecd84044e393a4dbb0ad7de07b3650f7701 2024-03-19T03:30:02.837 INFO:journalctl@ceph.mds.d.smithi117.stdout:Mar 19 03:30:02 smithi117 systemd[1]: Started Ceph mds.d for a3a82456-e59f-11ee-95c9-87774f69a715. 2024-03-19T03:30:03.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.007+0000 7fddf6200640 1 -- 172.21.15.117:0/3685027357 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddf0101840 msgr2=0x7fddf0101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:03.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.007+0000 7fddf6200640 1 --2- 172.21.15.117:0/3685027357 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddf0101840 0x7fddf0101c60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdde8009ca0 tx=0x7fdde802c100 comp rx=0 tx=0).stop 2024-03-19T03:30:03.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.008+0000 7fddf6200640 1 -- 172.21.15.117:0/3685027357 shutdown_connections 2024-03-19T03:30:03.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.008+0000 7fddf6200640 1 --2- 172.21.15.117:0/3685027357 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fddf01033f0 0x7fddf0109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:03.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.008+0000 7fddf6200640 1 --2- 172.21.15.117:0/3685027357 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fddf0102a30 0x7fddf0102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:03.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.008+0000 7fddf6200640 1 --2- 172.21.15.117:0/3685027357 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddf0101840 0x7fddf0101c60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:03.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.008+0000 7fddf6200640 1 -- 172.21.15.117:0/3685027357 >> 172.21.15.117:0/3685027357 conn(0x7fddf00fcdd0 msgr2=0x7fddf00ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:03.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.008+0000 7fddf6200640 1 -- 172.21.15.117:0/3685027357 shutdown_connections 2024-03-19T03:30:03.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.008+0000 7fddf6200640 1 -- 172.21.15.117:0/3685027357 wait complete. 2024-03-19T03:30:03.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.009+0000 7fddf6200640 1 Processor -- start 2024-03-19T03:30:03.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.009+0000 7fddf6200640 1 -- start start 2024-03-19T03:30:03.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.009+0000 7fddf6200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fddf0101840 0x7fddf01075c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:03.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.009+0000 7fddf6200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fddf0102a30 0x7fddf0107b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:03.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.009+0000 7fddf6200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddf01033f0 0x7fddf0104130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:03.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.009+0000 7fddf6200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fddf010d4c0 con 0x7fddf0102a30 2024-03-19T03:30:03.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.009+0000 7fddf6200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fddf010d340 con 0x7fddf01033f0 2024-03-19T03:30:03.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.009+0000 7fddf6200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fddf010d640 con 0x7fddf0101840 2024-03-19T03:30:03.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.009+0000 7fddefe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddf01033f0 0x7fddf0104130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:03.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.009+0000 7fddefe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddf01033f0 0x7fddf0104130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:49752/0 (socket says 172.21.15.117:49752) 2024-03-19T03:30:03.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.009+0000 7fddefe00640 1 -- 172.21.15.117:0/2778439214 learned_addr learned my addr 172.21.15.117:0/2778439214 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:03.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.009+0000 7fddeea00640 1 --2- 172.21.15.117:0/2778439214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fddf0102a30 0x7fddf0107b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:03.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.009+0000 7fddef400640 1 --2- 172.21.15.117:0/2778439214 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fddf0101840 0x7fddf01075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:03.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.010+0000 7fddef400640 1 -- 172.21.15.117:0/2778439214 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddf01033f0 msgr2=0x7fddf0104130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:03.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.010+0000 7fddef400640 1 --2- 172.21.15.117:0/2778439214 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddf01033f0 0x7fddf0104130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:03.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.010+0000 7fddef400640 1 -- 172.21.15.117:0/2778439214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fddf0102a30 msgr2=0x7fddf0107b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:03.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.010+0000 7fddef400640 1 --2- 172.21.15.117:0/2778439214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fddf0102a30 0x7fddf0107b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:03.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.010+0000 7fddef400640 1 -- 172.21.15.117:0/2778439214 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddf01049f0 con 0x7fddf0101840 2024-03-19T03:30:03.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.010+0000 7fddefe00640 1 --2- 172.21.15.117:0/2778439214 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddf01033f0 0x7fddf0104130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:30:03.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.010+0000 7fddef400640 1 --2- 172.21.15.117:0/2778439214 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fddf0101840 0x7fddf01075c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdde8037040 tx=0x7fdde8035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:03.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.010+0000 7fdde7e00640 1 -- 172.21.15.117:0/2778439214 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde8039040 con 0x7fddf0101840 2024-03-19T03:30:03.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.010+0000 7fdde7e00640 1 -- 172.21.15.117:0/2778439214 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdde8003970 con 0x7fddf0101840 2024-03-19T03:30:03.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.010+0000 7fddf6200640 1 -- 172.21.15.117:0/2778439214 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddf01a61a0 con 0x7fddf0101840 2024-03-19T03:30:03.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.010+0000 7fdde7e00640 1 -- 172.21.15.117:0/2778439214 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde80043c0 con 0x7fddf0101840 2024-03-19T03:30:03.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.011+0000 7fddf6200640 1 -- 172.21.15.117:0/2778439214 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddf01a6550 con 0x7fddf0101840 2024-03-19T03:30:03.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.012+0000 7fddf6200640 1 -- 172.21.15.117:0/2778439214 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddf01a6780 con 0x7fddf0101840 2024-03-19T03:30:03.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.013+0000 7fdde7e00640 1 -- 172.21.15.117:0/2778439214 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fdde8046030 con 0x7fddf0101840 2024-03-19T03:30:03.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.015+0000 7fdde7e00640 1 --2- 172.21.15.117:0/2778439214 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fddd40a2880 0x7fddd40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:03.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.015+0000 7fdde7e00640 1 -- 172.21.15.117:0/2778439214 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fdde80f0180 con 0x7fddf0101840 2024-03-19T03:30:03.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.015+0000 7fddeea00640 1 --2- 172.21.15.117:0/2778439214 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fddd40a2880 0x7fddd40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:03.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.015+0000 7fddeea00640 1 --2- 172.21.15.117:0/2778439214 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fddd40a2880 0x7fddd40a4d40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fddd80095f0 tx=0x7fddd8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:03.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.019+0000 7fdde7e00640 1 -- 172.21.15.117:0/2778439214 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdde804d060 con 0x7fddf0101840 2024-03-19T03:30:03.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.262+0000 7fddf6200640 1 -- 172.21.15.117:0/2778439214 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fddf01a6950 con 0x7fddf0101840 2024-03-19T03:30:03.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.263+0000 7fdde7e00640 1 -- 172.21.15.117:0/2778439214 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7fdde80bd6b0 con 0x7fddf0101840 2024-03-19T03:30:03.263 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:30:03.263 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/54638259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":54638259},{"type":"v1","addr":"172.21.15.117:6835","nonce":54638259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/1291833214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1291833214},{"type":"v1","addr":"172.21.15.119:6833","nonce":1291833214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-19T03:30:03.263 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 4 2024-03-19T03:30:03.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.265+0000 7fddf6200640 1 -- 172.21.15.117:0/2778439214 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fddd40a2880 msgr2=0x7fddd40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:03.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.265+0000 7fddf6200640 1 --2- 172.21.15.117:0/2778439214 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fddd40a2880 0x7fddd40a4d40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fddd80095f0 tx=0x7fddd8005f90 comp rx=0 tx=0).stop 2024-03-19T03:30:03.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.265+0000 7fddf6200640 1 -- 172.21.15.117:0/2778439214 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fddf0101840 msgr2=0x7fddf01075c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:03.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.265+0000 7fddf6200640 1 --2- 172.21.15.117:0/2778439214 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fddf0101840 0x7fddf01075c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdde8037040 tx=0x7fdde8035d10 comp rx=0 tx=0).stop 2024-03-19T03:30:03.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.266+0000 7fddf6200640 1 -- 172.21.15.117:0/2778439214 shutdown_connections 2024-03-19T03:30:03.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.266+0000 7fddf6200640 1 --2- 172.21.15.117:0/2778439214 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fddd40a2880 0x7fddd40a4d40 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:03.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.266+0000 7fddf6200640 1 --2- 172.21.15.117:0/2778439214 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddf01033f0 0x7fddf0104130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:03.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.266+0000 7fddf6200640 1 --2- 172.21.15.117:0/2778439214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fddf0102a30 0x7fddf0107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:03.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.266+0000 7fddf6200640 1 --2- 172.21.15.117:0/2778439214 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fddf0101840 0x7fddf01075c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:03.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.266+0000 7fddf6200640 1 -- 172.21.15.117:0/2778439214 >> 172.21.15.117:0/2778439214 conn(0x7fddf00fcdd0 msgr2=0x7fddf0073a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:03.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.266+0000 7fddf6200640 1 -- 172.21.15.117:0/2778439214 shutdown_connections 2024-03-19T03:30:03.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:03.266+0000 7fddf6200640 1 -- 172.21.15.117:0/2778439214 wait complete. 2024-03-19T03:30:03.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:03 smithi117 ceph-mon[31838]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:03.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:03 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:03.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:03 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:03.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:03 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:03.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:03 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:30:03.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:03 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:30:03.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:03 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:03.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:03 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2778439214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:03.622 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:03 smithi119 ceph-mon[66861]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:03.622 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:03 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:03.622 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:03 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:03.622 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:03 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:03.622 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:03 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:30:03.622 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:03 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:30:03.622 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:03 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:03.622 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:03 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2778439214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:03.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:03 smithi178 ceph-mon[67578]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:03.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:03 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:03.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:03 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:03.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:03 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:03.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:03 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:30:03.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:03 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:30:03.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:03 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:03.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:03 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2778439214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:04.638 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:04 smithi119 ceph-mon[66861]: Deploying daemon mds.e on smithi119 2024-03-19T03:30:04.638 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:04 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.117:6836/3608185039,v1:172.21.15.117:6837/3608185039] up:boot 2024-03-19T03:30:04.638 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:04 smithi119 ceph-mon[66861]: fsmap 4 up:standby 2024-03-19T03:30:04.638 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:04 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-19T03:30:04.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:04 smithi117 ceph-mon[31838]: Deploying daemon mds.e on smithi119 2024-03-19T03:30:04.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:04 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.117:6836/3608185039,v1:172.21.15.117:6837/3608185039] up:boot 2024-03-19T03:30:04.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:04 smithi117 ceph-mon[31838]: fsmap 4 up:standby 2024-03-19T03:30:04.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:04 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-19T03:30:04.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:04 smithi178 ceph-mon[67578]: Deploying daemon mds.e on smithi119 2024-03-19T03:30:04.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:04 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.117:6836/3608185039,v1:172.21.15.117:6837/3608185039] up:boot 2024-03-19T03:30:04.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:04 smithi178 ceph-mon[67578]: fsmap 4 up:standby 2024-03-19T03:30:04.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:04 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-19T03:30:05.234 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 19 03:30:05 smithi119 systemd[1]: Starting Ceph mds.e for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:30:05.284 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:30:05.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.499+0000 7f8e03200640 1 -- 172.21.15.117:0/3947556025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e04105420 msgr2=0x7f8e0410bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:05.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.499+0000 7f8e03200640 1 --2- 172.21.15.117:0/3947556025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e04105420 0x7f8e0410bcb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc009900 tx=0x7f8dfc02bc30 comp rx=0 tx=0).stop 2024-03-19T03:30:05.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.499+0000 7f8e03200640 1 -- 172.21.15.117:0/3947556025 shutdown_connections 2024-03-19T03:30:05.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.499+0000 7f8e03200640 1 --2- 172.21.15.117:0/3947556025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e04105420 0x7f8e0410bcb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:05.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.499+0000 7f8e03200640 1 --2- 172.21.15.117:0/3947556025 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8e04104a60 0x7f8e04104ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:05.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.499+0000 7f8e03200640 1 --2- 172.21.15.117:0/3947556025 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e04103870 0x7f8e04103c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:05.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.499+0000 7f8e03200640 1 -- 172.21.15.117:0/3947556025 >> 172.21.15.117:0/3947556025 conn(0x7f8e040fee00 msgr2=0x7f8e04101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:05.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.500+0000 7f8e03200640 1 -- 172.21.15.117:0/3947556025 shutdown_connections 2024-03-19T03:30:05.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.500+0000 7f8e03200640 1 -- 172.21.15.117:0/3947556025 wait complete. 2024-03-19T03:30:05.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.500+0000 7f8e03200640 1 Processor -- start 2024-03-19T03:30:05.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.501+0000 7f8e03200640 1 -- start start 2024-03-19T03:30:05.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.501+0000 7f8e03200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e04103870 0x7f8e04199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:05.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.501+0000 7f8e03200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e04104a60 0x7f8e0419a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:05.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.501+0000 7f8e03200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8e04105420 0x7f8e041a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:05.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.501+0000 7f8e03200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0410f540 con 0x7f8e04104a60 2024-03-19T03:30:05.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.501+0000 7f8e03200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0410f3c0 con 0x7f8e04103870 2024-03-19T03:30:05.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.501+0000 7f8e03200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0410f6c0 con 0x7f8e04105420 2024-03-19T03:30:05.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.501+0000 7f8e01e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e04103870 0x7f8e04199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:05.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.501+0000 7f8df9400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e04104a60 0x7f8e0419a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:05.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.501+0000 7f8e01e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e04103870 0x7f8e04199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:49770/0 (socket says 172.21.15.117:49770) 2024-03-19T03:30:05.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.501+0000 7f8e01e00640 1 -- 172.21.15.117:0/1518004901 learned_addr learned my addr 172.21.15.117:0/1518004901 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:05.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.501+0000 7f8e02800640 1 --2- 172.21.15.117:0/1518004901 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8e04105420 0x7f8e041a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:05.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.501+0000 7f8e01e00640 1 -- 172.21.15.117:0/1518004901 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8e04105420 msgr2=0x7f8e041a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:05.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.501+0000 7f8e01e00640 1 --2- 172.21.15.117:0/1518004901 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8e04105420 0x7f8e041a10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:05.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.501+0000 7f8e01e00640 1 -- 172.21.15.117:0/1518004901 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e04104a60 msgr2=0x7f8e0419a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:05.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.501+0000 7f8e01e00640 1 --2- 172.21.15.117:0/1518004901 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e04104a60 0x7f8e0419a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:05.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.501+0000 7f8e01e00640 1 -- 172.21.15.117:0/1518004901 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e041a1740 con 0x7f8e04103870 2024-03-19T03:30:05.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.502+0000 7f8e01e00640 1 --2- 172.21.15.117:0/1518004901 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e04103870 0x7f8e04199b20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8df4009d20 tx=0x7f8df400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:05.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.502+0000 7f8dfb400640 1 -- 172.21.15.117:0/1518004901 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df4010630 con 0x7f8e04103870 2024-03-19T03:30:05.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.502+0000 7f8dfb400640 1 -- 172.21.15.117:0/1518004901 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8df4010cb0 con 0x7f8e04103870 2024-03-19T03:30:05.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.502+0000 7f8e03200640 1 -- 172.21.15.117:0/1518004901 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e041a1a30 con 0x7f8e04103870 2024-03-19T03:30:05.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.502+0000 7f8dfb400640 1 -- 172.21.15.117:0/1518004901 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df4018de0 con 0x7f8e04103870 2024-03-19T03:30:05.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.502+0000 7f8e03200640 1 -- 172.21.15.117:0/1518004901 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e041a1fd0 con 0x7f8e04103870 2024-03-19T03:30:05.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.503+0000 7f8e03200640 1 -- 172.21.15.117:0/1518004901 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e0410aac0 con 0x7f8e04103870 2024-03-19T03:30:05.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.505+0000 7f8dfb400640 1 -- 172.21.15.117:0/1518004901 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f8df401f020 con 0x7f8e04103870 2024-03-19T03:30:05.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.506+0000 7f8dfb400640 1 --2- 172.21.15.117:0/1518004901 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8de80a2880 0x7f8de80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:05.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.506+0000 7f8dfb400640 1 -- 172.21.15.117:0/1518004901 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8df40cea30 con 0x7f8e04103870 2024-03-19T03:30:05.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.506+0000 7f8df9400640 1 --2- 172.21.15.117:0/1518004901 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8de80a2880 0x7f8de80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:05.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.507+0000 7f8df9400640 1 --2- 172.21.15.117:0/1518004901 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8de80a2880 0x7f8de80a4d40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8e0419b030 tx=0x7f8dec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:05.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.510+0000 7f8dfb400640 1 -- 172.21.15.117:0/1518004901 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8df409bed0 con 0x7f8e04103870 2024-03-19T03:30:05.626 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:05 smithi119 ceph-mon[66861]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:05.627 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 19 03:30:05 smithi119 podman[96386]: 2024-03-19 03:30:05.23750335 +0000 UTC m=+0.018861687 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:30:05.627 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 19 03:30:05 smithi119 podman[96386]: 2024-03-19 03:30:05.356912773 +0000 UTC m=+0.138271139 container create 0a8ea9d7e2a159b06da19404d30704eeabaef44b4f412d6f24cef6389893e859 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-e, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-19T03:30:05.627 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 19 03:30:05 smithi119 podman[96386]: 2024-03-19 03:30:05.626509203 +0000 UTC m=+0.407867537 container init 0a8ea9d7e2a159b06da19404d30704eeabaef44b4f412d6f24cef6389893e859 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-19T03:30:05.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.744+0000 7f8e03200640 1 -- 172.21.15.117:0/1518004901 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e0402d1f0 con 0x7f8e04103870 2024-03-19T03:30:05.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.745+0000 7f8dfb400640 1 -- 172.21.15.117:0/1518004901 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3874 (secure 0 0 0) 0x7f8df409b7d0 con 0x7f8e04103870 2024-03-19T03:30:05.745 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:30:05.745 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/54638259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":54638259},{"type":"v1","addr":"172.21.15.117:6835","nonce":54638259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3608185039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3608185039},{"type":"v1","addr":"172.21.15.117:6837","nonce":3608185039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/1291833214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1291833214},{"type":"v1","addr":"172.21.15.119:6833","nonce":1291833214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-19T03:30:05.745 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 5 2024-03-19T03:30:05.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.748+0000 7f8e03200640 1 -- 172.21.15.117:0/1518004901 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8de80a2880 msgr2=0x7f8de80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:05.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.748+0000 7f8e03200640 1 --2- 172.21.15.117:0/1518004901 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8de80a2880 0x7f8de80a4d40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8e0419b030 tx=0x7f8dec005f90 comp rx=0 tx=0).stop 2024-03-19T03:30:05.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.748+0000 7f8e03200640 1 -- 172.21.15.117:0/1518004901 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e04103870 msgr2=0x7f8e04199b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:05.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.748+0000 7f8e03200640 1 --2- 172.21.15.117:0/1518004901 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e04103870 0x7f8e04199b20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8df4009d20 tx=0x7f8df400c6a0 comp rx=0 tx=0).stop 2024-03-19T03:30:05.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.748+0000 7f8e03200640 1 -- 172.21.15.117:0/1518004901 shutdown_connections 2024-03-19T03:30:05.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.748+0000 7f8e03200640 1 --2- 172.21.15.117:0/1518004901 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8e04105420 0x7f8e041a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:05.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.748+0000 7f8e03200640 1 --2- 172.21.15.117:0/1518004901 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e04104a60 0x7f8e0419a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:05.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.748+0000 7f8e03200640 1 --2- 172.21.15.117:0/1518004901 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8de80a2880 0x7f8de80a4d40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:05.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.748+0000 7f8e03200640 1 --2- 172.21.15.117:0/1518004901 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e04103870 0x7f8e04199b20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:05.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.748+0000 7f8e03200640 1 -- 172.21.15.117:0/1518004901 >> 172.21.15.117:0/1518004901 conn(0x7f8e040fee00 msgr2=0x7f8e04107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:05.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.748+0000 7f8e03200640 1 -- 172.21.15.117:0/1518004901 shutdown_connections 2024-03-19T03:30:05.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:05.748+0000 7f8e03200640 1 -- 172.21.15.117:0/1518004901 wait complete. 2024-03-19T03:30:05.763 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:05 smithi117 ceph-mon[31838]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:05.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:05 smithi178 ceph-mon[67578]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:05.984 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 19 03:30:05 smithi119 podman[96386]: 2024-03-19 03:30:05.629341169 +0000 UTC m=+0.410699497 container start 0a8ea9d7e2a159b06da19404d30704eeabaef44b4f412d6f24cef6389893e859 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-e, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-19T03:30:05.984 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 19 03:30:05 smithi119 ceph-mds[96406]: -- 172.21.15.119:0/956489638 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55de94a85a40 con 0x55de94a58c00 2024-03-19T03:30:05.984 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 19 03:30:05 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-e[96402]: starting mds.e at 2024-03-19T03:30:05.984 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 19 03:30:05 smithi119 bash[96386]: 0a8ea9d7e2a159b06da19404d30704eeabaef44b4f412d6f24cef6389893e859 2024-03-19T03:30:05.984 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 19 03:30:05 smithi119 systemd[1]: Started Ceph mds.e for a3a82456-e59f-11ee-95c9-87774f69a715. 2024-03-19T03:30:06.706 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:06 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1518004901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:06.706 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:06 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:06.706 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:06 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:06.706 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:06 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:06.706 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:06 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:30:06.706 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:06 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:30:06.706 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:06 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:06.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:06 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1518004901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:06.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:06 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:06.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:06 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:06.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:06 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:06.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:06 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:30:06.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:06 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:30:06.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:06 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:06.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:06 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1518004901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:06.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:06 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:06.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:06 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:06.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:06 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:06.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:06 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:30:06.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:06 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:30:06.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:06 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:07.695 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:07 smithi117 ceph-mon[31838]: Deploying daemon mds.f on smithi178 2024-03-19T03:30:07.695 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:07 smithi117 ceph-mon[31838]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:07.695 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:07 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.119:6834/1686938941,v1:172.21.15.119:6835/1686938941] up:boot 2024-03-19T03:30:07.695 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:07 smithi117 ceph-mon[31838]: fsmap 5 up:standby 2024-03-19T03:30:07.696 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:07 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T03:30:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:07 smithi119 ceph-mon[66861]: Deploying daemon mds.f on smithi178 2024-03-19T03:30:07.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:07 smithi119 ceph-mon[66861]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:07.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:07 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.119:6834/1686938941,v1:172.21.15.119:6835/1686938941] up:boot 2024-03-19T03:30:07.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:07 smithi119 ceph-mon[66861]: fsmap 5 up:standby 2024-03-19T03:30:07.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:07 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T03:30:07.766 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:30:07.822 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:07 smithi178 ceph-mon[67578]: Deploying daemon mds.f on smithi178 2024-03-19T03:30:07.822 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:07 smithi178 ceph-mon[67578]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:07.822 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:07 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.119:6834/1686938941,v1:172.21.15.119:6835/1686938941] up:boot 2024-03-19T03:30:07.822 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:07 smithi178 ceph-mon[67578]: fsmap 5 up:standby 2024-03-19T03:30:07.822 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:07 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T03:30:07.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.976+0000 7f5019a00640 1 -- 172.21.15.117:0/2792477213 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5014104c50 msgr2=0x7f50141050d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:07.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.976+0000 7f5019a00640 1 --2- 172.21.15.117:0/2792477213 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5014104c50 0x7f50141050d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc009900 tx=0x7f4ffc02bc30 comp rx=0 tx=0).stop 2024-03-19T03:30:07.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.976+0000 7f5019a00640 1 -- 172.21.15.117:0/2792477213 shutdown_connections 2024-03-19T03:30:07.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.976+0000 7f5019a00640 1 --2- 172.21.15.117:0/2792477213 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5014105610 0x7f501410bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:07.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.976+0000 7f5019a00640 1 --2- 172.21.15.117:0/2792477213 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5014104c50 0x7f50141050d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:07.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.976+0000 7f5019a00640 1 --2- 172.21.15.117:0/2792477213 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5014103a60 0x7f5014103e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:07.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.976+0000 7f5019a00640 1 -- 172.21.15.117:0/2792477213 >> 172.21.15.117:0/2792477213 conn(0x7f50140fefd0 msgr2=0x7f5014101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:07.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.976+0000 7f5019a00640 1 -- 172.21.15.117:0/2792477213 shutdown_connections 2024-03-19T03:30:07.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.976+0000 7f5019a00640 1 -- 172.21.15.117:0/2792477213 wait complete. 2024-03-19T03:30:07.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.977+0000 7f5019a00640 1 Processor -- start 2024-03-19T03:30:07.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.978+0000 7f5019a00640 1 -- start start 2024-03-19T03:30:07.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.978+0000 7f5019a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5014103a60 0x7f5014199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:07.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.978+0000 7f5019a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5014104c50 0x7f501419a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:07.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.978+0000 7f5019a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5014105610 0x7f50141a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:07.979 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.978+0000 7f5019a00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f501410f730 con 0x7f5014105610 2024-03-19T03:30:07.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.978+0000 7f5019a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f501410f5b0 con 0x7f5014103a60 2024-03-19T03:30:07.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.978+0000 7f5019a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f501410f8b0 con 0x7f5014104c50 2024-03-19T03:30:07.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.978+0000 7f5013400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5014105610 0x7f50141a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:07.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.978+0000 7f5013400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5014105610 0x7f50141a1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:37640/0 (socket says 172.21.15.117:37640) 2024-03-19T03:30:07.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.978+0000 7f5013400640 1 -- 172.21.15.117:0/3105270009 learned_addr learned my addr 172.21.15.117:0/3105270009 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:07.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.978+0000 7f5012a00640 1 --2- 172.21.15.117:0/3105270009 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5014103a60 0x7f5014199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:07.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.978+0000 7f500be00640 1 --2- 172.21.15.117:0/3105270009 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5014104c50 0x7f501419a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:07.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.979+0000 7f5013400640 1 -- 172.21.15.117:0/3105270009 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5014104c50 msgr2=0x7f501419a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:07.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.979+0000 7f5013400640 1 --2- 172.21.15.117:0/3105270009 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5014104c50 0x7f501419a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:07.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.979+0000 7f5013400640 1 -- 172.21.15.117:0/3105270009 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5014103a60 msgr2=0x7f5014199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:07.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.979+0000 7f5013400640 1 --2- 172.21.15.117:0/3105270009 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5014103a60 0x7f5014199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:07.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.979+0000 7f5013400640 1 -- 172.21.15.117:0/3105270009 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50141a1990 con 0x7f5014105610 2024-03-19T03:30:07.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.979+0000 7f5013400640 1 --2- 172.21.15.117:0/3105270009 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5014105610 0x7f50141a1320 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f500c00eb10 tx=0x7f500c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:07.980 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.979+0000 7f500b400640 1 -- 172.21.15.117:0/3105270009 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c004b20 con 0x7f5014105610 2024-03-19T03:30:07.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.979+0000 7f500b400640 1 -- 172.21.15.117:0/3105270009 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f500c004cc0 con 0x7f5014105610 2024-03-19T03:30:07.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.979+0000 7f5019a00640 1 -- 172.21.15.117:0/3105270009 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50141a1c80 con 0x7f5014105610 2024-03-19T03:30:07.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.979+0000 7f500b400640 1 -- 172.21.15.117:0/3105270009 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c010760 con 0x7f5014105610 2024-03-19T03:30:07.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.979+0000 7f5019a00640 1 -- 172.21.15.117:0/3105270009 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50141a21a0 con 0x7f5014105610 2024-03-19T03:30:07.981 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.980+0000 7f5019a00640 1 -- 172.21.15.117:0/3105270009 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f501410ad80 con 0x7f5014105610 2024-03-19T03:30:07.982 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.983+0000 7f500b400640 1 -- 172.21.15.117:0/3105270009 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f500c005020 con 0x7f5014105610 2024-03-19T03:30:07.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.984+0000 7f500b400640 1 --2- 172.21.15.117:0/3105270009 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4ff80a2880 0x7f4ff80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:07.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.984+0000 7f500b400640 1 -- 172.21.15.117:0/3105270009 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f500c0ceac0 con 0x7f5014105610 2024-03-19T03:30:07.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.984+0000 7f5012a00640 1 --2- 172.21.15.117:0/3105270009 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4ff80a2880 0x7f4ff80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:07.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.985+0000 7f5012a00640 1 --2- 172.21.15.117:0/3105270009 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4ff80a2880 0x7f4ff80a4d40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5014104ab0 tx=0x7f5004005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:07.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:07.989+0000 7f500b400640 1 -- 172.21.15.117:0/3105270009 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f500c09bfe0 con 0x7f5014105610 2024-03-19T03:30:08.096 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 19 03:30:08 smithi178 systemd[1]: Starting Ceph mds.f for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:30:08.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:08.227+0000 7f5019a00640 1 -- 172.21.15.117:0/3105270009 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f501402d1f0 con 0x7f5014105610 2024-03-19T03:30:08.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:08.228+0000 7f500b400640 1 -- 172.21.15.117:0/3105270009 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4702 (secure 0 0 0) 0x7f500c09b8e0 con 0x7f5014105610 2024-03-19T03:30:08.229 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:30:08.229 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/54638259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":54638259},{"type":"v1","addr":"172.21.15.117:6835","nonce":54638259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3608185039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3608185039},{"type":"v1","addr":"172.21.15.117:6837","nonce":3608185039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/1291833214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1291833214},{"type":"v1","addr":"172.21.15.119:6833","nonce":1291833214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24515,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/1686938941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1686938941},{"type":"v1","addr":"172.21.15.119:6835","nonce":1686938941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-19T03:30:08.229 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 6 2024-03-19T03:30:08.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:08.231+0000 7f5019a00640 1 -- 172.21.15.117:0/3105270009 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4ff80a2880 msgr2=0x7f4ff80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:08.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:08.231+0000 7f5019a00640 1 --2- 172.21.15.117:0/3105270009 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4ff80a2880 0x7f4ff80a4d40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5014104ab0 tx=0x7f5004005f90 comp rx=0 tx=0).stop 2024-03-19T03:30:08.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:08.231+0000 7f5019a00640 1 -- 172.21.15.117:0/3105270009 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5014105610 msgr2=0x7f50141a1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:08.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:08.231+0000 7f5019a00640 1 --2- 172.21.15.117:0/3105270009 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5014105610 0x7f50141a1320 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f500c00eb10 tx=0x7f500c00c2d0 comp rx=0 tx=0).stop 2024-03-19T03:30:08.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:08.231+0000 7f5019a00640 1 -- 172.21.15.117:0/3105270009 shutdown_connections 2024-03-19T03:30:08.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:08.231+0000 7f5019a00640 1 --2- 172.21.15.117:0/3105270009 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5014105610 0x7f50141a1320 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:08.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:08.231+0000 7f5019a00640 1 --2- 172.21.15.117:0/3105270009 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5014104c50 0x7f501419a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:08.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:08.231+0000 7f5019a00640 1 --2- 172.21.15.117:0/3105270009 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4ff80a2880 0x7f4ff80a4d40 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:08.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:08.231+0000 7f5019a00640 1 --2- 172.21.15.117:0/3105270009 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5014103a60 0x7f5014199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:08.232 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:08.231+0000 7f5019a00640 1 -- 172.21.15.117:0/3105270009 >> 172.21.15.117:0/3105270009 conn(0x7f50140fefd0 msgr2=0x7f5014107f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:08.232 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:08.231+0000 7f5019a00640 1 -- 172.21.15.117:0/3105270009 shutdown_connections 2024-03-19T03:30:08.232 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:08.231+0000 7f5019a00640 1 -- 172.21.15.117:0/3105270009 wait complete. 2024-03-19T03:30:08.491 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:08 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3105270009' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:08.491 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 19 03:30:08 smithi178 podman[98116]: 2024-03-19 03:30:08.122113849 +0000 UTC m=+0.017632998 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:30:08.491 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 19 03:30:08 smithi178 podman[98116]: 2024-03-19 03:30:08.230240269 +0000 UTC m=+0.125759397 container create 21115f9198fa374a76bfcf782607f6ccc4ab77ba39e5953e08ea9f62920b1b2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-f, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-19T03:30:08.491 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 19 03:30:08 smithi178 podman[98116]: 2024-03-19 03:30:08.491336451 +0000 UTC m=+0.386855586 container init 21115f9198fa374a76bfcf782607f6ccc4ab77ba39e5953e08ea9f62920b1b2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-f, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T03:30:08.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:08 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3105270009' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:08.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:08 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3105270009' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:08.845 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 19 03:30:08 smithi178 podman[98116]: 2024-03-19 03:30:08.494133335 +0000 UTC m=+0.389652474 container start 21115f9198fa374a76bfcf782607f6ccc4ab77ba39e5953e08ea9f62920b1b2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-f, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS) 2024-03-19T03:30:08.846 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 19 03:30:08 smithi178 ceph-mds[98135]: -- 172.21.15.178:0/7599967 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5625ab297a40 con 0x5625ab26b000 2024-03-19T03:30:08.846 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 19 03:30:08 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-f[98131]: starting mds.f at 2024-03-19T03:30:08.846 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 19 03:30:08 smithi178 bash[98116]: 21115f9198fa374a76bfcf782607f6ccc4ab77ba39e5953e08ea9f62920b1b2f 2024-03-19T03:30:08.846 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 19 03:30:08 smithi178 systemd[1]: Started Ceph mds.f for a3a82456-e59f-11ee-95c9-87774f69a715. 2024-03-19T03:30:09.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:09 smithi178 ceph-mon[67578]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:09.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:09 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:09.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:09 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:09.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:09 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:09.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:09 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:30:09.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:09 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:30:09.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:09 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:09.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:09 smithi119 ceph-mon[66861]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:09.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:09 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:09.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:09 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:09.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:09 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:09.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:09 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:30:09.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:09 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:30:09.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:09 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:09.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:09 smithi117 ceph-mon[31838]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:09.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:09 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:09.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:09 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:09.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:09 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:09.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:09 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:30:09.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:09 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:30:09.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:09 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:10.249 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:30:10.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.469+0000 7f0f1b800640 1 -- 172.21.15.117:0/669560708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f14108fb0 msgr2=0x7f0f14109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:10.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.469+0000 7f0f1b800640 1 --2- 172.21.15.117:0/669560708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f14108fb0 0x7f0f14109430 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0f00009860 tx=0x7f0f0002bb70 comp rx=0 tx=0).stop 2024-03-19T03:30:10.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.470+0000 7f0f1b800640 1 -- 172.21.15.117:0/669560708 shutdown_connections 2024-03-19T03:30:10.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.470+0000 7f0f1b800640 1 --2- 172.21.15.117:0/669560708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f14109970 0x7f0f14110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:10.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.470+0000 7f0f1b800640 1 --2- 172.21.15.117:0/669560708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f14108fb0 0x7f0f14109430 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:10.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.470+0000 7f0f1b800640 1 --2- 172.21.15.117:0/669560708 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f14107e60 0x7f0f14108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:10.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.470+0000 7f0f1b800640 1 -- 172.21.15.117:0/669560708 >> 172.21.15.117:0/669560708 conn(0x7f0f14076e30 msgr2=0x7f0f14079250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:10.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.470+0000 7f0f1b800640 1 -- 172.21.15.117:0/669560708 shutdown_connections 2024-03-19T03:30:10.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.470+0000 7f0f1b800640 1 -- 172.21.15.117:0/669560708 wait complete. 2024-03-19T03:30:10.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.471+0000 7f0f1b800640 1 Processor -- start 2024-03-19T03:30:10.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.471+0000 7f0f1b800640 1 -- start start 2024-03-19T03:30:10.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.471+0000 7f0f1b800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f14107e60 0x7f0f1419e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:10.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.471+0000 7f0f1b800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f14108fb0 0x7f0f1419e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:10.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.471+0000 7f0f1b800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f14109970 0x7f0f141a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:10.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.471+0000 7f0f1b800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0f14113ae0 con 0x7f0f14108fb0 2024-03-19T03:30:10.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.471+0000 7f0f1b800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0f14113960 con 0x7f0f14107e60 2024-03-19T03:30:10.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.471+0000 7f0f1b800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f0f14113c60 con 0x7f0f14109970 2024-03-19T03:30:10.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.472+0000 7f0f13e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f14108fb0 0x7f0f1419e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:10.473 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.472+0000 7f0f18e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f14107e60 0x7f0f1419e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:10.473 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.472+0000 7f0f18e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f14107e60 0x7f0f1419e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:60000/0 (socket says 172.21.15.117:60000) 2024-03-19T03:30:10.473 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.472+0000 7f0f19800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f14109970 0x7f0f141a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:10.473 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.472+0000 7f0f18e00640 1 -- 172.21.15.117:0/4121021235 learned_addr learned my addr 172.21.15.117:0/4121021235 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:10.473 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.472+0000 7f0f13e00640 1 -- 172.21.15.117:0/4121021235 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f14109970 msgr2=0x7f0f141a5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:10.473 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.472+0000 7f0f13e00640 1 --2- 172.21.15.117:0/4121021235 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f14109970 0x7f0f141a5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:10.473 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.472+0000 7f0f13e00640 1 -- 172.21.15.117:0/4121021235 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f14107e60 msgr2=0x7f0f1419e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:10.473 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.472+0000 7f0f13e00640 1 --2- 172.21.15.117:0/4121021235 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f14107e60 0x7f0f1419e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:10.473 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.472+0000 7f0f13e00640 1 -- 172.21.15.117:0/4121021235 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f141a5d70 con 0x7f0f14108fb0 2024-03-19T03:30:10.473 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.472+0000 7f0f13e00640 1 --2- 172.21.15.117:0/4121021235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f14108fb0 0x7f0f1419e690 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0f00009780 tx=0x7f0f00032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:10.474 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.473+0000 7f0f11600640 1 -- 172.21.15.117:0/4121021235 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f00041020 con 0x7f0f14108fb0 2024-03-19T03:30:10.474 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.473+0000 7f0f1b800640 1 -- 172.21.15.117:0/4121021235 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f141a6000 con 0x7f0f14108fb0 2024-03-19T03:30:10.474 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.473+0000 7f0f11600640 1 -- 172.21.15.117:0/4121021235 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f0002c9f0 con 0x7f0f14108fb0 2024-03-19T03:30:10.474 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.473+0000 7f0f11600640 1 -- 172.21.15.117:0/4121021235 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f00039040 con 0x7f0f14108fb0 2024-03-19T03:30:10.474 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.473+0000 7f0f1b800640 1 -- 172.21.15.117:0/4121021235 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f141a6510 con 0x7f0f14108fb0 2024-03-19T03:30:10.474 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.474+0000 7f0f1b800640 1 -- 172.21.15.117:0/4121021235 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f1410f170 con 0x7f0f14108fb0 2024-03-19T03:30:10.475 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.476+0000 7f0f11600640 1 -- 172.21.15.117:0/4121021235 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f0f00044020 con 0x7f0f14108fb0 2024-03-19T03:30:10.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.477+0000 7f0f11600640 1 --2- 172.21.15.117:0/4121021235 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0efc0a2880 0x7f0efc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:10.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.477+0000 7f0f11600640 1 -- 172.21.15.117:0/4121021235 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f0f000f0190 con 0x7f0f14108fb0 2024-03-19T03:30:10.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.477+0000 7f0f18e00640 1 --2- 172.21.15.117:0/4121021235 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0efc0a2880 0x7f0efc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:10.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.478+0000 7f0f18e00640 1 --2- 172.21.15.117:0/4121021235 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0efc0a2880 0x7f0efc0a4d40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0f08005c80 tx=0x7f0f08009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:10.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.480+0000 7f0f11600640 1 -- 172.21.15.117:0/4121021235 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0f0004b020 con 0x7f0f14108fb0 2024-03-19T03:30:10.684 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:10 smithi178 ceph-mon[67578]: Deploying daemon mds.c on smithi178 2024-03-19T03:30:10.684 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:10 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.178:6834/1579860804,v1:172.21.15.178:6835/1579860804] up:boot 2024-03-19T03:30:10.684 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:10 smithi178 ceph-mon[67578]: fsmap 6 up:standby 2024-03-19T03:30:10.684 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:10 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T03:30:10.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.719+0000 7f0f1b800640 1 -- 172.21.15.117:0/4121021235 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f1402d440 con 0x7f0f14108fb0 2024-03-19T03:30:10.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.720+0000 7f0f11600640 1 -- 172.21.15.117:0/4121021235 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5530 (secure 0 0 0) 0x7f0f000bd6b0 con 0x7f0f14108fb0 2024-03-19T03:30:10.720 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:30:10.720 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/54638259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":54638259},{"type":"v1","addr":"172.21.15.117:6835","nonce":54638259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3608185039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3608185039},{"type":"v1","addr":"172.21.15.117:6837","nonce":3608185039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/1291833214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1291833214},{"type":"v1","addr":"172.21.15.119:6833","nonce":1291833214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1579860804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1579860804},{"type":"v1","addr":"172.21.15.178:6835","nonce":1579860804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24515,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/1686938941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1686938941},{"type":"v1","addr":"172.21.15.119:6835","nonce":1686938941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-19T03:30:10.721 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 7 2024-03-19T03:30:10.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.723+0000 7f0f1b800640 1 -- 172.21.15.117:0/4121021235 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0efc0a2880 msgr2=0x7f0efc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:10.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.723+0000 7f0f1b800640 1 --2- 172.21.15.117:0/4121021235 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0efc0a2880 0x7f0efc0a4d40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0f08005c80 tx=0x7f0f08009260 comp rx=0 tx=0).stop 2024-03-19T03:30:10.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.723+0000 7f0f1b800640 1 -- 172.21.15.117:0/4121021235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f14108fb0 msgr2=0x7f0f1419e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:10.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.723+0000 7f0f1b800640 1 --2- 172.21.15.117:0/4121021235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f14108fb0 0x7f0f1419e690 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0f00009780 tx=0x7f0f00032430 comp rx=0 tx=0).stop 2024-03-19T03:30:10.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.723+0000 7f0f1b800640 1 -- 172.21.15.117:0/4121021235 shutdown_connections 2024-03-19T03:30:10.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.723+0000 7f0f1b800640 1 --2- 172.21.15.117:0/4121021235 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f14109970 0x7f0f141a5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:10.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.723+0000 7f0f1b800640 1 --2- 172.21.15.117:0/4121021235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f14108fb0 0x7f0f1419e690 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:10.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.723+0000 7f0f1b800640 1 --2- 172.21.15.117:0/4121021235 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0efc0a2880 0x7f0efc0a4d40 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:10.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.723+0000 7f0f1b800640 1 --2- 172.21.15.117:0/4121021235 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f14107e60 0x7f0f1419e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:10.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.723+0000 7f0f1b800640 1 -- 172.21.15.117:0/4121021235 >> 172.21.15.117:0/4121021235 conn(0x7f0f14076e30 msgr2=0x7f0f1410c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:10.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.723+0000 7f0f1b800640 1 -- 172.21.15.117:0/4121021235 shutdown_connections 2024-03-19T03:30:10.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:10.723+0000 7f0f1b800640 1 -- 172.21.15.117:0/4121021235 wait complete. 2024-03-19T03:30:10.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:10 smithi119 ceph-mon[66861]: Deploying daemon mds.c on smithi178 2024-03-19T03:30:10.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:10 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.178:6834/1579860804,v1:172.21.15.178:6835/1579860804] up:boot 2024-03-19T03:30:10.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:10 smithi119 ceph-mon[66861]: fsmap 6 up:standby 2024-03-19T03:30:10.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:10 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T03:30:10.738 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:10 smithi117 ceph-mon[31838]: Deploying daemon mds.c on smithi178 2024-03-19T03:30:10.738 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:10 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.178:6834/1579860804,v1:172.21.15.178:6835/1579860804] up:boot 2024-03-19T03:30:10.738 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:10 smithi117 ceph-mon[31838]: fsmap 6 up:standby 2024-03-19T03:30:10.738 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:10 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T03:30:11.085 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 19 03:30:10 smithi178 systemd[1]: Starting Ceph mds.c for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:30:11.337 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 19 03:30:11 smithi178 podman[98426]: 2024-03-19 03:30:10.987492038 +0000 UTC m=+0.017774265 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:30:11.337 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 19 03:30:11 smithi178 podman[98426]: 2024-03-19 03:30:11.117750645 +0000 UTC m=+0.148032863 container create 5baaacaf8c4aeb6619a24dcf956a5026d682b459fbbe14d9fb454598fc9cc514 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-19T03:30:11.337 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 19 03:30:11 smithi178 podman[98426]: 2024-03-19 03:30:11.337267919 +0000 UTC m=+0.367550144 container init 5baaacaf8c4aeb6619a24dcf956a5026d682b459fbbe14d9fb454598fc9cc514 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-19T03:30:11.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:11 smithi178 ceph-mon[67578]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:11.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:11 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/4121021235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:11.596 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 19 03:30:11 smithi178 podman[98426]: 2024-03-19 03:30:11.34001534 +0000 UTC m=+0.370297574 container start 5baaacaf8c4aeb6619a24dcf956a5026d682b459fbbe14d9fb454598fc9cc514 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:30:11.596 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 19 03:30:11 smithi178 ceph-mds[98447]: -- 172.21.15.178:0/3368576175 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f6736fda40 con 0x55f6736d0800 2024-03-19T03:30:11.596 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 19 03:30:11 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-c[98443]: starting mds.c at 2024-03-19T03:30:11.596 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 19 03:30:11 smithi178 bash[98426]: 5baaacaf8c4aeb6619a24dcf956a5026d682b459fbbe14d9fb454598fc9cc514 2024-03-19T03:30:11.596 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 19 03:30:11 smithi178 systemd[1]: Started Ceph mds.c for a3a82456-e59f-11ee-95c9-87774f69a715. 2024-03-19T03:30:11.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:11 smithi119 ceph-mon[66861]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:11.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:11 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/4121021235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:11.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:11 smithi117 ceph-mon[31838]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:11.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:11 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/4121021235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:12.463 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:12 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.178:6836/1770913538,v1:172.21.15.178:6837/1770913538] up:boot 2024-03-19T03:30:12.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:12 smithi119 ceph-mon[66861]: fsmap 7 up:standby 2024-03-19T03:30:12.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:12 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T03:30:12.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:12 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:12.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:12 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:12.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:12 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:12.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:12 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:30:12.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:12 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:30:12.464 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:12 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:12.742 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:30:12.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:12 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.178:6836/1770913538,v1:172.21.15.178:6837/1770913538] up:boot 2024-03-19T03:30:12.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:12 smithi117 ceph-mon[31838]: fsmap 7 up:standby 2024-03-19T03:30:12.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:12 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T03:30:12.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:12 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:12.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:12 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:12.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:12 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:12.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:12 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:30:12.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:12 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:30:12.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:12 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:12.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:12 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.178:6836/1770913538,v1:172.21.15.178:6837/1770913538] up:boot 2024-03-19T03:30:12.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:12 smithi178 ceph-mon[67578]: fsmap 7 up:standby 2024-03-19T03:30:12.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:12 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T03:30:12.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:12 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:12.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:12 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:12.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:12 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:12.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:12 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:30:12.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:12 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:30:12.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:12 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:12.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.961+0000 7fd50ee00640 1 -- 172.21.15.117:0/4180285911 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd508103a60 msgr2=0x7fd508103e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:12.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.961+0000 7fd50ee00640 1 --2- 172.21.15.117:0/4180285911 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd508103a60 0x7fd508103e80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8009ca0 tx=0x7fd4f802c0f0 comp rx=0 tx=0).stop 2024-03-19T03:30:12.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.961+0000 7fd50ee00640 1 -- 172.21.15.117:0/4180285911 shutdown_connections 2024-03-19T03:30:12.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.961+0000 7fd50ee00640 1 --2- 172.21.15.117:0/4180285911 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd508105610 0x7fd50810bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:12.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.961+0000 7fd50ee00640 1 --2- 172.21.15.117:0/4180285911 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd508104c50 0x7fd5081050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:12.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.961+0000 7fd50ee00640 1 --2- 172.21.15.117:0/4180285911 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd508103a60 0x7fd508103e80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:12.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.961+0000 7fd50ee00640 1 -- 172.21.15.117:0/4180285911 >> 172.21.15.117:0/4180285911 conn(0x7fd5080fefd0 msgr2=0x7fd508101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:12.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.962+0000 7fd50ee00640 1 -- 172.21.15.117:0/4180285911 shutdown_connections 2024-03-19T03:30:12.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.962+0000 7fd50ee00640 1 -- 172.21.15.117:0/4180285911 wait complete. 2024-03-19T03:30:12.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.963+0000 7fd50ee00640 1 Processor -- start 2024-03-19T03:30:12.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.963+0000 7fd50ee00640 1 -- start start 2024-03-19T03:30:12.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.963+0000 7fd50ee00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd508103a60 0x7fd508199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:12.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.963+0000 7fd50ee00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd508104c50 0x7fd50819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:12.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.963+0000 7fd50ee00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd508105610 0x7fd5081a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:12.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.963+0000 7fd50ee00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd50810f6b0 con 0x7fd508104c50 2024-03-19T03:30:12.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.963+0000 7fd50ee00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd50810f530 con 0x7fd508105610 2024-03-19T03:30:12.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.964+0000 7fd50ee00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fd50810f830 con 0x7fd508103a60 2024-03-19T03:30:12.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.964+0000 7fd507400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd508104c50 0x7fd50819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:12.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.964+0000 7fd507e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd508103a60 0x7fd508199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:12.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.964+0000 7fd507e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd508103a60 0x7fd508199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:42426/0 (socket says 172.21.15.117:42426) 2024-03-19T03:30:12.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.964+0000 7fd507e00640 1 -- 172.21.15.117:0/1344521708 learned_addr learned my addr 172.21.15.117:0/1344521708 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:12.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.964+0000 7fd50ce00640 1 --2- 172.21.15.117:0/1344521708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd508105610 0x7fd5081a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:12.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.964+0000 7fd507e00640 1 -- 172.21.15.117:0/1344521708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd508105610 msgr2=0x7fd5081a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:12.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.964+0000 7fd507e00640 1 --2- 172.21.15.117:0/1344521708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd508105610 0x7fd5081a12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:12.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.964+0000 7fd507e00640 1 -- 172.21.15.117:0/1344521708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd508104c50 msgr2=0x7fd50819a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:12.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.964+0000 7fd507e00640 1 --2- 172.21.15.117:0/1344521708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd508104c50 0x7fd50819a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:12.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.964+0000 7fd507e00640 1 -- 172.21.15.117:0/1344521708 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5081a1950 con 0x7fd508103a60 2024-03-19T03:30:12.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.964+0000 7fd507e00640 1 --2- 172.21.15.117:0/1344521708 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd508103a60 0x7fd508199d40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8000f80 tx=0x7fd4f8032ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:12.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.965+0000 7fd504c00640 1 -- 172.21.15.117:0/1344521708 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f8041020 con 0x7fd508103a60 2024-03-19T03:30:12.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.965+0000 7fd504c00640 1 -- 172.21.15.117:0/1344521708 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4f8034cd0 con 0x7fd508103a60 2024-03-19T03:30:12.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.965+0000 7fd504c00640 1 -- 172.21.15.117:0/1344521708 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f8039040 con 0x7fd508103a60 2024-03-19T03:30:12.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.965+0000 7fd50ee00640 1 -- 172.21.15.117:0/1344521708 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5081a1be0 con 0x7fd508103a60 2024-03-19T03:30:12.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.965+0000 7fd50ee00640 1 -- 172.21.15.117:0/1344521708 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5081a20f0 con 0x7fd508103a60 2024-03-19T03:30:12.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.966+0000 7fd50ee00640 1 -- 172.21.15.117:0/1344521708 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd508068a60 con 0x7fd508103a60 2024-03-19T03:30:12.968 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.968+0000 7fd504c00640 1 -- 172.21.15.117:0/1344521708 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fd4f8044020 con 0x7fd508103a60 2024-03-19T03:30:12.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.969+0000 7fd504c00640 1 --2- 172.21.15.117:0/1344521708 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd4ec0a2880 0x7fd4ec0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:12.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.969+0000 7fd504c00640 1 -- 172.21.15.117:0/1344521708 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd4f80f0410 con 0x7fd508103a60 2024-03-19T03:30:12.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.969+0000 7fd507400640 1 --2- 172.21.15.117:0/1344521708 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd4ec0a2880 0x7fd4ec0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:12.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.970+0000 7fd507400640 1 --2- 172.21.15.117:0/1344521708 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd4ec0a2880 0x7fd4ec0a4d40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd4f40095f0 tx=0x7fd4f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:12.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:12.973+0000 7fd504c00640 1 -- 172.21.15.117:0/1344521708 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd4f804b020 con 0x7fd508103a60 2024-03-19T03:30:13.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:13.208+0000 7fd50ee00640 1 -- 172.21.15.117:0/1344521708 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd508062420 con 0x7fd508103a60 2024-03-19T03:30:13.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:13.209+0000 7fd504c00640 1 -- 172.21.15.117:0/1344521708 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6358 (secure 0 0 0) 0x7fd4f80348f0 con 0x7fd508103a60 2024-03-19T03:30:13.210 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:30:13.210 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/54638259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":54638259},{"type":"v1","addr":"172.21.15.117:6835","nonce":54638259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3608185039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3608185039},{"type":"v1","addr":"172.21.15.117:6837","nonce":3608185039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/1291833214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1291833214},{"type":"v1","addr":"172.21.15.119:6833","nonce":1291833214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1579860804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1579860804},{"type":"v1","addr":"172.21.15.178:6835","nonce":1579860804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24515,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/1686938941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1686938941},{"type":"v1","addr":"172.21.15.119:6835","nonce":1686938941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1770913538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":1770913538},{"type":"v1","addr":"172.21.15.178:6837","nonce":1770913538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-19T03:30:13.210 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 8 2024-03-19T03:30:13.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:13.212+0000 7fd50ee00640 1 -- 172.21.15.117:0/1344521708 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd4ec0a2880 msgr2=0x7fd4ec0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:13.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:13.212+0000 7fd50ee00640 1 --2- 172.21.15.117:0/1344521708 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd4ec0a2880 0x7fd4ec0a4d40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd4f40095f0 tx=0x7fd4f4005f90 comp rx=0 tx=0).stop 2024-03-19T03:30:13.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:13.212+0000 7fd50ee00640 1 -- 172.21.15.117:0/1344521708 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd508103a60 msgr2=0x7fd508199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:13.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:13.212+0000 7fd50ee00640 1 --2- 172.21.15.117:0/1344521708 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd508103a60 0x7fd508199d40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8000f80 tx=0x7fd4f8032ec0 comp rx=0 tx=0).stop 2024-03-19T03:30:13.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:13.212+0000 7fd50ee00640 1 -- 172.21.15.117:0/1344521708 shutdown_connections 2024-03-19T03:30:13.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:13.212+0000 7fd50ee00640 1 --2- 172.21.15.117:0/1344521708 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd4ec0a2880 0x7fd4ec0a4d40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:13.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:13.212+0000 7fd50ee00640 1 --2- 172.21.15.117:0/1344521708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd508105610 0x7fd5081a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:13.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:13.212+0000 7fd50ee00640 1 --2- 172.21.15.117:0/1344521708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd508104c50 0x7fd50819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:13.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:13.212+0000 7fd50ee00640 1 --2- 172.21.15.117:0/1344521708 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd508103a60 0x7fd508199d40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:13.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:13.212+0000 7fd50ee00640 1 -- 172.21.15.117:0/1344521708 >> 172.21.15.117:0/1344521708 conn(0x7fd5080fefd0 msgr2=0x7fd508107fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:13.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:13.212+0000 7fd50ee00640 1 -- 172.21.15.117:0/1344521708 shutdown_connections 2024-03-19T03:30:13.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:13.212+0000 7fd50ee00640 1 -- 172.21.15.117:0/1344521708 wait complete. 2024-03-19T03:30:13.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:13 smithi119 ceph-mon[66861]: Deploying daemon mds.h on smithi119 2024-03-19T03:30:13.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:13 smithi119 ceph-mon[66861]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:13.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:13 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1344521708' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:13.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:13 smithi117 ceph-mon[31838]: Deploying daemon mds.h on smithi119 2024-03-19T03:30:13.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:13 smithi117 ceph-mon[31838]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:13.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:13 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1344521708' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:13.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:13 smithi178 ceph-mon[67578]: Deploying daemon mds.h on smithi119 2024-03-19T03:30:13.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:13 smithi178 ceph-mon[67578]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:13.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:13 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1344521708' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:13.925 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 19 03:30:13 smithi119 systemd[1]: Starting Ceph mds.h for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:30:14.206 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 19 03:30:13 smithi119 podman[96695]: 2024-03-19 03:30:13.828472339 +0000 UTC m=+0.017975916 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:30:14.207 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 19 03:30:13 smithi119 podman[96695]: 2024-03-19 03:30:13.954098625 +0000 UTC m=+0.143602192 container create f0cc29854e2d57f6a1359c42b62395592430fb77051ac76851893d36d2903c80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-h, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-19T03:30:14.207 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 19 03:30:14 smithi119 podman[96695]: 2024-03-19 03:30:14.206845412 +0000 UTC m=+0.396348982 container init f0cc29854e2d57f6a1359c42b62395592430fb77051ac76851893d36d2903c80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-h, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True) 2024-03-19T03:30:14.484 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 19 03:30:14 smithi119 podman[96695]: 2024-03-19 03:30:14.20954542 +0000 UTC m=+0.399048994 container start f0cc29854e2d57f6a1359c42b62395592430fb77051ac76851893d36d2903c80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-19T03:30:14.484 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 19 03:30:14 smithi119 ceph-mds[96714]: -- 172.21.15.119:0/1145895283 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5599db553a40 con 0x5599db527000 2024-03-19T03:30:14.484 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 19 03:30:14 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-h[96710]: starting mds.h at 2024-03-19T03:30:14.484 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 19 03:30:14 smithi119 bash[96695]: f0cc29854e2d57f6a1359c42b62395592430fb77051ac76851893d36d2903c80 2024-03-19T03:30:14.484 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 19 03:30:14 smithi119 systemd[1]: Started Ceph mds.h for a3a82456-e59f-11ee-95c9-87774f69a715. 2024-03-19T03:30:15.233 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:30:15.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.457+0000 7fab26800640 1 -- 172.21.15.117:0/4255889195 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab20104c50 msgr2=0x7fab201050d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:15.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.457+0000 7fab26800640 1 --2- 172.21.15.117:0/4255889195 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab20104c50 0x7fab201050d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fab0c009900 tx=0x7fab0c02bc30 comp rx=0 tx=0).stop 2024-03-19T03:30:15.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.457+0000 7fab26800640 1 -- 172.21.15.117:0/4255889195 shutdown_connections 2024-03-19T03:30:15.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.457+0000 7fab26800640 1 --2- 172.21.15.117:0/4255889195 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fab20105610 0x7fab2010bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:15.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.457+0000 7fab26800640 1 --2- 172.21.15.117:0/4255889195 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab20104c50 0x7fab201050d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:15.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.457+0000 7fab26800640 1 --2- 172.21.15.117:0/4255889195 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab20103a60 0x7fab20103e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:15.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.457+0000 7fab26800640 1 -- 172.21.15.117:0/4255889195 >> 172.21.15.117:0/4255889195 conn(0x7fab200fefd0 msgr2=0x7fab20101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:15.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.457+0000 7fab26800640 1 -- 172.21.15.117:0/4255889195 shutdown_connections 2024-03-19T03:30:15.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.457+0000 7fab26800640 1 -- 172.21.15.117:0/4255889195 wait complete. 2024-03-19T03:30:15.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.458+0000 7fab26800640 1 Processor -- start 2024-03-19T03:30:15.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.458+0000 7fab26800640 1 -- start start 2024-03-19T03:30:15.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.458+0000 7fab26800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fab20103a60 0x7fab20199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:15.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.458+0000 7fab26800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab20104c50 0x7fab2019a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:15.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.458+0000 7fab26800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab20105610 0x7fab201a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:15.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.458+0000 7fab26800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fab2010f730 con 0x7fab20105610 2024-03-19T03:30:15.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.458+0000 7fab26800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fab2010f5b0 con 0x7fab20104c50 2024-03-19T03:30:15.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.458+0000 7fab26800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fab2010f8b0 con 0x7fab20103a60 2024-03-19T03:30:15.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.459+0000 7fab1f400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fab20103a60 0x7fab20199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:15.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.459+0000 7fab1f400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fab20103a60 0x7fab20199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:42446/0 (socket says 172.21.15.117:42446) 2024-03-19T03:30:15.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.459+0000 7fab1f400640 1 -- 172.21.15.117:0/220497299 learned_addr learned my addr 172.21.15.117:0/220497299 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:15.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.459+0000 7fab1fe00640 1 --2- 172.21.15.117:0/220497299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab20105610 0x7fab201a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:15.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.459+0000 7fab1ea00640 1 --2- 172.21.15.117:0/220497299 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab20104c50 0x7fab2019a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:15.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.459+0000 7fab1f400640 1 -- 172.21.15.117:0/220497299 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab20104c50 msgr2=0x7fab2019a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:15.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.459+0000 7fab1f400640 1 --2- 172.21.15.117:0/220497299 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab20104c50 0x7fab2019a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:15.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.459+0000 7fab1f400640 1 -- 172.21.15.117:0/220497299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab20105610 msgr2=0x7fab201a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:15.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.459+0000 7fab1f400640 1 --2- 172.21.15.117:0/220497299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab20105610 0x7fab201a12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:15.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.459+0000 7fab1f400640 1 -- 172.21.15.117:0/220497299 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab201a1960 con 0x7fab20103a60 2024-03-19T03:30:15.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.459+0000 7fab1f400640 1 --2- 172.21.15.117:0/220497299 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fab20103a60 0x7fab20199d40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fab1000e9c0 tx=0x7fab1000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:15.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.460+0000 7fab17e00640 1 -- 172.21.15.117:0/220497299 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1000cd80 con 0x7fab20103a60 2024-03-19T03:30:15.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.460+0000 7fab17e00640 1 -- 172.21.15.117:0/220497299 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab100035f0 con 0x7fab20103a60 2024-03-19T03:30:15.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.460+0000 7fab26800640 1 -- 172.21.15.117:0/220497299 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab201a1c50 con 0x7fab20103a60 2024-03-19T03:30:15.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.460+0000 7fab17e00640 1 -- 172.21.15.117:0/220497299 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab10010690 con 0x7fab20103a60 2024-03-19T03:30:15.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.460+0000 7fab26800640 1 -- 172.21.15.117:0/220497299 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab201a21c0 con 0x7fab20103a60 2024-03-19T03:30:15.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.461+0000 7fab26800640 1 -- 172.21.15.117:0/220497299 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab2010ad20 con 0x7fab20103a60 2024-03-19T03:30:15.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.463+0000 7fab17e00640 1 -- 172.21.15.117:0/220497299 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fab100037a0 con 0x7fab20103a60 2024-03-19T03:30:15.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.464+0000 7fab17e00640 1 --2- 172.21.15.117:0/220497299 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fab040a2880 0x7fab040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:15.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.464+0000 7fab17e00640 1 -- 172.21.15.117:0/220497299 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fab100ce970 con 0x7fab20103a60 2024-03-19T03:30:15.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.464+0000 7fab1ea00640 1 --2- 172.21.15.117:0/220497299 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fab040a2880 0x7fab040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:15.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.465+0000 7fab1ea00640 1 --2- 172.21.15.117:0/220497299 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fab040a2880 0x7fab040a4d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fab0c02c240 tx=0x7fab0c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:15.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.468+0000 7fab17e00640 1 -- 172.21.15.117:0/220497299 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fab1009be10 con 0x7fab20103a60 2024-03-19T03:30:15.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.705+0000 7fab26800640 1 -- 172.21.15.117:0/220497299 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab2002d1f0 con 0x7fab20103a60 2024-03-19T03:30:15.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.706+0000 7fab17e00640 1 -- 172.21.15.117:0/220497299 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7186 (secure 0 0 0) 0x7fab1009b710 con 0x7fab20103a60 2024-03-19T03:30:15.707 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:30:15.708 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/54638259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":54638259},{"type":"v1","addr":"172.21.15.117:6835","nonce":54638259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3608185039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3608185039},{"type":"v1","addr":"172.21.15.117:6837","nonce":3608185039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/1291833214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1291833214},{"type":"v1","addr":"172.21.15.119:6833","nonce":1291833214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1579860804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1579860804},{"type":"v1","addr":"172.21.15.178:6835","nonce":1579860804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24515,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/1686938941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1686938941},{"type":"v1","addr":"172.21.15.119:6835","nonce":1686938941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1770913538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":1770913538},{"type":"v1","addr":"172.21.15.178:6837","nonce":1770913538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2054992738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2054992738},{"type":"v1","addr":"172.21.15.119:6837","nonce":2054992738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-19T03:30:15.708 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 9 2024-03-19T03:30:15.711 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.709+0000 7fab26800640 1 -- 172.21.15.117:0/220497299 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fab040a2880 msgr2=0x7fab040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:15.711 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.709+0000 7fab26800640 1 --2- 172.21.15.117:0/220497299 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fab040a2880 0x7fab040a4d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fab0c02c240 tx=0x7fab0c02df90 comp rx=0 tx=0).stop 2024-03-19T03:30:15.711 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.709+0000 7fab26800640 1 -- 172.21.15.117:0/220497299 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fab20103a60 msgr2=0x7fab20199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:15.711 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.709+0000 7fab26800640 1 --2- 172.21.15.117:0/220497299 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fab20103a60 0x7fab20199d40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fab1000e9c0 tx=0x7fab1000ee80 comp rx=0 tx=0).stop 2024-03-19T03:30:15.711 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.709+0000 7fab26800640 1 -- 172.21.15.117:0/220497299 shutdown_connections 2024-03-19T03:30:15.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.709+0000 7fab26800640 1 --2- 172.21.15.117:0/220497299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab20105610 0x7fab201a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:15.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.709+0000 7fab26800640 1 --2- 172.21.15.117:0/220497299 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fab040a2880 0x7fab040a4d40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:15.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.709+0000 7fab26800640 1 --2- 172.21.15.117:0/220497299 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab20104c50 0x7fab2019a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:15.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.709+0000 7fab26800640 1 --2- 172.21.15.117:0/220497299 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fab20103a60 0x7fab20199d40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:15.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.709+0000 7fab26800640 1 -- 172.21.15.117:0/220497299 >> 172.21.15.117:0/220497299 conn(0x7fab200fefd0 msgr2=0x7fab20107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:15.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.709+0000 7fab26800640 1 -- 172.21.15.117:0/220497299 shutdown_connections 2024-03-19T03:30:15.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:15.709+0000 7fab26800640 1 -- 172.21.15.117:0/220497299 wait complete. 2024-03-19T03:30:15.719 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:15 smithi178 ceph-mon[67578]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:15.719 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:15 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:15.719 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:15 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:15.719 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:15 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:15.719 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:15 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:30:15.719 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:15 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:30:15.719 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:15 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:15.720 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:15 smithi178 ceph-mon[67578]: Deploying daemon mds.i on smithi178 2024-03-19T03:30:15.720 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:15 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] up:boot 2024-03-19T03:30:15.720 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:15 smithi178 ceph-mon[67578]: fsmap 8 up:standby 2024-03-19T03:30:15.720 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:15 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T03:30:15.727 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:15 smithi117 ceph-mon[31838]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:15.727 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:15 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:15.727 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:15 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:15.727 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:15 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:15.727 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:15 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:30:15.727 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:15 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:30:15.727 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:15 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:15.727 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:15 smithi117 ceph-mon[31838]: Deploying daemon mds.i on smithi178 2024-03-19T03:30:15.727 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:15 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] up:boot 2024-03-19T03:30:15.727 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:15 smithi117 ceph-mon[31838]: fsmap 8 up:standby 2024-03-19T03:30:15.728 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:15 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T03:30:15.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:15 smithi119 ceph-mon[66861]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:15.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:15 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:15.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:15 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:15.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:15 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:15.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:15 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:30:15.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:15 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:30:15.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:15 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:15.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:15 smithi119 ceph-mon[66861]: Deploying daemon mds.i on smithi178 2024-03-19T03:30:15.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:15 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] up:boot 2024-03-19T03:30:15.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:15 smithi119 ceph-mon[66861]: fsmap 8 up:standby 2024-03-19T03:30:15.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:15 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T03:30:16.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:16 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/220497299' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:16.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:16 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/220497299' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:16.804 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:16 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/220497299' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:16.805 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 19 03:30:16 smithi178 systemd[1]: Starting Ceph mds.i for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:30:17.061 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 19 03:30:16 smithi178 podman[98734]: 2024-03-19 03:30:16.707936342 +0000 UTC m=+0.017998884 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:30:17.061 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 19 03:30:16 smithi178 podman[98734]: 2024-03-19 03:30:16.806635047 +0000 UTC m=+0.116697570 container create 5ef48f5e4afc1de7477ca158fdbaa1b4bb598435a8bfe5d63bd1b2c95358ad65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-i, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T03:30:17.061 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 19 03:30:17 smithi178 podman[98734]: 2024-03-19 03:30:17.034462836 +0000 UTC m=+0.344525358 container init 5ef48f5e4afc1de7477ca158fdbaa1b4bb598435a8bfe5d63bd1b2c95358ad65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-i, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-19T03:30:17.061 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 19 03:30:17 smithi178 podman[98734]: 2024-03-19 03:30:17.037022171 +0000 UTC m=+0.347084710 container start 5ef48f5e4afc1de7477ca158fdbaa1b4bb598435a8bfe5d63bd1b2c95358ad65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-i, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-19T03:30:17.061 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 19 03:30:17 smithi178 ceph-mds[98753]: -- 172.21.15.178:0/1437300932 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e69da5fa40 con 0x55e69da33000 2024-03-19T03:30:17.346 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 19 03:30:17 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-i[98749]: starting mds.i at 2024-03-19T03:30:17.346 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 19 03:30:17 smithi178 bash[98734]: 5ef48f5e4afc1de7477ca158fdbaa1b4bb598435a8bfe5d63bd1b2c95358ad65 2024-03-19T03:30:17.346 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 19 03:30:17 smithi178 systemd[1]: Started Ceph mds.i for a3a82456-e59f-11ee-95c9-87774f69a715. 2024-03-19T03:30:17.730 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:30:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:17 smithi119 ceph-mon[66861]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:17 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:17 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:17 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:17 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:30:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:17 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:30:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:17 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:17.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:17 smithi117 ceph-mon[31838]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:17.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:17 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:17.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:17 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:17.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:17 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:17.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:17 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:30:17.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:17 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:30:17.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:17 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:17.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:17 smithi178 ceph-mon[67578]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:17.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:17 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:17.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:17 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:17.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:17 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:17.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:17 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:30:17.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:17 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:30:17.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:17 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:17.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.936+0000 7f61ef800640 1 -- 172.21.15.117:0/1503222302 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61e8077dc0 msgr2=0x7f61e8078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:17.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.936+0000 7f61ef800640 1 --2- 172.21.15.117:0/1503222302 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61e8077dc0 0x7f61e8078660 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f61e0009ca0 tx=0x7f61e002c0f0 comp rx=0 tx=0).stop 2024-03-19T03:30:17.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.936+0000 7f61ef800640 1 -- 172.21.15.117:0/1503222302 shutdown_connections 2024-03-19T03:30:17.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.936+0000 7f61ef800640 1 --2- 172.21.15.117:0/1503222302 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61e8077dc0 0x7f61e8078660 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:17.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.936+0000 7f61ef800640 1 --2- 172.21.15.117:0/1503222302 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61e8077420 0x7f61e8077880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:17.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.936+0000 7f61ef800640 1 --2- 172.21.15.117:0/1503222302 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61e80761e0 0x7f61e8076600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:17.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.936+0000 7f61ef800640 1 -- 172.21.15.117:0/1503222302 >> 172.21.15.117:0/1503222302 conn(0x7f61e80ff010 msgr2=0x7f61e8101430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:17.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.936+0000 7f61ef800640 1 -- 172.21.15.117:0/1503222302 shutdown_connections 2024-03-19T03:30:17.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.936+0000 7f61ef800640 1 -- 172.21.15.117:0/1503222302 wait complete. 2024-03-19T03:30:17.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.937+0000 7f61ef800640 1 Processor -- start 2024-03-19T03:30:17.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.937+0000 7f61ef800640 1 -- start start 2024-03-19T03:30:17.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.937+0000 7f61ef800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61e80761e0 0x7f61e819e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:17.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.938+0000 7f61ef800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61e8077420 0x7f61e819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:17.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.938+0000 7f61ef800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61e8077dc0 0x7f61e81a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:17.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.938+0000 7f61ef800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f61e8113af0 con 0x7f61e8077420 2024-03-19T03:30:17.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.938+0000 7f61ef800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f61e8113970 con 0x7f61e8077dc0 2024-03-19T03:30:17.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.938+0000 7f61ef800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f61e8113c70 con 0x7f61e80761e0 2024-03-19T03:30:17.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.938+0000 7f61ece00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61e80761e0 0x7f61e819e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:17.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.938+0000 7f61e7e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61e8077420 0x7f61e819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:17.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.938+0000 7f61ece00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61e80761e0 0x7f61e819e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:49150/0 (socket says 172.21.15.117:49150) 2024-03-19T03:30:17.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.938+0000 7f61ece00640 1 -- 172.21.15.117:0/3123729343 learned_addr learned my addr 172.21.15.117:0/3123729343 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:17.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.938+0000 7f61ed800640 1 --2- 172.21.15.117:0/3123729343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61e8077dc0 0x7f61e81a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:17.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.938+0000 7f61ece00640 1 -- 172.21.15.117:0/3123729343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61e8077dc0 msgr2=0x7f61e81a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:17.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.938+0000 7f61ece00640 1 --2- 172.21.15.117:0/3123729343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61e8077dc0 0x7f61e81a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:17.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.938+0000 7f61ece00640 1 -- 172.21.15.117:0/3123729343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61e8077420 msgr2=0x7f61e819e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:17.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.938+0000 7f61ece00640 1 --2- 172.21.15.117:0/3123729343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61e8077420 0x7f61e819e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:17.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.938+0000 7f61ece00640 1 -- 172.21.15.117:0/3123729343 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61e81a5d30 con 0x7f61e80761e0 2024-03-19T03:30:17.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.939+0000 7f61ece00640 1 --2- 172.21.15.117:0/3123729343 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61e80761e0 0x7f61e819e110 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f61dc00d950 tx=0x7f61dc00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:17.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.939+0000 7f61e5600640 1 -- 172.21.15.117:0/3123729343 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61dc017070 con 0x7f61e80761e0 2024-03-19T03:30:17.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.939+0000 7f61e5600640 1 -- 172.21.15.117:0/3123729343 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61dc009c30 con 0x7f61e80761e0 2024-03-19T03:30:17.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.939+0000 7f61ef800640 1 -- 172.21.15.117:0/3123729343 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61e81a5fc0 con 0x7f61e80761e0 2024-03-19T03:30:17.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.939+0000 7f61e5600640 1 -- 172.21.15.117:0/3123729343 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61dc00f6b0 con 0x7f61e80761e0 2024-03-19T03:30:17.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.939+0000 7f61ef800640 1 -- 172.21.15.117:0/3123729343 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61e81a64d0 con 0x7f61e80761e0 2024-03-19T03:30:17.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.941+0000 7f61ef800640 1 -- 172.21.15.117:0/3123729343 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61e8110560 con 0x7f61e80761e0 2024-03-19T03:30:17.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.942+0000 7f61e5600640 1 -- 172.21.15.117:0/3123729343 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f61dc0109d0 con 0x7f61e80761e0 2024-03-19T03:30:17.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.944+0000 7f61e5600640 1 --2- 172.21.15.117:0/3123729343 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f61d00a2880 0x7f61d00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:17.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.944+0000 7f61e5600640 1 -- 172.21.15.117:0/3123729343 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f61dc0ce200 con 0x7f61e80761e0 2024-03-19T03:30:17.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.944+0000 7f61e7e00640 1 --2- 172.21.15.117:0/3123729343 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f61d00a2880 0x7f61d00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:17.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.945+0000 7f61e7e00640 1 --2- 172.21.15.117:0/3123729343 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f61d00a2880 0x7f61d00a4d40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f61e819f620 tx=0x7f61d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:17.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:17.948+0000 7f61e5600640 1 -- 172.21.15.117:0/3123729343 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f61dc009de0 con 0x7f61e80761e0 2024-03-19T03:30:18.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:18.195+0000 7f61ef800640 1 -- 172.21.15.117:0/3123729343 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61e8062420 con 0x7f61e80761e0 2024-03-19T03:30:18.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:18.196+0000 7f61e5600640 1 -- 172.21.15.117:0/3123729343 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8016 (secure 0 0 0) 0x7f61dc09b6a0 con 0x7f61e80761e0 2024-03-19T03:30:18.197 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:30:18.197 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/54638259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":54638259},{"type":"v1","addr":"172.21.15.117:6835","nonce":54638259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3608185039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3608185039},{"type":"v1","addr":"172.21.15.117:6837","nonce":3608185039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/1291833214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1291833214},{"type":"v1","addr":"172.21.15.119:6833","nonce":1291833214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1579860804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1579860804},{"type":"v1","addr":"172.21.15.178:6835","nonce":1579860804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24515,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/1686938941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1686938941},{"type":"v1","addr":"172.21.15.119:6835","nonce":1686938941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1770913538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":1770913538},{"type":"v1","addr":"172.21.15.178:6837","nonce":1770913538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6839/2655593383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2655593383},{"type":"v1","addr":"172.21.15.178:6839","nonce":2655593383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2054992738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2054992738},{"type":"v1","addr":"172.21.15.119:6837","nonce":2054992738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-19T03:30:18.197 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 10 2024-03-19T03:30:18.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:18.198+0000 7f61ef800640 1 -- 172.21.15.117:0/3123729343 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f61d00a2880 msgr2=0x7f61d00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:18.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:18.198+0000 7f61ef800640 1 --2- 172.21.15.117:0/3123729343 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f61d00a2880 0x7f61d00a4d40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f61e819f620 tx=0x7f61d4005f90 comp rx=0 tx=0).stop 2024-03-19T03:30:18.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:18.198+0000 7f61ef800640 1 -- 172.21.15.117:0/3123729343 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61e80761e0 msgr2=0x7f61e819e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:18.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:18.198+0000 7f61ef800640 1 --2- 172.21.15.117:0/3123729343 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61e80761e0 0x7f61e819e110 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f61dc00d950 tx=0x7f61dc00de10 comp rx=0 tx=0).stop 2024-03-19T03:30:18.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:18.198+0000 7f61ef800640 1 -- 172.21.15.117:0/3123729343 shutdown_connections 2024-03-19T03:30:18.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:18.198+0000 7f61ef800640 1 --2- 172.21.15.117:0/3123729343 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f61d00a2880 0x7f61d00a4d40 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:18.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:18.198+0000 7f61ef800640 1 --2- 172.21.15.117:0/3123729343 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61e8077dc0 0x7f61e81a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:18.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:18.198+0000 7f61ef800640 1 --2- 172.21.15.117:0/3123729343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61e8077420 0x7f61e819e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:18.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:18.198+0000 7f61ef800640 1 --2- 172.21.15.117:0/3123729343 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61e80761e0 0x7f61e819e110 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:18.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:18.198+0000 7f61ef800640 1 -- 172.21.15.117:0/3123729343 >> 172.21.15.117:0/3123729343 conn(0x7f61e80ff010 msgr2=0x7f61e81006b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:18.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:18.199+0000 7f61ef800640 1 -- 172.21.15.117:0/3123729343 shutdown_connections 2024-03-19T03:30:18.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:18.199+0000 7f61ef800640 1 -- 172.21.15.117:0/3123729343 wait complete. 2024-03-19T03:30:18.460 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:18 smithi178 ceph-mon[67578]: Deploying daemon mds.g on smithi117 2024-03-19T03:30:18.460 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:18 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.178:6838/2655593383,v1:172.21.15.178:6839/2655593383] up:boot 2024-03-19T03:30:18.461 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:18 smithi178 ceph-mon[67578]: fsmap 9 up:standby 2024-03-19T03:30:18.461 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:18 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T03:30:18.461 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:18 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3123729343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:18.474 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:18 smithi117 ceph-mon[31838]: Deploying daemon mds.g on smithi117 2024-03-19T03:30:18.474 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:18 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.178:6838/2655593383,v1:172.21.15.178:6839/2655593383] up:boot 2024-03-19T03:30:18.474 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:18 smithi117 ceph-mon[31838]: fsmap 9 up:standby 2024-03-19T03:30:18.475 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:18 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T03:30:18.475 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:18 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3123729343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:18.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:18 smithi119 ceph-mon[66861]: Deploying daemon mds.g on smithi117 2024-03-19T03:30:18.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:18 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.178:6838/2655593383,v1:172.21.15.178:6839/2655593383] up:boot 2024-03-19T03:30:18.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:18 smithi119 ceph-mon[66861]: fsmap 9 up:standby 2024-03-19T03:30:18.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:18 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T03:30:18.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:18 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3123729343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:19.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:19 smithi117 ceph-mon[31838]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:19.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:19 smithi119 ceph-mon[66861]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:19.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:19 smithi178 ceph-mon[67578]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:20.218 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:30:20.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.505+0000 7f24f1400640 1 -- 172.21.15.117:0/2646028609 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24ec103a60 msgr2=0x7f24ec103e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:20.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.505+0000 7f24f1400640 1 --2- 172.21.15.117:0/2646028609 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24ec103a60 0x7f24ec103e80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f24dc009900 tx=0x7f24dc02bc30 comp rx=0 tx=0).stop 2024-03-19T03:30:20.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.505+0000 7f24f1400640 1 -- 172.21.15.117:0/2646028609 shutdown_connections 2024-03-19T03:30:20.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.505+0000 7f24f1400640 1 --2- 172.21.15.117:0/2646028609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24ec105610 0x7f24ec10bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:20.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.505+0000 7f24f1400640 1 --2- 172.21.15.117:0/2646028609 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f24ec104c50 0x7f24ec1050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:20.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.505+0000 7f24f1400640 1 --2- 172.21.15.117:0/2646028609 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24ec103a60 0x7f24ec103e80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:20.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.505+0000 7f24f1400640 1 -- 172.21.15.117:0/2646028609 >> 172.21.15.117:0/2646028609 conn(0x7f24ec0ff010 msgr2=0x7f24ec101430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:20.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.505+0000 7f24f1400640 1 -- 172.21.15.117:0/2646028609 shutdown_connections 2024-03-19T03:30:20.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.505+0000 7f24f1400640 1 -- 172.21.15.117:0/2646028609 wait complete. 2024-03-19T03:30:20.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.507+0000 7f24f1400640 1 Processor -- start 2024-03-19T03:30:20.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.507+0000 7f24f1400640 1 -- start start 2024-03-19T03:30:20.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.507+0000 7f24f1400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24ec103a60 0x7f24ec199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:20.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.507+0000 7f24f1400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24ec104c50 0x7f24ec19a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:20.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.507+0000 7f24f1400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f24ec105610 0x7f24ec1a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:20.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.507+0000 7f24f1400640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f24ec10f6d0 con 0x7f24ec104c50 2024-03-19T03:30:20.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.507+0000 7f24f1400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f24ec10f550 con 0x7f24ec103a60 2024-03-19T03:30:20.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.507+0000 7f24f1400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f24ec10f850 con 0x7f24ec105610 2024-03-19T03:30:20.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.508+0000 7f24ea800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24ec103a60 0x7f24ec199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:20.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.508+0000 7f24ea800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24ec103a60 0x7f24ec199d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:59602/0 (socket says 172.21.15.117:59602) 2024-03-19T03:30:20.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.508+0000 7f24ea800640 1 -- 172.21.15.117:0/754804738 learned_addr learned my addr 172.21.15.117:0/754804738 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:20.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.508+0000 7f24e1e00640 1 --2- 172.21.15.117:0/754804738 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24ec104c50 0x7f24ec19a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:20.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.508+0000 7f24eb200640 1 --2- 172.21.15.117:0/754804738 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f24ec105610 0x7f24ec1a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:20.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.508+0000 7f24ea800640 1 -- 172.21.15.117:0/754804738 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f24ec105610 msgr2=0x7f24ec1a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:20.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.508+0000 7f24ea800640 1 --2- 172.21.15.117:0/754804738 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f24ec105610 0x7f24ec1a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:20.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.508+0000 7f24ea800640 1 -- 172.21.15.117:0/754804738 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24ec104c50 msgr2=0x7f24ec19a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:20.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.508+0000 7f24ea800640 1 --2- 172.21.15.117:0/754804738 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24ec104c50 0x7f24ec19a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:20.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.508+0000 7f24ea800640 1 -- 172.21.15.117:0/754804738 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24ec1a1990 con 0x7f24ec103a60 2024-03-19T03:30:20.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.508+0000 7f24ea800640 1 --2- 172.21.15.117:0/754804738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24ec103a60 0x7f24ec199d70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f24dc000c00 tx=0x7f24dc005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:20.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.509+0000 7f24e3e00640 1 -- 172.21.15.117:0/754804738 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24dc041070 con 0x7f24ec103a60 2024-03-19T03:30:20.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.509+0000 7f24e3e00640 1 -- 172.21.15.117:0/754804738 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24dc0044d0 con 0x7f24ec103a60 2024-03-19T03:30:20.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.509+0000 7f24f1400640 1 -- 172.21.15.117:0/754804738 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24ec1a1c20 con 0x7f24ec103a60 2024-03-19T03:30:20.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.509+0000 7f24e3e00640 1 -- 172.21.15.117:0/754804738 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24dc034440 con 0x7f24ec103a60 2024-03-19T03:30:20.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.509+0000 7f24f1400640 1 -- 172.21.15.117:0/754804738 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24ec1a2160 con 0x7f24ec103a60 2024-03-19T03:30:20.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.510+0000 7f24f1400640 1 -- 172.21.15.117:0/754804738 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24ec10ae00 con 0x7f24ec103a60 2024-03-19T03:30:20.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.512+0000 7f24e3e00640 1 -- 172.21.15.117:0/754804738 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f24dc044070 con 0x7f24ec103a60 2024-03-19T03:30:20.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.513+0000 7f24e3e00640 1 --2- 172.21.15.117:0/754804738 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f24d00a2880 0x7f24d00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:20.514 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.514+0000 7f24e3e00640 1 -- 172.21.15.117:0/754804738 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f24dc0f0280 con 0x7f24ec103a60 2024-03-19T03:30:20.515 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.514+0000 7f24e1e00640 1 --2- 172.21.15.117:0/754804738 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f24d00a2880 0x7f24d00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:20.515 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.514+0000 7f24e1e00640 1 --2- 172.21.15.117:0/754804738 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f24d00a2880 0x7f24d00a4d40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f24ec19b280 tx=0x7f24d400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:20.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.515+0000 7f24e3e00640 1 -- 172.21.15.117:0/754804738 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f24dc0bd170 con 0x7f24ec103a60 2024-03-19T03:30:20.766 INFO:journalctl@ceph.mds.g.smithi117.stdout:Mar 19 03:30:20 smithi117 systemd[1]: Starting Ceph mds.g for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:30:20.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.772+0000 7f24f1400640 1 -- 172.21.15.117:0/754804738 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24ec02d1f0 con 0x7f24ec103a60 2024-03-19T03:30:20.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.773+0000 7f24e3e00640 1 -- 172.21.15.117:0/754804738 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8016 (secure 0 0 0) 0x7f24dc0c1210 con 0x7f24ec103a60 2024-03-19T03:30:20.775 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:30:20.775 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/54638259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":54638259},{"type":"v1","addr":"172.21.15.117:6835","nonce":54638259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3608185039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3608185039},{"type":"v1","addr":"172.21.15.117:6837","nonce":3608185039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/1291833214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1291833214},{"type":"v1","addr":"172.21.15.119:6833","nonce":1291833214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1579860804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1579860804},{"type":"v1","addr":"172.21.15.178:6835","nonce":1579860804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24515,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/1686938941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1686938941},{"type":"v1","addr":"172.21.15.119:6835","nonce":1686938941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1770913538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":1770913538},{"type":"v1","addr":"172.21.15.178:6837","nonce":1770913538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6839/2655593383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2655593383},{"type":"v1","addr":"172.21.15.178:6839","nonce":2655593383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2054992738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2054992738},{"type":"v1","addr":"172.21.15.119:6837","nonce":2054992738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-19T03:30:20.775 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 10 2024-03-19T03:30:20.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.776+0000 7f24f1400640 1 -- 172.21.15.117:0/754804738 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f24d00a2880 msgr2=0x7f24d00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:20.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.776+0000 7f24f1400640 1 --2- 172.21.15.117:0/754804738 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f24d00a2880 0x7f24d00a4d40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f24ec19b280 tx=0x7f24d400b390 comp rx=0 tx=0).stop 2024-03-19T03:30:20.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.776+0000 7f24f1400640 1 -- 172.21.15.117:0/754804738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24ec103a60 msgr2=0x7f24ec199d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:20.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.776+0000 7f24f1400640 1 --2- 172.21.15.117:0/754804738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24ec103a60 0x7f24ec199d70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f24dc000c00 tx=0x7f24dc005d50 comp rx=0 tx=0).stop 2024-03-19T03:30:20.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.776+0000 7f24f1400640 1 -- 172.21.15.117:0/754804738 shutdown_connections 2024-03-19T03:30:20.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.776+0000 7f24f1400640 1 --2- 172.21.15.117:0/754804738 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f24ec105610 0x7f24ec1a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:20.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.776+0000 7f24f1400640 1 --2- 172.21.15.117:0/754804738 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24ec104c50 0x7f24ec19a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:20.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.776+0000 7f24f1400640 1 --2- 172.21.15.117:0/754804738 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f24d00a2880 0x7f24d00a4d40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:20.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.776+0000 7f24f1400640 1 --2- 172.21.15.117:0/754804738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24ec103a60 0x7f24ec199d70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:20.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.776+0000 7f24f1400640 1 -- 172.21.15.117:0/754804738 >> 172.21.15.117:0/754804738 conn(0x7f24ec0ff010 msgr2=0x7f24ec107fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:20.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.777+0000 7f24f1400640 1 -- 172.21.15.117:0/754804738 shutdown_connections 2024-03-19T03:30:20.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:20.777+0000 7f24f1400640 1 -- 172.21.15.117:0/754804738 wait complete. 2024-03-19T03:30:21.042 INFO:journalctl@ceph.mds.g.smithi117.stdout:Mar 19 03:30:20 smithi117 podman[73634]: 2024-03-19 03:30:20.670596965 +0000 UTC m=+0.042935991 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:30:21.042 INFO:journalctl@ceph.mds.g.smithi117.stdout:Mar 19 03:30:20 smithi117 podman[73634]: 2024-03-19 03:30:20.816152647 +0000 UTC m=+0.188491703 container create 71a8940c83ddc18d09d7780673e86a2effda560544d1198457ea71fdaf8f566b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True) 2024-03-19T03:30:21.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:21 smithi117 ceph-mon[31838]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:21.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:21 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/754804738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:21.542 INFO:journalctl@ceph.mds.g.smithi117.stdout:Mar 19 03:30:21 smithi117 podman[73634]: 2024-03-19 03:30:21.155046279 +0000 UTC m=+0.527385381 container init 71a8940c83ddc18d09d7780673e86a2effda560544d1198457ea71fdaf8f566b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-g, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:30:21.542 INFO:journalctl@ceph.mds.g.smithi117.stdout:Mar 19 03:30:21 smithi117 podman[73634]: 2024-03-19 03:30:21.165562375 +0000 UTC m=+0.537901401 container start 71a8940c83ddc18d09d7780673e86a2effda560544d1198457ea71fdaf8f566b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-g, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T03:30:21.542 INFO:journalctl@ceph.mds.g.smithi117.stdout:Mar 19 03:30:21 smithi117 ceph-mds[73655]: -- 172.21.15.117:0/1046152546 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55906bb81a40 con 0x55906bb55000 2024-03-19T03:30:21.542 INFO:journalctl@ceph.mds.g.smithi117.stdout:Mar 19 03:30:21 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-g[73651]: starting mds.g at 2024-03-19T03:30:21.542 INFO:journalctl@ceph.mds.g.smithi117.stdout:Mar 19 03:30:21 smithi117 bash[73634]: 71a8940c83ddc18d09d7780673e86a2effda560544d1198457ea71fdaf8f566b 2024-03-19T03:30:21.542 INFO:journalctl@ceph.mds.g.smithi117.stdout:Mar 19 03:30:21 smithi117 systemd[1]: Started Ceph mds.g for a3a82456-e59f-11ee-95c9-87774f69a715. 2024-03-19T03:30:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:21 smithi119 ceph-mon[66861]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:21 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/754804738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:21.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:21 smithi178 ceph-mon[67578]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:21.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:21 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/754804738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:22.657 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:22 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] up:boot 2024-03-19T03:30:22.657 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:22 smithi117 ceph-mon[31838]: fsmap 10 up:standby 2024-03-19T03:30:22.657 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:22 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T03:30:22.657 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:22 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:22.657 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:22 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:22.657 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:22 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:22.657 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:22 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:30:22.657 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:22 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:30:22.658 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:22 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:22.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:22 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] up:boot 2024-03-19T03:30:22.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:22 smithi119 ceph-mon[66861]: fsmap 10 up:standby 2024-03-19T03:30:22.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:22 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T03:30:22.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:22 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:22.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:22 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:22.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:22 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:22.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:22 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:30:22.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:22 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:30:22.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:22 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:22.796 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:30:22.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:22 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] up:boot 2024-03-19T03:30:22.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:22 smithi178 ceph-mon[67578]: fsmap 10 up:standby 2024-03-19T03:30:22.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:22 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T03:30:22.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:22 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:22.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:22 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:22.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:22 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:22.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:22 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:30:22.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:22 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:30:22.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:22 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:22.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.992+0000 7f38a9c00640 1 -- 172.21.15.117:0/34591394 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38a4109830 msgr2=0x7f38a41100c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:22.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.992+0000 7f38a9c00640 1 --2- 172.21.15.117:0/34591394 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38a4109830 0x7f38a41100c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f389c009ca0 tx=0x7f389c02c0f0 comp rx=0 tx=0).stop 2024-03-19T03:30:22.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.992+0000 7f38a9c00640 1 -- 172.21.15.117:0/34591394 shutdown_connections 2024-03-19T03:30:22.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.992+0000 7f38a9c00640 1 --2- 172.21.15.117:0/34591394 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38a4109830 0x7f38a41100c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:22.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.992+0000 7f38a9c00640 1 --2- 172.21.15.117:0/34591394 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38a4108e70 0x7f38a41092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:22.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.992+0000 7f38a9c00640 1 --2- 172.21.15.117:0/34591394 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38a4107c80 0x7f38a41080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:22.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.992+0000 7f38a9c00640 1 -- 172.21.15.117:0/34591394 >> 172.21.15.117:0/34591394 conn(0x7f38a4076c30 msgr2=0x7f38a4079050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:22.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.992+0000 7f38a9c00640 1 -- 172.21.15.117:0/34591394 shutdown_connections 2024-03-19T03:30:22.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.992+0000 7f38a9c00640 1 -- 172.21.15.117:0/34591394 wait complete. 2024-03-19T03:30:22.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.993+0000 7f38a9c00640 1 Processor -- start 2024-03-19T03:30:22.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.993+0000 7f38a9c00640 1 -- start start 2024-03-19T03:30:22.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.994+0000 7f38a9c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38a4107c80 0x7f38a419def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:22.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.994+0000 7f38a9c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38a4108e70 0x7f38a419e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:22.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.994+0000 7f38a9c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38a4109830 0x7f38a41a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:22.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.994+0000 7f38a9c00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f38a4113900 con 0x7f38a4108e70 2024-03-19T03:30:22.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.994+0000 7f38a9c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f38a4113780 con 0x7f38a4107c80 2024-03-19T03:30:22.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.994+0000 7f38a9c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f38a4113a80 con 0x7f38a4109830 2024-03-19T03:30:22.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.994+0000 7f389b400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38a4108e70 0x7f38a419e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:22.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.994+0000 7f38a3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38a4107c80 0x7f38a419def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:22.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.994+0000 7f389b400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38a4108e70 0x7f38a419e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:40568/0 (socket says 172.21.15.117:40568) 2024-03-19T03:30:22.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.994+0000 7f389b400640 1 -- 172.21.15.117:0/3405213610 learned_addr learned my addr 172.21.15.117:0/3405213610 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:22.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.994+0000 7f38a9200640 1 --2- 172.21.15.117:0/3405213610 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38a4109830 0x7f38a41a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:22.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.994+0000 7f38a3e00640 1 -- 172.21.15.117:0/3405213610 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38a4109830 msgr2=0x7f38a41a54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:22.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.994+0000 7f38a3e00640 1 --2- 172.21.15.117:0/3405213610 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38a4109830 0x7f38a41a54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:22.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.994+0000 7f38a3e00640 1 -- 172.21.15.117:0/3405213610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38a4108e70 msgr2=0x7f38a419e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:22.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.994+0000 7f38a3e00640 1 --2- 172.21.15.117:0/3405213610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38a4108e70 0x7f38a419e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:22.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.994+0000 7f38a3e00640 1 -- 172.21.15.117:0/3405213610 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38a41a5b10 con 0x7f38a4107c80 2024-03-19T03:30:22.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.994+0000 7f389b400640 1 --2- 172.21.15.117:0/3405213610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38a4108e70 0x7f38a419e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:30:22.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.995+0000 7f38a3e00640 1 --2- 172.21.15.117:0/3405213610 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38a4107c80 0x7f38a419def0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f389400e9c0 tx=0x7f389400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:22.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.995+0000 7f38a1600640 1 -- 172.21.15.117:0/3405213610 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389400ccc0 con 0x7f38a4107c80 2024-03-19T03:30:22.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.995+0000 7f38a1600640 1 -- 172.21.15.117:0/3405213610 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38940035f0 con 0x7f38a4107c80 2024-03-19T03:30:22.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.995+0000 7f38a9c00640 1 -- 172.21.15.117:0/3405213610 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38a41a5e00 con 0x7f38a4107c80 2024-03-19T03:30:22.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.995+0000 7f38a1600640 1 -- 172.21.15.117:0/3405213610 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389400ce60 con 0x7f38a4107c80 2024-03-19T03:30:22.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.995+0000 7f38a9c00640 1 -- 172.21.15.117:0/3405213610 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38a41a6260 con 0x7f38a4107c80 2024-03-19T03:30:22.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.996+0000 7f38a9c00640 1 -- 172.21.15.117:0/3405213610 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38a410ef40 con 0x7f38a4107c80 2024-03-19T03:30:22.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.998+0000 7f38a1600640 1 -- 172.21.15.117:0/3405213610 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f38940037a0 con 0x7f38a4107c80 2024-03-19T03:30:23.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:22.999+0000 7f38a1600640 1 --2- 172.21.15.117:0/3405213610 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f38880a2800 0x7f38880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:23.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:23.000+0000 7f38a1600640 1 -- 172.21.15.117:0/3405213610 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f38940cec40 con 0x7f38a4107c80 2024-03-19T03:30:23.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:23.000+0000 7f389b400640 1 --2- 172.21.15.117:0/3405213610 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f38880a2800 0x7f38880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:23.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:23.000+0000 7f389b400640 1 --2- 172.21.15.117:0/3405213610 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f38880a2800 0x7f38880a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f388c0095f0 tx=0x7f388c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:23.003 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:23.002+0000 7f38a1600640 1 -- 172.21.15.117:0/3405213610 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f389409c0e0 con 0x7f38a4107c80 2024-03-19T03:30:23.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:23.250+0000 7f38a9c00640 1 -- 172.21.15.117:0/3405213610 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38a402d430 con 0x7f38a4107c80 2024-03-19T03:30:23.252 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:23.251+0000 7f38a1600640 1 -- 172.21.15.117:0/3405213610 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8842 (secure 0 0 0) 0x7f389409b9e0 con 0x7f38a4107c80 2024-03-19T03:30:23.252 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:30:23.252 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/54638259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":54638259},{"type":"v1","addr":"172.21.15.117:6835","nonce":54638259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3608185039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3608185039},{"type":"v1","addr":"172.21.15.117:6837","nonce":3608185039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/1291833214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1291833214},{"type":"v1","addr":"172.21.15.119:6833","nonce":1291833214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1579860804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1579860804},{"type":"v1","addr":"172.21.15.178:6835","nonce":1579860804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24515,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/1686938941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1686938941},{"type":"v1","addr":"172.21.15.119:6835","nonce":1686938941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1770913538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":1770913538},{"type":"v1","addr":"172.21.15.178:6837","nonce":1770913538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6839/2655593383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2655593383},{"type":"v1","addr":"172.21.15.178:6839","nonce":2655593383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2054992738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2054992738},{"type":"v1","addr":"172.21.15.119:6837","nonce":2054992738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24569,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6839/684335078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":684335078},{"type":"v1","addr":"172.21.15.117:6839","nonce":684335078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-03-19T03:30:23.253 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 11 2024-03-19T03:30:23.255 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:23.254+0000 7f38a9c00640 1 -- 172.21.15.117:0/3405213610 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f38880a2800 msgr2=0x7f38880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:23.255 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:23.254+0000 7f38a9c00640 1 --2- 172.21.15.117:0/3405213610 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f38880a2800 0x7f38880a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f388c0095f0 tx=0x7f388c005f90 comp rx=0 tx=0).stop 2024-03-19T03:30:23.255 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:23.254+0000 7f38a9c00640 1 -- 172.21.15.117:0/3405213610 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38a4107c80 msgr2=0x7f38a419def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:23.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:23.254+0000 7f38a9c00640 1 --2- 172.21.15.117:0/3405213610 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38a4107c80 0x7f38a419def0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f389400e9c0 tx=0x7f389400ee80 comp rx=0 tx=0).stop 2024-03-19T03:30:23.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:23.254+0000 7f38a9c00640 1 -- 172.21.15.117:0/3405213610 shutdown_connections 2024-03-19T03:30:23.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:23.254+0000 7f38a9c00640 1 --2- 172.21.15.117:0/3405213610 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38a4109830 0x7f38a41a54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:23.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:23.254+0000 7f38a9c00640 1 --2- 172.21.15.117:0/3405213610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38a4108e70 0x7f38a419e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:23.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:23.254+0000 7f38a9c00640 1 --2- 172.21.15.117:0/3405213610 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f38880a2800 0x7f38880a4cc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:23.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:23.254+0000 7f38a9c00640 1 --2- 172.21.15.117:0/3405213610 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38a4107c80 0x7f38a419def0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:23.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:23.254+0000 7f38a9c00640 1 -- 172.21.15.117:0/3405213610 >> 172.21.15.117:0/3405213610 conn(0x7f38a4076c30 msgr2=0x7f38a410c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:23.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:23.255+0000 7f38a9c00640 1 -- 172.21.15.117:0/3405213610 shutdown_connections 2024-03-19T03:30:23.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:23.255+0000 7f38a9c00640 1 -- 172.21.15.117:0/3405213610 wait complete. 2024-03-19T03:30:23.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:23 smithi117 ceph-mon[31838]: Deploying daemon mds.a on smithi117 2024-03-19T03:30:23.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:23 smithi117 ceph-mon[31838]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:23.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:23 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3405213610' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:23.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:23 smithi119 ceph-mon[66861]: Deploying daemon mds.a on smithi117 2024-03-19T03:30:23.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:23 smithi119 ceph-mon[66861]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:23.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:23 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3405213610' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:23.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:23 smithi178 ceph-mon[67578]: Deploying daemon mds.a on smithi117 2024-03-19T03:30:23.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:23 smithi178 ceph-mon[67578]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:23.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:23 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3405213610' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:24.982 INFO:journalctl@ceph.mds.a.smithi117.stdout:Mar 19 03:30:24 smithi117 systemd[1]: Starting Ceph mds.a for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:30:25.274 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:30:25.293 INFO:journalctl@ceph.mds.a.smithi117.stdout:Mar 19 03:30:24 smithi117 podman[73983]: 2024-03-19 03:30:24.886637753 +0000 UTC m=+0.032197315 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:30:25.293 INFO:journalctl@ceph.mds.a.smithi117.stdout:Mar 19 03:30:25 smithi117 podman[73983]: 2024-03-19 03:30:25.016420413 +0000 UTC m=+0.161980011 container create 175e7bb211795cd8eff5e48e000853619e9cdadaaf1575e7cc494fe06a2e8e5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-a, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-19T03:30:25.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.492+0000 7f260b200640 1 -- 172.21.15.117:0/1440621590 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f260c102870 msgr2=0x7f260c102cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:25.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.492+0000 7f260b200640 1 --2- 172.21.15.117:0/1440621590 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f260c102870 0x7f260c102cf0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f25f8009860 tx=0x7f25f802bb70 comp rx=0 tx=0).stop 2024-03-19T03:30:25.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.492+0000 7f260b200640 1 -- 172.21.15.117:0/1440621590 shutdown_connections 2024-03-19T03:30:25.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.492+0000 7f260b200640 1 --2- 172.21.15.117:0/1440621590 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f260c103230 0x7f260c109ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:25.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.492+0000 7f260b200640 1 --2- 172.21.15.117:0/1440621590 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f260c102870 0x7f260c102cf0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:25.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.492+0000 7f260b200640 1 --2- 172.21.15.117:0/1440621590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f260c101680 0x7f260c101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:25.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.492+0000 7f260b200640 1 -- 172.21.15.117:0/1440621590 >> 172.21.15.117:0/1440621590 conn(0x7f260c0fcbf0 msgr2=0x7f260c0ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:25.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.492+0000 7f260b200640 1 -- 172.21.15.117:0/1440621590 shutdown_connections 2024-03-19T03:30:25.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.492+0000 7f260b200640 1 -- 172.21.15.117:0/1440621590 wait complete. 2024-03-19T03:30:25.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.493+0000 7f260b200640 1 Processor -- start 2024-03-19T03:30:25.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.493+0000 7f260b200640 1 -- start start 2024-03-19T03:30:25.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.493+0000 7f260b200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f260c101680 0x7f260c199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:25.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.493+0000 7f260b200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f260c102870 0x7f260c19a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:25.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.493+0000 7f260b200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f260c103230 0x7f260c1a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:25.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.493+0000 7f260b200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f260c10d2e0 con 0x7f260c103230 2024-03-19T03:30:25.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.493+0000 7f260b200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f260c10d160 con 0x7f260c102870 2024-03-19T03:30:25.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.493+0000 7f260b200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f260c10d460 con 0x7f260c101680 2024-03-19T03:30:25.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.494+0000 7f2609400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f260c102870 0x7f260c19a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:25.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.494+0000 7f260a800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f260c103230 0x7f260c1a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:25.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.494+0000 7f2609e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f260c101680 0x7f260c199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:25.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.494+0000 7f2609400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f260c102870 0x7f260c19a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:59666/0 (socket says 172.21.15.117:59666) 2024-03-19T03:30:25.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.494+0000 7f2609400640 1 -- 172.21.15.117:0/776127464 learned_addr learned my addr 172.21.15.117:0/776127464 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:25.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.494+0000 7f2609400640 1 -- 172.21.15.117:0/776127464 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f260c101680 msgr2=0x7f260c199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:25.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.494+0000 7f2609400640 1 --2- 172.21.15.117:0/776127464 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f260c101680 0x7f260c199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:25.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.494+0000 7f2609400640 1 -- 172.21.15.117:0/776127464 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f260c103230 msgr2=0x7f260c1a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:25.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.494+0000 7f2609400640 1 --2- 172.21.15.117:0/776127464 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f260c103230 0x7f260c1a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:25.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.494+0000 7f2609400640 1 -- 172.21.15.117:0/776127464 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f260c1a1790 con 0x7f260c102870 2024-03-19T03:30:25.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.494+0000 7f2609400640 1 --2- 172.21.15.117:0/776127464 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f260c102870 0x7f260c19a0b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f25f8005940 tx=0x7f25f8032470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:25.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.495+0000 7f2602a00640 1 -- 172.21.15.117:0/776127464 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f8041020 con 0x7f260c102870 2024-03-19T03:30:25.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.495+0000 7f2602a00640 1 -- 172.21.15.117:0/776127464 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25f802c9f0 con 0x7f260c102870 2024-03-19T03:30:25.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.495+0000 7f2602a00640 1 -- 172.21.15.117:0/776127464 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f8039040 con 0x7f260c102870 2024-03-19T03:30:25.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.495+0000 7f260b200640 1 -- 172.21.15.117:0/776127464 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f260c1a1a20 con 0x7f260c102870 2024-03-19T03:30:25.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.495+0000 7f260b200640 1 -- 172.21.15.117:0/776127464 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f260c1a1f60 con 0x7f260c102870 2024-03-19T03:30:25.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.496+0000 7f260b200640 1 -- 172.21.15.117:0/776127464 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f260c1a2190 con 0x7f260c102870 2024-03-19T03:30:25.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.498+0000 7f2602a00640 1 -- 172.21.15.117:0/776127464 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f25f8044020 con 0x7f260c102870 2024-03-19T03:30:25.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.499+0000 7f2602a00640 1 --2- 172.21.15.117:0/776127464 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f25f00a2880 0x7f25f00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:25.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.500+0000 7f2602a00640 1 -- 172.21.15.117:0/776127464 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f25f80f0290 con 0x7f260c102870 2024-03-19T03:30:25.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.500+0000 7f2609e00640 1 --2- 172.21.15.117:0/776127464 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f25f00a2880 0x7f25f00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:25.501 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.500+0000 7f2609e00640 1 --2- 172.21.15.117:0/776127464 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f25f00a2880 0x7f25f00a4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f260c1026d0 tx=0x7f25fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:25.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.502+0000 7f2602a00640 1 -- 172.21.15.117:0/776127464 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f25f804b020 con 0x7f260c102870 2024-03-19T03:30:25.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:25 smithi119 ceph-mon[66861]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:25.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.776+0000 7f260b200640 1 -- 172.21.15.117:0/776127464 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f260c1a2360 con 0x7f260c102870 2024-03-19T03:30:25.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.777+0000 7f2602a00640 1 -- 172.21.15.117:0/776127464 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9671 (secure 0 0 0) 0x7f25f80bd7b0 con 0x7f260c102870 2024-03-19T03:30:25.778 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:30:25.778 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/54638259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":54638259},{"type":"v1","addr":"172.21.15.117:6835","nonce":54638259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3608185039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3608185039},{"type":"v1","addr":"172.21.15.117:6837","nonce":3608185039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/1291833214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1291833214},{"type":"v1","addr":"172.21.15.119:6833","nonce":1291833214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1579860804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1579860804},{"type":"v1","addr":"172.21.15.178:6835","nonce":1579860804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24515,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/1686938941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1686938941},{"type":"v1","addr":"172.21.15.119:6835","nonce":1686938941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1770913538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":1770913538},{"type":"v1","addr":"172.21.15.178:6837","nonce":1770913538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6839/2655593383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2655593383},{"type":"v1","addr":"172.21.15.178:6839","nonce":2655593383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2054992738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2054992738},{"type":"v1","addr":"172.21.15.119:6837","nonce":2054992738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24569,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6839/684335078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":684335078},{"type":"v1","addr":"172.21.15.117:6839","nonce":684335078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24581,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6841/1329933892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6840","nonce":1329933892},{"type":"v1","addr":"172.21.15.117:6841","nonce":1329933892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[]} 2024-03-19T03:30:25.779 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 12 2024-03-19T03:30:25.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.780+0000 7f260b200640 1 -- 172.21.15.117:0/776127464 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f25f00a2880 msgr2=0x7f25f00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:25.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.780+0000 7f260b200640 1 --2- 172.21.15.117:0/776127464 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f25f00a2880 0x7f25f00a4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f260c1026d0 tx=0x7f25fc009260 comp rx=0 tx=0).stop 2024-03-19T03:30:25.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.780+0000 7f260b200640 1 -- 172.21.15.117:0/776127464 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f260c102870 msgr2=0x7f260c19a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:25.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.780+0000 7f260b200640 1 --2- 172.21.15.117:0/776127464 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f260c102870 0x7f260c19a0b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f25f8005940 tx=0x7f25f8032470 comp rx=0 tx=0).stop 2024-03-19T03:30:25.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.780+0000 7f260b200640 1 -- 172.21.15.117:0/776127464 shutdown_connections 2024-03-19T03:30:25.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.780+0000 7f260b200640 1 --2- 172.21.15.117:0/776127464 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f260c103230 0x7f260c1a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:25.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.780+0000 7f260b200640 1 --2- 172.21.15.117:0/776127464 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f25f00a2880 0x7f25f00a4d40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:25.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.780+0000 7f260b200640 1 --2- 172.21.15.117:0/776127464 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f260c102870 0x7f260c19a0b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:25.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.780+0000 7f260b200640 1 --2- 172.21.15.117:0/776127464 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f260c101680 0x7f260c199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:25.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.780+0000 7f260b200640 1 -- 172.21.15.117:0/776127464 >> 172.21.15.117:0/776127464 conn(0x7f260c0fcbf0 msgr2=0x7f260c073a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:25.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.780+0000 7f260b200640 1 -- 172.21.15.117:0/776127464 shutdown_connections 2024-03-19T03:30:25.782 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:25.780+0000 7f260b200640 1 -- 172.21.15.117:0/776127464 wait complete. 2024-03-19T03:30:25.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:25 smithi117 ceph-mon[31838]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:25.792 INFO:journalctl@ceph.mds.a.smithi117.stdout:Mar 19 03:30:25 smithi117 podman[73983]: 2024-03-19 03:30:25.319579943 +0000 UTC m=+0.465139512 container init 175e7bb211795cd8eff5e48e000853619e9cdadaaf1575e7cc494fe06a2e8e5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:30:25.792 INFO:journalctl@ceph.mds.a.smithi117.stdout:Mar 19 03:30:25 smithi117 podman[73983]: 2024-03-19 03:30:25.332014233 +0000 UTC m=+0.477573814 container start 175e7bb211795cd8eff5e48e000853619e9cdadaaf1575e7cc494fe06a2e8e5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-a, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-19T03:30:25.792 INFO:journalctl@ceph.mds.a.smithi117.stdout:Mar 19 03:30:25 smithi117 ceph-mds[74018]: -- 172.21.15.117:0/1856232268 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557e020e9a40 con 0x557e020bcc00 2024-03-19T03:30:25.792 INFO:journalctl@ceph.mds.a.smithi117.stdout:Mar 19 03:30:25 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-a[73998]: starting mds.a at 2024-03-19T03:30:25.792 INFO:journalctl@ceph.mds.a.smithi117.stdout:Mar 19 03:30:25 smithi117 bash[73983]: 175e7bb211795cd8eff5e48e000853619e9cdadaaf1575e7cc494fe06a2e8e5c 2024-03-19T03:30:25.792 INFO:journalctl@ceph.mds.a.smithi117.stdout:Mar 19 03:30:25 smithi117 systemd[1]: Started Ceph mds.a for a3a82456-e59f-11ee-95c9-87774f69a715. 2024-03-19T03:30:25.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:25 smithi178 ceph-mon[67578]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:26.535 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:26 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] up:boot 2024-03-19T03:30:26.535 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:26 smithi119 ceph-mon[66861]: fsmap 11 up:standby 2024-03-19T03:30:26.535 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:26 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T03:30:26.535 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:26 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:26.535 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:26 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:26.535 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:26 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:26.535 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:26 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:30:26.535 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:26 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:30:26.535 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:26 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:26.535 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:26 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/776127464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:26.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:26 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] up:boot 2024-03-19T03:30:26.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:26 smithi117 ceph-mon[31838]: fsmap 11 up:standby 2024-03-19T03:30:26.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:26 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T03:30:26.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:26 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:26.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:26 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:26.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:26 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:26.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:26 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:30:26.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:26 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:30:26.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:26 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:26.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:26 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/776127464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:26.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:26 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] up:boot 2024-03-19T03:30:26.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:26 smithi178 ceph-mon[67578]: fsmap 11 up:standby 2024-03-19T03:30:26.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:26 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T03:30:26.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:26 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:26.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:26 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:26.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:26 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:26.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:26 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T03:30:26.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:26 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T03:30:26.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:26 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:26.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:26 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/776127464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:27.593 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:27 smithi119 ceph-mon[66861]: Deploying daemon mds.b on smithi119 2024-03-19T03:30:27.593 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:27 smithi119 ceph-mon[66861]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:27.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:27 smithi117 ceph-mon[31838]: Deploying daemon mds.b on smithi119 2024-03-19T03:30:27.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:27 smithi117 ceph-mon[31838]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:27.799 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:30:27.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:27 smithi178 ceph-mon[67578]: Deploying daemon mds.b on smithi119 2024-03-19T03:30:27.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:27 smithi178 ceph-mon[67578]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:27.984 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 19 03:30:27 smithi119 systemd[1]: Starting Ceph mds.b for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:30:28.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.021+0000 7fcf1cc00640 1 -- 172.21.15.117:0/2481571160 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf18109970 msgr2=0x7fcf18110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:28.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.021+0000 7fcf1cc00640 1 --2- 172.21.15.117:0/2481571160 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf18109970 0x7fcf18110290 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcf10009860 tx=0x7fcf1002bb70 comp rx=0 tx=0).stop 2024-03-19T03:30:28.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.021+0000 7fcf1cc00640 1 -- 172.21.15.117:0/2481571160 shutdown_connections 2024-03-19T03:30:28.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.021+0000 7fcf1cc00640 1 --2- 172.21.15.117:0/2481571160 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf18109970 0x7fcf18110290 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:28.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.021+0000 7fcf1cc00640 1 --2- 172.21.15.117:0/2481571160 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf18108fb0 0x7fcf18109430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:28.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.021+0000 7fcf1cc00640 1 --2- 172.21.15.117:0/2481571160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf18107e60 0x7fcf18108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:28.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.021+0000 7fcf1cc00640 1 -- 172.21.15.117:0/2481571160 >> 172.21.15.117:0/2481571160 conn(0x7fcf18076e30 msgr2=0x7fcf18079250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:28.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.021+0000 7fcf1cc00640 1 -- 172.21.15.117:0/2481571160 shutdown_connections 2024-03-19T03:30:28.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.021+0000 7fcf1cc00640 1 -- 172.21.15.117:0/2481571160 wait complete. 2024-03-19T03:30:28.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.022+0000 7fcf1cc00640 1 Processor -- start 2024-03-19T03:30:28.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.023+0000 7fcf1cc00640 1 -- start start 2024-03-19T03:30:28.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.023+0000 7fcf1cc00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf18107e60 0x7fcf1819e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:28.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.023+0000 7fcf1cc00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf18108fb0 0x7fcf1819e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:28.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.023+0000 7fcf1cc00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf18109970 0x7fcf181a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:28.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.023+0000 7fcf1cc00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcf18113ae0 con 0x7fcf18107e60 2024-03-19T03:30:28.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.023+0000 7fcf1cc00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fcf18113960 con 0x7fcf18109970 2024-03-19T03:30:28.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.023+0000 7fcf1cc00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fcf18113c60 con 0x7fcf18108fb0 2024-03-19T03:30:28.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.023+0000 7fcf15e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf18107e60 0x7fcf1819e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:28.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.023+0000 7fcf15e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf18107e60 0x7fcf1819e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:54344/0 (socket says 172.21.15.117:54344) 2024-03-19T03:30:28.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.023+0000 7fcf15e00640 1 -- 172.21.15.117:0/1363408136 learned_addr learned my addr 172.21.15.117:0/1363408136 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:28.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.023+0000 7fcf15400640 1 --2- 172.21.15.117:0/1363408136 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf18108fb0 0x7fcf1819e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:28.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.023+0000 7fcf16800640 1 --2- 172.21.15.117:0/1363408136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf18109970 0x7fcf181a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:28.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.023+0000 7fcf15e00640 1 -- 172.21.15.117:0/1363408136 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf18108fb0 msgr2=0x7fcf1819e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:28.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.023+0000 7fcf15e00640 1 --2- 172.21.15.117:0/1363408136 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf18108fb0 0x7fcf1819e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:28.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.023+0000 7fcf15e00640 1 -- 172.21.15.117:0/1363408136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf18109970 msgr2=0x7fcf181a5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:28.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.023+0000 7fcf15e00640 1 --2- 172.21.15.117:0/1363408136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf18109970 0x7fcf181a5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:28.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.023+0000 7fcf15e00640 1 -- 172.21.15.117:0/1363408136 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf181a5cc0 con 0x7fcf18107e60 2024-03-19T03:30:28.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.024+0000 7fcf15e00640 1 --2- 172.21.15.117:0/1363408136 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf18107e60 0x7fcf1819e0a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcf08009d50 tx=0x7fcf0800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:28.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.024+0000 7fcf0ea00640 1 -- 172.21.15.117:0/1363408136 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf08003b70 con 0x7fcf18107e60 2024-03-19T03:30:28.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.024+0000 7fcf0ea00640 1 -- 172.21.15.117:0/1363408136 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf08003d10 con 0x7fcf18107e60 2024-03-19T03:30:28.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.024+0000 7fcf1cc00640 1 -- 172.21.15.117:0/1363408136 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf181a5fb0 con 0x7fcf18107e60 2024-03-19T03:30:28.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.024+0000 7fcf0ea00640 1 -- 172.21.15.117:0/1363408136 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf080108f0 con 0x7fcf18107e60 2024-03-19T03:30:28.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.024+0000 7fcf1cc00640 1 -- 172.21.15.117:0/1363408136 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf181a6410 con 0x7fcf18107e60 2024-03-19T03:30:28.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.025+0000 7fcf1cc00640 1 -- 172.21.15.117:0/1363408136 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf1810f110 con 0x7fcf18107e60 2024-03-19T03:30:28.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.027+0000 7fcf0ea00640 1 -- 172.21.15.117:0/1363408136 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fcf08010a90 con 0x7fcf18107e60 2024-03-19T03:30:28.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.030+0000 7fcf0ea00640 1 --2- 172.21.15.117:0/1363408136 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fcefc0a2800 0x7fcefc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:28.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.030+0000 7fcf15400640 1 --2- 172.21.15.117:0/1363408136 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fcefc0a2800 0x7fcefc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:28.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.030+0000 7fcf0ea00640 1 -- 172.21.15.117:0/1363408136 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fcf080cee80 con 0x7fcf18107e60 2024-03-19T03:30:28.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.030+0000 7fcf15400640 1 --2- 172.21.15.117:0/1363408136 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fcefc0a2800 0x7fcefc0a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fcf1819f5b0 tx=0x7fcf04009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:28.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.033+0000 7fcf0ea00640 1 -- 172.21.15.117:0/1363408136 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcf08053740 con 0x7fcf18107e60 2024-03-19T03:30:28.270 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.269+0000 7fcf1cc00640 1 -- 172.21.15.117:0/1363408136 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf1802d1f0 con 0x7fcf18107e60 2024-03-19T03:30:28.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.271+0000 7fcf0ea00640 1 -- 172.21.15.117:0/1363408136 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9671 (secure 0 0 0) 0x7fcf08086090 con 0x7fcf18107e60 2024-03-19T03:30:28.272 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:30:28.272 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/54638259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":54638259},{"type":"v1","addr":"172.21.15.117:6835","nonce":54638259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3608185039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3608185039},{"type":"v1","addr":"172.21.15.117:6837","nonce":3608185039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/1291833214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1291833214},{"type":"v1","addr":"172.21.15.119:6833","nonce":1291833214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1579860804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1579860804},{"type":"v1","addr":"172.21.15.178:6835","nonce":1579860804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24515,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/1686938941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1686938941},{"type":"v1","addr":"172.21.15.119:6835","nonce":1686938941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1770913538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":1770913538},{"type":"v1","addr":"172.21.15.178:6837","nonce":1770913538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6839/2655593383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2655593383},{"type":"v1","addr":"172.21.15.178:6839","nonce":2655593383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2054992738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2054992738},{"type":"v1","addr":"172.21.15.119:6837","nonce":2054992738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24569,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6839/684335078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":684335078},{"type":"v1","addr":"172.21.15.117:6839","nonce":684335078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24581,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6841/1329933892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6840","nonce":1329933892},{"type":"v1","addr":"172.21.15.117:6841","nonce":1329933892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[]} 2024-03-19T03:30:28.273 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 12 2024-03-19T03:30:28.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.274+0000 7fcf1cc00640 1 -- 172.21.15.117:0/1363408136 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fcefc0a2800 msgr2=0x7fcefc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:28.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.274+0000 7fcf1cc00640 1 --2- 172.21.15.117:0/1363408136 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fcefc0a2800 0x7fcefc0a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fcf1819f5b0 tx=0x7fcf04009260 comp rx=0 tx=0).stop 2024-03-19T03:30:28.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.274+0000 7fcf1cc00640 1 -- 172.21.15.117:0/1363408136 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf18107e60 msgr2=0x7fcf1819e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:28.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.274+0000 7fcf1cc00640 1 --2- 172.21.15.117:0/1363408136 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf18107e60 0x7fcf1819e0a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcf08009d50 tx=0x7fcf0800c6a0 comp rx=0 tx=0).stop 2024-03-19T03:30:28.275 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.274+0000 7fcf1cc00640 1 -- 172.21.15.117:0/1363408136 shutdown_connections 2024-03-19T03:30:28.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.274+0000 7fcf1cc00640 1 --2- 172.21.15.117:0/1363408136 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fcefc0a2800 0x7fcefc0a4cc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:28.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.274+0000 7fcf1cc00640 1 --2- 172.21.15.117:0/1363408136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf18109970 0x7fcf181a5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:28.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.274+0000 7fcf1cc00640 1 --2- 172.21.15.117:0/1363408136 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf18108fb0 0x7fcf1819e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:28.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.274+0000 7fcf1cc00640 1 --2- 172.21.15.117:0/1363408136 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf18107e60 0x7fcf1819e0a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:28.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.274+0000 7fcf1cc00640 1 -- 172.21.15.117:0/1363408136 >> 172.21.15.117:0/1363408136 conn(0x7fcf18076e30 msgr2=0x7fcf1810c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:28.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.274+0000 7fcf1cc00640 1 -- 172.21.15.117:0/1363408136 shutdown_connections 2024-03-19T03:30:28.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:28.274+0000 7fcf1cc00640 1 -- 172.21.15.117:0/1363408136 wait complete. 2024-03-19T03:30:28.296 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 19 03:30:27 smithi119 podman[97005]: 2024-03-19 03:30:27.899833354 +0000 UTC m=+0.017589169 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:30:28.296 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 19 03:30:28 smithi119 podman[97005]: 2024-03-19 03:30:28.034132241 +0000 UTC m=+0.151888063 container create de4c1844e91141545d56c0000d4caf0b17dbbe4314c83a196b3e14f4a8f8d1ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T03:30:28.296 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 19 03:30:28 smithi119 podman[97005]: 2024-03-19 03:30:28.295755234 +0000 UTC m=+0.413511050 container init de4c1844e91141545d56c0000d4caf0b17dbbe4314c83a196b3e14f4a8f8d1ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-b, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True) 2024-03-19T03:30:28.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:28 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1363408136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:28.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:28 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1363408136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:28.734 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 19 03:30:28 smithi119 podman[97005]: 2024-03-19 03:30:28.298535586 +0000 UTC m=+0.416291396 container start de4c1844e91141545d56c0000d4caf0b17dbbe4314c83a196b3e14f4a8f8d1ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2) 2024-03-19T03:30:28.734 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 19 03:30:28 smithi119 ceph-mds[97025]: -- 172.21.15.119:0/1344625416 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fabcd81a40 con 0x55fabcd55000 2024-03-19T03:30:28.734 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 19 03:30:28 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-b[97021]: starting mds.b at 2024-03-19T03:30:28.734 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 19 03:30:28 smithi119 bash[97005]: de4c1844e91141545d56c0000d4caf0b17dbbe4314c83a196b3e14f4a8f8d1ea 2024-03-19T03:30:28.734 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 19 03:30:28 smithi119 systemd[1]: Started Ceph mds.b for a3a82456-e59f-11ee-95c9-87774f69a715. 2024-03-19T03:30:28.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:28 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1363408136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:29.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:29 smithi119 ceph-mon[66861]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:29.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:29 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] up:boot 2024-03-19T03:30:29.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:29 smithi119 ceph-mon[66861]: fsmap 12 up:standby 2024-03-19T03:30:29.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:29 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T03:30:29.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:29 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:29.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:29 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:29.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:29 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:29.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:29 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:29.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:29 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:29.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:29 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:30:29.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:29 smithi117 ceph-mon[31838]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:29.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:29 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] up:boot 2024-03-19T03:30:29.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:29 smithi117 ceph-mon[31838]: fsmap 12 up:standby 2024-03-19T03:30:29.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:29 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T03:30:29.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:29 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:29.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:29 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:29.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:29 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:29.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:29 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:29.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:29 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:29.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:29 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:30:29.802 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:29 smithi178 ceph-mon[67578]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:29.803 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:29 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] up:boot 2024-03-19T03:30:29.803 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:29 smithi178 ceph-mon[67578]: fsmap 12 up:standby 2024-03-19T03:30:29.803 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:29 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T03:30:29.803 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:29 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:29.803 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:29 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:29.803 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:29 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:29.803 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:29 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:29.803 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:29 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:29.803 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:29 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:30:30.295 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:30:30.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.498+0000 7f0433200640 1 -- 172.21.15.117:0/3941737542 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0434104a60 msgr2=0x7f0434104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:30.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.498+0000 7f0433200640 1 --2- 172.21.15.117:0/3941737542 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0434104a60 0x7f0434104ee0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0420009900 tx=0x7f042002bc30 comp rx=0 tx=0).stop 2024-03-19T03:30:30.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.498+0000 7f0433200640 1 -- 172.21.15.117:0/3941737542 shutdown_connections 2024-03-19T03:30:30.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.498+0000 7f0433200640 1 --2- 172.21.15.117:0/3941737542 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0434105420 0x7f043410bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:30.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.498+0000 7f0433200640 1 --2- 172.21.15.117:0/3941737542 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0434104a60 0x7f0434104ee0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:30.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.498+0000 7f0433200640 1 --2- 172.21.15.117:0/3941737542 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0434103870 0x7f0434103c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:30.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.498+0000 7f0433200640 1 -- 172.21.15.117:0/3941737542 >> 172.21.15.117:0/3941737542 conn(0x7f04340fee00 msgr2=0x7f0434101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:30.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.498+0000 7f0433200640 1 -- 172.21.15.117:0/3941737542 shutdown_connections 2024-03-19T03:30:30.500 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.498+0000 7f0433200640 1 -- 172.21.15.117:0/3941737542 wait complete. 2024-03-19T03:30:30.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.499+0000 7f0433200640 1 Processor -- start 2024-03-19T03:30:30.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.499+0000 7f0433200640 1 -- start start 2024-03-19T03:30:30.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.499+0000 7f0433200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0434103870 0x7f0434199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:30.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.499+0000 7f0433200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0434104a60 0x7f043419a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:30.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.499+0000 7f0433200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0434105420 0x7f04341a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:30.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.499+0000 7f0433200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f043410f4d0 con 0x7f0434104a60 2024-03-19T03:30:30.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.499+0000 7f0433200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f043410f350 con 0x7f0434105420 2024-03-19T03:30:30.502 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.499+0000 7f0433200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f043410f650 con 0x7f0434103870 2024-03-19T03:30:30.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.499+0000 7f0431e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0434103870 0x7f0434199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:30.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.499+0000 7f0431e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0434103870 0x7f0434199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:35012/0 (socket says 172.21.15.117:35012) 2024-03-19T03:30:30.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.499+0000 7f0431e00640 1 -- 172.21.15.117:0/2355979594 learned_addr learned my addr 172.21.15.117:0/2355979594 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:30.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.500+0000 7f0432800640 1 --2- 172.21.15.117:0/2355979594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0434105420 0x7f04341a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:30.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.500+0000 7f0431400640 1 --2- 172.21.15.117:0/2355979594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0434104a60 0x7f043419a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:30.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.500+0000 7f0431e00640 1 -- 172.21.15.117:0/2355979594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0434105420 msgr2=0x7f04341a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:30.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.500+0000 7f0431e00640 1 --2- 172.21.15.117:0/2355979594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0434105420 0x7f04341a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:30.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.500+0000 7f0431e00640 1 -- 172.21.15.117:0/2355979594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0434104a60 msgr2=0x7f043419a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:30.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.500+0000 7f0431e00640 1 --2- 172.21.15.117:0/2355979594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0434104a60 0x7f043419a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:30.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.500+0000 7f0431e00640 1 -- 172.21.15.117:0/2355979594 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04341a1710 con 0x7f0434103870 2024-03-19T03:30:30.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.500+0000 7f0431e00640 1 --2- 172.21.15.117:0/2355979594 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0434103870 0x7f0434199af0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f042400e9c0 tx=0x7f042400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:30.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.500+0000 7f042aa00640 1 -- 172.21.15.117:0/2355979594 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042400cd80 con 0x7f0434103870 2024-03-19T03:30:30.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.500+0000 7f042aa00640 1 -- 172.21.15.117:0/2355979594 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04240035f0 con 0x7f0434103870 2024-03-19T03:30:30.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.500+0000 7f042aa00640 1 -- 172.21.15.117:0/2355979594 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0424010690 con 0x7f0434103870 2024-03-19T03:30:30.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.501+0000 7f0433200640 1 -- 172.21.15.117:0/2355979594 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04341a1a00 con 0x7f0434103870 2024-03-19T03:30:30.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.501+0000 7f0433200640 1 -- 172.21.15.117:0/2355979594 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04341a1ed0 con 0x7f0434103870 2024-03-19T03:30:30.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.502+0000 7f0433200640 1 -- 172.21.15.117:0/2355979594 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f043410ab90 con 0x7f0434103870 2024-03-19T03:30:30.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.503+0000 7f042aa00640 1 -- 172.21.15.117:0/2355979594 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f04240037a0 con 0x7f0434103870 2024-03-19T03:30:30.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.505+0000 7f042aa00640 1 --2- 172.21.15.117:0/2355979594 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f04180a2880 0x7f04180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:30.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.505+0000 7f042aa00640 1 -- 172.21.15.117:0/2355979594 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f04240ce990 con 0x7f0434103870 2024-03-19T03:30:30.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.505+0000 7f0431400640 1 --2- 172.21.15.117:0/2355979594 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f04180a2880 0x7f04180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:30.506 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.506+0000 7f0431400640 1 --2- 172.21.15.117:0/2355979594 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f04180a2880 0x7f04180a4d40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f042002c7b0 tx=0x7f0420005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:30.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.509+0000 7f042aa00640 1 -- 172.21.15.117:0/2355979594 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f042409be30 con 0x7f0434103870 2024-03-19T03:30:30.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.764+0000 7f0433200640 1 -- 172.21.15.117:0/2355979594 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04340624c0 con 0x7f0434103870 2024-03-19T03:30:30.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.765+0000 7f042aa00640 1 -- 172.21.15.117:0/2355979594 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10500 (secure 0 0 0) 0x7f042409b730 con 0x7f0434103870 2024-03-19T03:30:30.766 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:30:30.766 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/54638259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":54638259},{"type":"v1","addr":"172.21.15.117:6835","nonce":54638259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3608185039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3608185039},{"type":"v1","addr":"172.21.15.117:6837","nonce":3608185039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/1291833214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1291833214},{"type":"v1","addr":"172.21.15.119:6833","nonce":1291833214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1579860804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1579860804},{"type":"v1","addr":"172.21.15.178:6835","nonce":1579860804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24515,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/1686938941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1686938941},{"type":"v1","addr":"172.21.15.119:6835","nonce":1686938941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1770913538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":1770913538},{"type":"v1","addr":"172.21.15.178:6837","nonce":1770913538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6839/2655593383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2655593383},{"type":"v1","addr":"172.21.15.178:6839","nonce":2655593383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2054992738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2054992738},{"type":"v1","addr":"172.21.15.119:6837","nonce":2054992738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24569,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6839/684335078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":684335078},{"type":"v1","addr":"172.21.15.117:6839","nonce":684335078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24581,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6841/1329933892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6840","nonce":1329933892},{"type":"v1","addr":"172.21.15.117:6841","nonce":1329933892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24599,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6839/2383354833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2383354833},{"type":"v1","addr":"172.21.15.119:6839","nonce":2383354833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-03-19T03:30:30.766 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 13 2024-03-19T03:30:30.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.767+0000 7f0433200640 1 -- 172.21.15.117:0/2355979594 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f04180a2880 msgr2=0x7f04180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:30.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.767+0000 7f0433200640 1 --2- 172.21.15.117:0/2355979594 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f04180a2880 0x7f04180a4d40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f042002c7b0 tx=0x7f0420005fd0 comp rx=0 tx=0).stop 2024-03-19T03:30:30.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.767+0000 7f0433200640 1 -- 172.21.15.117:0/2355979594 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0434103870 msgr2=0x7f0434199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:30.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.767+0000 7f0433200640 1 --2- 172.21.15.117:0/2355979594 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0434103870 0x7f0434199af0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f042400e9c0 tx=0x7f042400ee80 comp rx=0 tx=0).stop 2024-03-19T03:30:30.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.767+0000 7f0433200640 1 -- 172.21.15.117:0/2355979594 shutdown_connections 2024-03-19T03:30:30.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.767+0000 7f0433200640 1 --2- 172.21.15.117:0/2355979594 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f04180a2880 0x7f04180a4d40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:30.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.767+0000 7f0433200640 1 --2- 172.21.15.117:0/2355979594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0434105420 0x7f04341a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:30.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.767+0000 7f0433200640 1 --2- 172.21.15.117:0/2355979594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0434104a60 0x7f043419a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:30.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.767+0000 7f0433200640 1 --2- 172.21.15.117:0/2355979594 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0434103870 0x7f0434199af0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:30.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.767+0000 7f0433200640 1 -- 172.21.15.117:0/2355979594 >> 172.21.15.117:0/2355979594 conn(0x7f04340fee00 msgr2=0x7f0434107d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:30.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.768+0000 7f0433200640 1 -- 172.21.15.117:0/2355979594 shutdown_connections 2024-03-19T03:30:30.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.768+0000 7f0433200640 1 -- 172.21.15.117:0/2355979594 wait complete. 2024-03-19T03:30:30.786 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:30:30.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.986+0000 7f7bc0400640 1 -- 172.21.15.117:0/1462808383 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7bb8101840 msgr2=0x7f7bb8101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:30.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.986+0000 7f7bc0400640 1 --2- 172.21.15.117:0/1462808383 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7bb8101840 0x7f7bb8101c60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7bac009900 tx=0x7f7bac02bc50 comp rx=0 tx=0).stop 2024-03-19T03:30:30.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.986+0000 7f7bc0400640 1 -- 172.21.15.117:0/1462808383 shutdown_connections 2024-03-19T03:30:30.987 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.986+0000 7f7bc0400640 1 --2- 172.21.15.117:0/1462808383 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bb81033f0 0x7f7bb8109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:30.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.986+0000 7f7bc0400640 1 --2- 172.21.15.117:0/1462808383 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7bb8102a30 0x7f7bb8102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:30.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.986+0000 7f7bc0400640 1 --2- 172.21.15.117:0/1462808383 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7bb8101840 0x7f7bb8101c60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:30.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.986+0000 7f7bc0400640 1 -- 172.21.15.117:0/1462808383 >> 172.21.15.117:0/1462808383 conn(0x7f7bb80fcdd0 msgr2=0x7f7bb80ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:30.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.986+0000 7f7bc0400640 1 -- 172.21.15.117:0/1462808383 shutdown_connections 2024-03-19T03:30:30.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.986+0000 7f7bc0400640 1 -- 172.21.15.117:0/1462808383 wait complete. 2024-03-19T03:30:30.989 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.987+0000 7f7bc0400640 1 Processor -- start 2024-03-19T03:30:30.989 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.987+0000 7f7bc0400640 1 -- start start 2024-03-19T03:30:30.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.988+0000 7f7bc0400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bb8101840 0x7f7bb8197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:30.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.988+0000 7f7bc0400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7bb8102a30 0x7f7bb81980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:30.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.988+0000 7f7bc0400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7bb81033f0 0x7f7bb819f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:30.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.988+0000 7f7bc0400640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb810d510 con 0x7f7bb81033f0 2024-03-19T03:30:30.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.988+0000 7f7bc0400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb810d390 con 0x7f7bb8102a30 2024-03-19T03:30:30.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.988+0000 7f7bc0400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb810d690 con 0x7f7bb8101840 2024-03-19T03:30:30.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.988+0000 7f7bbda00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bb8101840 0x7f7bb8197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:30.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.988+0000 7f7bbd000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7bb8102a30 0x7f7bb81980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:30.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.988+0000 7f7bbda00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bb8101840 0x7f7bb8197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:35032/0 (socket says 172.21.15.117:35032) 2024-03-19T03:30:30.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.988+0000 7f7bbda00640 1 -- 172.21.15.117:0/2174710813 learned_addr learned my addr 172.21.15.117:0/2174710813 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:30.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.988+0000 7f7bbe400640 1 --2- 172.21.15.117:0/2174710813 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7bb81033f0 0x7f7bb819f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:30.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.988+0000 7f7bbd000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7bb8102a30 0x7f7bb81980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:59044/0 (socket says 172.21.15.117:59044) 2024-03-19T03:30:30.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.988+0000 7f7bbda00640 1 -- 172.21.15.117:0/2174710813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7bb8102a30 msgr2=0x7f7bb81980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:30.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.988+0000 7f7bbda00640 1 --2- 172.21.15.117:0/2174710813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7bb8102a30 0x7f7bb81980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:30.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.988+0000 7f7bbda00640 1 -- 172.21.15.117:0/2174710813 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7bb81033f0 msgr2=0x7f7bb819f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:30.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.988+0000 7f7bbda00640 1 --2- 172.21.15.117:0/2174710813 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7bb81033f0 0x7f7bb819f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:30.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.989+0000 7f7bbda00640 1 -- 172.21.15.117:0/2174710813 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bb819f7a0 con 0x7f7bb8101840 2024-03-19T03:30:30.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.989+0000 7f7bbd000640 1 --2- 172.21.15.117:0/2174710813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7bb8102a30 0x7f7bb81980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:30:30.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.989+0000 7f7bbda00640 1 --2- 172.21.15.117:0/2174710813 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bb8101840 0x7f7bb8197b80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7bac02c260 tx=0x7f7bac02c670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:30.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.989+0000 7f7bb6000640 1 -- 172.21.15.117:0/2174710813 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bac041070 con 0x7f7bb8101840 2024-03-19T03:30:30.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.989+0000 7f7bb6000640 1 -- 172.21.15.117:0/2174710813 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bac004450 con 0x7f7bb8101840 2024-03-19T03:30:30.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.989+0000 7f7bb6000640 1 -- 172.21.15.117:0/2174710813 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bac034440 con 0x7f7bb8101840 2024-03-19T03:30:30.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.990+0000 7f7bc0400640 1 -- 172.21.15.117:0/2174710813 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bb819fa30 con 0x7f7bb8101840 2024-03-19T03:30:30.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.990+0000 7f7bc0400640 1 -- 172.21.15.117:0/2174710813 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bb819ff30 con 0x7f7bb8101840 2024-03-19T03:30:30.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.991+0000 7f7bc0400640 1 -- 172.21.15.117:0/2174710813 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bb8108b60 con 0x7f7bb8101840 2024-03-19T03:30:30.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.992+0000 7f7bb6000640 1 -- 172.21.15.117:0/2174710813 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f7bac044070 con 0x7f7bb8101840 2024-03-19T03:30:30.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.994+0000 7f7bb6000640 1 --2- 172.21.15.117:0/2174710813 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7ba00a2800 0x7f7ba00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:30.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.994+0000 7f7bb6000640 1 -- 172.21.15.117:0/2174710813 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f7bac0f00e0 con 0x7f7bb8101840 2024-03-19T03:30:30.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.994+0000 7f7bbd000640 1 --2- 172.21.15.117:0/2174710813 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7ba00a2800 0x7f7ba00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:30.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.995+0000 7f7bbd000640 1 --2- 172.21.15.117:0/2174710813 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7ba00a2800 0x7f7ba00a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8199090 tx=0x7f7ba4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:30.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:30.998+0000 7f7bb6000640 1 -- 172.21.15.117:0/2174710813 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7bac0bcf30 con 0x7f7bb8101840 2024-03-19T03:30:31.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.240+0000 7f7bc0400640 1 -- 172.21.15.117:0/2174710813 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7bb8062420 con 0x7f7bb8101840 2024-03-19T03:30:31.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.241+0000 7f7bb6000640 1 -- 172.21.15.117:0/2174710813 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18538 (secure 0 0 0) 0x7f7bac0c0fd0 con 0x7f7bb8101840 2024-03-19T03:30:31.242 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:30:31.242 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":81,"fsid":"a3a82456-e59f-11ee-95c9-87774f69a715","created":"2024-03-19T03:21:30.427136+0000","modified":"2024-03-19T03:29:34.539579+0000","last_up_change":"2024-03-19T03:29:32.967505+0000","last_in_change":"2024-03-19T03:29:12.848740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:25:18.824270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b32eb0ee-9d4b-4748-87ec-eda02c907415","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6803","nonce":3014934900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6805","nonce":3014934900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6809","nonce":3014934900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6807","nonce":3014934900}]},"public_addr":"172.21.15.117:6803/3014934900","cluster_addr":"172.21.15.117:6805/3014934900","heartbeat_back_addr":"172.21.15.117:6809/3014934900","heartbeat_front_addr":"172.21.15.117:6807/3014934900","state":["exists","up"]},{"osd":1,"uuid":"e07585be-c016-4466-b07c-abbc22a58de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6801","nonce":2425927341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6803","nonce":2425927341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6807","nonce":2425927341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6805","nonce":2425927341}]},"public_addr":"172.21.15.119:6801/2425927341","cluster_addr":"172.21.15.119:6803/2425927341","heartbeat_back_addr":"172.21.15.119:6807/2425927341","heartbeat_front_addr":"172.21.15.119:6805/2425927341","state":["exists","up"]},{"osd":2,"uuid":"ede9edda-3b9f-41ca-ad04-5dcb738ab757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6801","nonce":3733812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6803","nonce":3733812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6807","nonce":3733812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6805","nonce":3733812989}]},"public_addr":"172.21.15.178:6801/3733812989","cluster_addr":"172.21.15.178:6803/3733812989","heartbeat_back_addr":"172.21.15.178:6807/3733812989","heartbeat_front_addr":"172.21.15.178:6805/3733812989","state":["exists","up"]},{"osd":3,"uuid":"f81c41e1-3a69-40db-a90f-58151286b081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6811","nonce":2178310135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6813","nonce":2178310135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6817","nonce":2178310135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6815","nonce":2178310135}]},"public_addr":"172.21.15.117:6811/2178310135","cluster_addr":"172.21.15.117:6813/2178310135","heartbeat_back_addr":"172.21.15.117:6817/2178310135","heartbeat_front_addr":"172.21.15.117:6815/2178310135","state":["exists","up"]},{"osd":4,"uuid":"caf7ce86-caf0-4eaf-bbab-bd2d93e63c33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6809","nonce":3455398086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6811","nonce":3455398086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6815","nonce":3455398086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6813","nonce":3455398086}]},"public_addr":"172.21.15.119:6809/3455398086","cluster_addr":"172.21.15.119:6811/3455398086","heartbeat_back_addr":"172.21.15.119:6815/3455398086","heartbeat_front_addr":"172.21.15.119:6813/3455398086","state":["exists","up"]},{"osd":5,"uuid":"20ba0b8a-2e0f-4ea7-b435-430e44766200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6809","nonce":1259631912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6811","nonce":1259631912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6815","nonce":1259631912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6813","nonce":1259631912}]},"public_addr":"172.21.15.178:6809/1259631912","cluster_addr":"172.21.15.178:6811/1259631912","heartbeat_back_addr":"172.21.15.178:6815/1259631912","heartbeat_front_addr":"172.21.15.178:6813/1259631912","state":["exists","up"]},{"osd":6,"uuid":"75c2ddd7-f837-42e0-ac47-8cfbcf97a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6819","nonce":864146757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6821","nonce":864146757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6825","nonce":864146757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6823","nonce":864146757}]},"public_addr":"172.21.15.117:6819/864146757","cluster_addr":"172.21.15.117:6821/864146757","heartbeat_back_addr":"172.21.15.117:6825/864146757","heartbeat_front_addr":"172.21.15.117:6823/864146757","state":["exists","up"]},{"osd":7,"uuid":"929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6817","nonce":1273365545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6819","nonce":1273365545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6823","nonce":1273365545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6821","nonce":1273365545}]},"public_addr":"172.21.15.119:6817/1273365545","cluster_addr":"172.21.15.119:6819/1273365545","heartbeat_back_addr":"172.21.15.119:6823/1273365545","heartbeat_front_addr":"172.21.15.119:6821/1273365545","state":["exists","up"]},{"osd":8,"uuid":"fd371461-a538-4781-92e1-5eead4ca3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6817","nonce":1059587501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6819","nonce":1059587501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6823","nonce":1059587501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6821","nonce":1059587501}]},"public_addr":"172.21.15.178:6817/1059587501","cluster_addr":"172.21.15.178:6819/1059587501","heartbeat_back_addr":"172.21.15.178:6823/1059587501","heartbeat_front_addr":"172.21.15.178:6821/1059587501","state":["exists","up"]},{"osd":9,"uuid":"0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6827","nonce":3653875600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6829","nonce":3653875600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6833","nonce":3653875600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6831","nonce":3653875600}]},"public_addr":"172.21.15.117:6827/3653875600","cluster_addr":"172.21.15.117:6829/3653875600","heartbeat_back_addr":"172.21.15.117:6833/3653875600","heartbeat_front_addr":"172.21.15.117:6831/3653875600","state":["exists","up"]},{"osd":10,"uuid":"de2049da-d19e-4386-afbf-fd6b5fddbcba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6825","nonce":1050976194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6827","nonce":1050976194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6831","nonce":1050976194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6829","nonce":1050976194}]},"public_addr":"172.21.15.119:6825/1050976194","cluster_addr":"172.21.15.119:6827/1050976194","heartbeat_back_addr":"172.21.15.119:6831/1050976194","heartbeat_front_addr":"172.21.15.119:6829/1050976194","state":["exists","up"]},{"osd":11,"uuid":"3cccc0f9-d273-4b6b-9b8e-c926371472dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6825","nonce":2915080368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6827","nonce":2915080368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6831","nonce":2915080368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6829","nonce":2915080368}]},"public_addr":"172.21.15.178:6825/2915080368","cluster_addr":"172.21.15.178:6827/2915080368","heartbeat_back_addr":"172.21.15.178:6831/2915080368","heartbeat_front_addr":"172.21.15.178:6829/2915080368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:27.876274+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:53.094706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:16.598813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:48.160419+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:14.048925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:38.814304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:12.542328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:38.942747+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:04.488963+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:38.196053+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:05.254728+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:30.874915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.117:0/185684447":"2024-03-20T03:22:38.134578+0000","172.21.15.117:0/361367101":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/2788067603":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3506200675":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6800/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3283471541":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3750920169":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/2780813298":"2024-03-20T03:22:03.905461+0000","172.21.15.117:6800/2788067603":"2024-03-20T03:22:38.134578+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:30:31.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.243+0000 7f7bc0400640 1 -- 172.21.15.117:0/2174710813 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7ba00a2800 msgr2=0x7f7ba00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:31.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.243+0000 7f7bc0400640 1 --2- 172.21.15.117:0/2174710813 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7ba00a2800 0x7f7ba00a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8199090 tx=0x7f7ba4005f90 comp rx=0 tx=0).stop 2024-03-19T03:30:31.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.243+0000 7f7bc0400640 1 -- 172.21.15.117:0/2174710813 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bb8101840 msgr2=0x7f7bb8197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:31.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.243+0000 7f7bc0400640 1 --2- 172.21.15.117:0/2174710813 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bb8101840 0x7f7bb8197b80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7bac02c260 tx=0x7f7bac02c670 comp rx=0 tx=0).stop 2024-03-19T03:30:31.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.243+0000 7f7bc0400640 1 -- 172.21.15.117:0/2174710813 shutdown_connections 2024-03-19T03:30:31.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.243+0000 7f7bc0400640 1 --2- 172.21.15.117:0/2174710813 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7bb81033f0 0x7f7bb819f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:31.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.243+0000 7f7bc0400640 1 --2- 172.21.15.117:0/2174710813 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7ba00a2800 0x7f7ba00a4cc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:31.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.243+0000 7f7bc0400640 1 --2- 172.21.15.117:0/2174710813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7bb8102a30 0x7f7bb81980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:31.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.243+0000 7f7bc0400640 1 --2- 172.21.15.117:0/2174710813 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bb8101840 0x7f7bb8197b80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:31.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.243+0000 7f7bc0400640 1 -- 172.21.15.117:0/2174710813 >> 172.21.15.117:0/2174710813 conn(0x7f7bb80fcdd0 msgr2=0x7f7bb8105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:31.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.244+0000 7f7bc0400640 1 -- 172.21.15.117:0/2174710813 shutdown_connections 2024-03-19T03:30:31.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.244+0000 7f7bc0400640 1 -- 172.21.15.117:0/2174710813 wait complete. 2024-03-19T03:30:31.264 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T03:25:18.824270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T03:30:31.265 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T03:30:31.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.459+0000 7f2346600640 1 -- 172.21.15.117:0/1529224252 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23401028f0 msgr2=0x7f2340102d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:31.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.459+0000 7f2346600640 1 --2- 172.21.15.117:0/1529224252 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23401028f0 0x7f2340102d70 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f232c009900 tx=0x7f232c02bc30 comp rx=0 tx=0).stop 2024-03-19T03:30:31.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.459+0000 7f2346600640 1 -- 172.21.15.117:0/1529224252 shutdown_connections 2024-03-19T03:30:31.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.459+0000 7f2346600640 1 --2- 172.21.15.117:0/1529224252 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f23401032b0 0x7f2340109b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:31.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.459+0000 7f2346600640 1 --2- 172.21.15.117:0/1529224252 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23401028f0 0x7f2340102d70 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:31.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.459+0000 7f2346600640 1 --2- 172.21.15.117:0/1529224252 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2340101700 0x7f2340101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:31.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.459+0000 7f2346600640 1 -- 172.21.15.117:0/1529224252 >> 172.21.15.117:0/1529224252 conn(0x7f23400fcc90 msgr2=0x7f23400ff0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:31.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.459+0000 7f2346600640 1 -- 172.21.15.117:0/1529224252 shutdown_connections 2024-03-19T03:30:31.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.459+0000 7f2346600640 1 -- 172.21.15.117:0/1529224252 wait complete. 2024-03-19T03:30:31.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.460+0000 7f2346600640 1 Processor -- start 2024-03-19T03:30:31.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.460+0000 7f2346600640 1 -- start start 2024-03-19T03:30:31.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.461+0000 7f2346600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2340101700 0x7f23400755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:31.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.461+0000 7f2346600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23401028f0 0x7f2340073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:31.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.461+0000 7f2346600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23401032b0 0x7f2340074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:31.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.461+0000 7f2346600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f234010d3d0 con 0x7f23401032b0 2024-03-19T03:30:31.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.461+0000 7f2346600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f234010d250 con 0x7f23401028f0 2024-03-19T03:30:31.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.461+0000 7f2346600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f234010d550 con 0x7f2340101700 2024-03-19T03:30:31.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.461+0000 7f2345c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23401032b0 0x7f2340074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:31.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.461+0000 7f233fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23401028f0 0x7f2340073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:31.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.461+0000 7f2345c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23401032b0 0x7f2340074150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:54398/0 (socket says 172.21.15.117:54398) 2024-03-19T03:30:31.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.461+0000 7f2345200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2340101700 0x7f23400755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:31.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.461+0000 7f2345c00640 1 -- 172.21.15.117:0/1124055614 learned_addr learned my addr 172.21.15.117:0/1124055614 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:31.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.461+0000 7f233fe00640 1 -- 172.21.15.117:0/1124055614 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2340101700 msgr2=0x7f23400755d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:31.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.461+0000 7f233fe00640 1 --2- 172.21.15.117:0/1124055614 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2340101700 0x7f23400755d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:31.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.461+0000 7f233fe00640 1 -- 172.21.15.117:0/1124055614 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23401032b0 msgr2=0x7f2340074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:31.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.461+0000 7f233fe00640 1 --2- 172.21.15.117:0/1124055614 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23401032b0 0x7f2340074150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:31.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.461+0000 7f233fe00640 1 -- 172.21.15.117:0/1124055614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23400749e0 con 0x7f23401028f0 2024-03-19T03:30:31.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.462+0000 7f233fe00640 1 --2- 172.21.15.117:0/1124055614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23401028f0 0x7f2340073c10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f232c02c170 tx=0x7f232c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:31.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.462+0000 7f233d600640 1 -- 172.21.15.117:0/1124055614 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232c043020 con 0x7f23401028f0 2024-03-19T03:30:31.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.462+0000 7f2346600640 1 -- 172.21.15.117:0/1124055614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23401a1b50 con 0x7f23401028f0 2024-03-19T03:30:31.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.462+0000 7f233d600640 1 -- 172.21.15.117:0/1124055614 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f232c0054e0 con 0x7f23401028f0 2024-03-19T03:30:31.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.462+0000 7f233d600640 1 -- 172.21.15.117:0/1124055614 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232c003ca0 con 0x7f23401028f0 2024-03-19T03:30:31.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.463+0000 7f2346600640 1 -- 172.21.15.117:0/1124055614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23401a20c0 con 0x7f23401028f0 2024-03-19T03:30:31.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.464+0000 7f2346600640 1 -- 172.21.15.117:0/1124055614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23401089c0 con 0x7f23401028f0 2024-03-19T03:30:31.466 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.465+0000 7f233d600640 1 -- 172.21.15.117:0/1124055614 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f232c046020 con 0x7f23401028f0 2024-03-19T03:30:31.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.467+0000 7f233d600640 1 --2- 172.21.15.117:0/1124055614 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f23280a2880 0x7f23280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:31.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.468+0000 7f233d600640 1 -- 172.21.15.117:0/1124055614 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f232c0f0580 con 0x7f23401028f0 2024-03-19T03:30:31.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.468+0000 7f2345200640 1 --2- 172.21.15.117:0/1124055614 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f23280a2880 0x7f23280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:31.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.468+0000 7f2345200640 1 --2- 172.21.15.117:0/1124055614 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f23280a2880 0x7f23280a4d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f23340095f0 tx=0x7f2334005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:31.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.471+0000 7f233d600640 1 -- 172.21.15.117:0/1124055614 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f232c04d020 con 0x7f23401028f0 2024-03-19T03:30:31.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:31 smithi117 ceph-mon[31838]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:31.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:31 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:31.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:31 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:31.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:31 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:31.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:31 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:31.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:31 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2355979594' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:31.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:31 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2174710813' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:30:31.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.676+0000 7f2346600640 1 -- 172.21.15.117:0/1124055614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2340062420 con 0x7f23401028f0 2024-03-19T03:30:31.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.676+0000 7f233d600640 1 -- 172.21.15.117:0/1124055614 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f232c0dbe20 con 0x7f23401028f0 2024-03-19T03:30:31.677 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 1 2024-03-19T03:30:31.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.679+0000 7f2346600640 1 -- 172.21.15.117:0/1124055614 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f23280a2880 msgr2=0x7f23280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:31.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.679+0000 7f2346600640 1 --2- 172.21.15.117:0/1124055614 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f23280a2880 0x7f23280a4d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f23340095f0 tx=0x7f2334005f90 comp rx=0 tx=0).stop 2024-03-19T03:30:31.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.679+0000 7f2346600640 1 -- 172.21.15.117:0/1124055614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23401028f0 msgr2=0x7f2340073c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:31.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.679+0000 7f2346600640 1 --2- 172.21.15.117:0/1124055614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23401028f0 0x7f2340073c10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f232c02c170 tx=0x7f232c02c650 comp rx=0 tx=0).stop 2024-03-19T03:30:31.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.679+0000 7f2346600640 1 -- 172.21.15.117:0/1124055614 shutdown_connections 2024-03-19T03:30:31.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.679+0000 7f2346600640 1 --2- 172.21.15.117:0/1124055614 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23401032b0 0x7f2340074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:31.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.679+0000 7f2346600640 1 --2- 172.21.15.117:0/1124055614 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f23280a2880 0x7f23280a4d40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:31.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.679+0000 7f2346600640 1 --2- 172.21.15.117:0/1124055614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23401028f0 0x7f2340073c10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:31.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.679+0000 7f2346600640 1 --2- 172.21.15.117:0/1124055614 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2340101700 0x7f23400755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:31.680 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.679+0000 7f2346600640 1 -- 172.21.15.117:0/1124055614 >> 172.21.15.117:0/1124055614 conn(0x7f23400fcc90 msgr2=0x7f2340105b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:31.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.679+0000 7f2346600640 1 -- 172.21.15.117:0/1124055614 shutdown_connections 2024-03-19T03:30:31.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.679+0000 7f2346600640 1 -- 172.21.15.117:0/1124055614 wait complete. 2024-03-19T03:30:31.698 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-19T03:30:31.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:31 smithi119 ceph-mon[66861]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:31.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:31 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:31.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:31 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:31.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:31 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:31.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:31 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:31.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:31 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2355979594' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:31.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:31 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2174710813' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:30:31.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:31 smithi178 ceph-mon[67578]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:31.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:31 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:31.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:31 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:31.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:31 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:31.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:31 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:31.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:31 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2355979594' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:31.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:31 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2174710813' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:30:31.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.903+0000 7f7473c00640 1 -- 172.21.15.117:0/1023932533 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f746c107e60 msgr2=0x7f746c108280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:31.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.903+0000 7f7473c00640 1 --2- 172.21.15.117:0/1023932533 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f746c107e60 0x7f746c108280 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7460009900 tx=0x7f746002bc30 comp rx=0 tx=0).stop 2024-03-19T03:30:31.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.903+0000 7f7473c00640 1 -- 172.21.15.117:0/1023932533 shutdown_connections 2024-03-19T03:30:31.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.903+0000 7f7473c00640 1 --2- 172.21.15.117:0/1023932533 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f746c109970 0x7f746c110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:31.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.903+0000 7f7473c00640 1 --2- 172.21.15.117:0/1023932533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f746c108fb0 0x7f746c109430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:31.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.903+0000 7f7473c00640 1 --2- 172.21.15.117:0/1023932533 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f746c107e60 0x7f746c108280 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:31.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.903+0000 7f7473c00640 1 -- 172.21.15.117:0/1023932533 >> 172.21.15.117:0/1023932533 conn(0x7f746c076e30 msgr2=0x7f746c079250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:31.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.903+0000 7f7473c00640 1 -- 172.21.15.117:0/1023932533 shutdown_connections 2024-03-19T03:30:31.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.904+0000 7f7473c00640 1 -- 172.21.15.117:0/1023932533 wait complete. 2024-03-19T03:30:31.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.904+0000 7f7473c00640 1 Processor -- start 2024-03-19T03:30:31.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.905+0000 7f7473c00640 1 -- start start 2024-03-19T03:30:31.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.905+0000 7f7473c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f746c107e60 0x7f746c19e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:31.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.905+0000 7f7473c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f746c108fb0 0x7f746c19e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:31.907 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.905+0000 7f7473c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f746c109970 0x7f746c1a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:31.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.905+0000 7f7473c00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f746c113b00 con 0x7f746c109970 2024-03-19T03:30:31.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.905+0000 7f7473c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f746c113980 con 0x7f746c107e60 2024-03-19T03:30:31.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.905+0000 7f7473c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f746c113c80 con 0x7f746c108fb0 2024-03-19T03:30:31.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.905+0000 7f7471200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f746c107e60 0x7f746c19e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:31.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.905+0000 7f7471200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f746c107e60 0x7f746c19e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:59092/0 (socket says 172.21.15.117:59092) 2024-03-19T03:30:31.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.905+0000 7f7471200640 1 -- 172.21.15.117:0/3657150897 learned_addr learned my addr 172.21.15.117:0/3657150897 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:31.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.905+0000 7f746be00640 1 --2- 172.21.15.117:0/3657150897 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f746c108fb0 0x7f746c19e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:31.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.905+0000 7f7471c00640 1 --2- 172.21.15.117:0/3657150897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f746c109970 0x7f746c1a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:31.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.906+0000 7f7471200640 1 -- 172.21.15.117:0/3657150897 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f746c108fb0 msgr2=0x7f746c19e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:31.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.906+0000 7f7471200640 1 --2- 172.21.15.117:0/3657150897 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f746c108fb0 0x7f746c19e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:31.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.906+0000 7f7471200640 1 -- 172.21.15.117:0/3657150897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f746c109970 msgr2=0x7f746c1a56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:31.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.906+0000 7f7471200640 1 --2- 172.21.15.117:0/3657150897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f746c109970 0x7f746c1a56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:31.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.906+0000 7f7471200640 1 -- 172.21.15.117:0/3657150897 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746c1a5d60 con 0x7f746c107e60 2024-03-19T03:30:31.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.906+0000 7f7471c00640 1 --2- 172.21.15.117:0/3657150897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f746c109970 0x7f746c1a56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:30:31.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.906+0000 7f7471200640 1 --2- 172.21.15.117:0/3657150897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f746c107e60 0x7f746c19e140 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7460009230 tx=0x7f746002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:31.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.906+0000 7f7469600640 1 -- 172.21.15.117:0/3657150897 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7460041070 con 0x7f746c107e60 2024-03-19T03:30:31.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.906+0000 7f7469600640 1 -- 172.21.15.117:0/3657150897 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7460004450 con 0x7f746c107e60 2024-03-19T03:30:31.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.906+0000 7f7473c00640 1 -- 172.21.15.117:0/3657150897 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f746c1a5ff0 con 0x7f746c107e60 2024-03-19T03:30:31.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.906+0000 7f7469600640 1 -- 172.21.15.117:0/3657150897 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7460003e20 con 0x7f746c107e60 2024-03-19T03:30:31.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.906+0000 7f7473c00640 1 -- 172.21.15.117:0/3657150897 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f746c1a6450 con 0x7f746c107e60 2024-03-19T03:30:31.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.907+0000 7f7473c00640 1 -- 172.21.15.117:0/3657150897 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f746c10f110 con 0x7f746c107e60 2024-03-19T03:30:31.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.910+0000 7f7469600640 1 -- 172.21.15.117:0/3657150897 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f7460044070 con 0x7f746c107e60 2024-03-19T03:30:31.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.912+0000 7f7469600640 1 --2- 172.21.15.117:0/3657150897 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f74540a2880 0x7f74540a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:31.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.912+0000 7f7469600640 1 -- 172.21.15.117:0/3657150897 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f74600f0200 con 0x7f746c107e60 2024-03-19T03:30:31.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.912+0000 7f746be00640 1 --2- 172.21.15.117:0/3657150897 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f74540a2880 0x7f74540a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:31.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.913+0000 7f746be00640 1 --2- 172.21.15.117:0/3657150897 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f74540a2880 0x7f74540a4d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f746c19f650 tx=0x7f7458005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:31.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:31.920+0000 7f7469600640 1 -- 172.21.15.117:0/3657150897 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f74600bd6a0 con 0x7f746c107e60 2024-03-19T03:30:32.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.132+0000 7f7473c00640 1 -- 172.21.15.117:0/3657150897 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f746c0625a0 con 0x7f746c107e60 2024-03-19T03:30:32.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.133+0000 7f7469600640 1 -- 172.21.15.117:0/3657150897 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7f74600bcfa0 con 0x7f746c107e60 2024-03-19T03:30:32.134 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:30:32.134 INFO:teuthology.orchestra.run.smithi117.stdout:[ 2024-03-19T03:30:32.134 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:30:32.134 INFO:teuthology.orchestra.run.smithi117.stdout: "poolnum": 1, 2024-03-19T03:30:32.134 INFO:teuthology.orchestra.run.smithi117.stdout: "poolname": ".mgr" 2024-03-19T03:30:32.135 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:30:32.135 INFO:teuthology.orchestra.run.smithi117.stdout:] 2024-03-19T03:30:32.137 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.135+0000 7f7473c00640 1 -- 172.21.15.117:0/3657150897 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f74540a2880 msgr2=0x7f74540a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:32.137 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.135+0000 7f7473c00640 1 --2- 172.21.15.117:0/3657150897 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f74540a2880 0x7f74540a4d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f746c19f650 tx=0x7f7458005f90 comp rx=0 tx=0).stop 2024-03-19T03:30:32.137 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.135+0000 7f7473c00640 1 -- 172.21.15.117:0/3657150897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f746c107e60 msgr2=0x7f746c19e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:32.137 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.135+0000 7f7473c00640 1 --2- 172.21.15.117:0/3657150897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f746c107e60 0x7f746c19e140 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7460009230 tx=0x7f746002c650 comp rx=0 tx=0).stop 2024-03-19T03:30:32.137 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.136+0000 7f7473c00640 1 -- 172.21.15.117:0/3657150897 shutdown_connections 2024-03-19T03:30:32.137 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.136+0000 7f7473c00640 1 --2- 172.21.15.117:0/3657150897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f746c109970 0x7f746c1a56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:32.137 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.136+0000 7f7473c00640 1 --2- 172.21.15.117:0/3657150897 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f746c108fb0 0x7f746c19e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:32.137 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.136+0000 7f7473c00640 1 --2- 172.21.15.117:0/3657150897 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f74540a2880 0x7f74540a4d40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:32.137 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.136+0000 7f7473c00640 1 --2- 172.21.15.117:0/3657150897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f746c107e60 0x7f746c19e140 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:32.137 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.136+0000 7f7473c00640 1 -- 172.21.15.117:0/3657150897 >> 172.21.15.117:0/3657150897 conn(0x7f746c076e30 msgr2=0x7f746c10c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:32.138 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.136+0000 7f7473c00640 1 -- 172.21.15.117:0/3657150897 shutdown_connections 2024-03-19T03:30:32.138 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.136+0000 7f7473c00640 1 -- 172.21.15.117:0/3657150897 wait complete. 2024-03-19T03:30:32.155 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-19T03:30:32.155 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-19T03:30:32.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.347+0000 7f0ce0800640 1 -- 172.21.15.117:0/2107616924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cd81022a0 msgr2=0x7f0cd8104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:32.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.347+0000 7f0ce0800640 1 --2- 172.21.15.117:0/2107616924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cd81022a0 0x7f0cd8104690 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc009900 tx=0x7f0ccc02bc30 comp rx=0 tx=0).stop 2024-03-19T03:30:32.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.348+0000 7f0ce0800640 1 -- 172.21.15.117:0/2107616924 shutdown_connections 2024-03-19T03:30:32.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.348+0000 7f0ce0800640 1 --2- 172.21.15.117:0/2107616924 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0cd8107500 0x7f0cd8109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:32.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.348+0000 7f0ce0800640 1 --2- 172.21.15.117:0/2107616924 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0cd8104bd0 0x7f0cd8106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:32.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.348+0000 7f0ce0800640 1 --2- 172.21.15.117:0/2107616924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cd81022a0 0x7f0cd8104690 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:32.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.348+0000 7f0ce0800640 1 -- 172.21.15.117:0/2107616924 >> 172.21.15.117:0/2107616924 conn(0x7f0cd80fbcb0 msgr2=0x7f0cd80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:32.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.348+0000 7f0ce0800640 1 -- 172.21.15.117:0/2107616924 shutdown_connections 2024-03-19T03:30:32.350 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.348+0000 7f0ce0800640 1 -- 172.21.15.117:0/2107616924 wait complete. 2024-03-19T03:30:32.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.348+0000 7f0ce0800640 1 Processor -- start 2024-03-19T03:30:32.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.349+0000 7f0ce0800640 1 -- start start 2024-03-19T03:30:32.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.349+0000 7f0ce0800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0cd81022a0 0x7f0cd8197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:32.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.349+0000 7f0ce0800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0cd8104bd0 0x7f0cd81980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:32.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.349+0000 7f0ce0800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cd8107500 0x7f0cd819f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:32.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.349+0000 7f0ce0800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd810d510 con 0x7f0cd8107500 2024-03-19T03:30:32.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.349+0000 7f0ce0800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd810d390 con 0x7f0cd81022a0 2024-03-19T03:30:32.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.349+0000 7f0ce0800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd810d690 con 0x7f0cd8104bd0 2024-03-19T03:30:32.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.349+0000 7f0cdde00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0cd81022a0 0x7f0cd8197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:32.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.349+0000 7f0cdd400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0cd8104bd0 0x7f0cd81980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:32.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.349+0000 7f0cdde00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0cd81022a0 0x7f0cd8197bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:59108/0 (socket says 172.21.15.117:59108) 2024-03-19T03:30:32.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.349+0000 7f0cdde00640 1 -- 172.21.15.117:0/2983054344 learned_addr learned my addr 172.21.15.117:0/2983054344 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:32.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.349+0000 7f0cde800640 1 --2- 172.21.15.117:0/2983054344 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cd8107500 0x7f0cd819f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:32.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.350+0000 7f0cdde00640 1 -- 172.21.15.117:0/2983054344 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0cd8104bd0 msgr2=0x7f0cd81980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:32.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.350+0000 7f0cdde00640 1 --2- 172.21.15.117:0/2983054344 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0cd8104bd0 0x7f0cd81980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:32.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.350+0000 7f0cdde00640 1 -- 172.21.15.117:0/2983054344 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cd8107500 msgr2=0x7f0cd819f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:32.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.350+0000 7f0cdde00640 1 --2- 172.21.15.117:0/2983054344 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cd8107500 0x7f0cd819f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:32.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.350+0000 7f0cdde00640 1 -- 172.21.15.117:0/2983054344 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cd819f7d0 con 0x7f0cd81022a0 2024-03-19T03:30:32.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.350+0000 7f0cdd400640 1 --2- 172.21.15.117:0/2983054344 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0cd8104bd0 0x7f0cd81980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:30:32.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.350+0000 7f0cdde00640 1 --2- 172.21.15.117:0/2983054344 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0cd81022a0 0x7f0cd8197bb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc000c00 tx=0x7f0ccc02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:32.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.350+0000 7f0cd2a00640 1 -- 172.21.15.117:0/2983054344 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc043020 con 0x7f0cd81022a0 2024-03-19T03:30:32.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.350+0000 7f0cd2a00640 1 -- 172.21.15.117:0/2983054344 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ccc0054e0 con 0x7f0cd81022a0 2024-03-19T03:30:32.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.350+0000 7f0ce0800640 1 -- 172.21.15.117:0/2983054344 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cd819fa60 con 0x7f0cd81022a0 2024-03-19T03:30:32.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.350+0000 7f0cd2a00640 1 -- 172.21.15.117:0/2983054344 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc003c40 con 0x7f0cd81022a0 2024-03-19T03:30:32.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.350+0000 7f0ce0800640 1 -- 172.21.15.117:0/2983054344 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cd819ffb0 con 0x7f0cd81022a0 2024-03-19T03:30:32.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.351+0000 7f0ce0800640 1 -- 172.21.15.117:0/2983054344 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cd80fd7a0 con 0x7f0cd81022a0 2024-03-19T03:30:32.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.353+0000 7f0cd2a00640 1 -- 172.21.15.117:0/2983054344 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f0ccc046020 con 0x7f0cd81022a0 2024-03-19T03:30:32.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.355+0000 7f0cd2a00640 1 --2- 172.21.15.117:0/2983054344 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0cc00a2800 0x7f0cc00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:32.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.355+0000 7f0cd2a00640 1 -- 172.21.15.117:0/2983054344 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f0ccc0f0a60 con 0x7f0cd81022a0 2024-03-19T03:30:32.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.355+0000 7f0cdd400640 1 --2- 172.21.15.117:0/2983054344 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0cc00a2800 0x7f0cc00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:32.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.355+0000 7f0cdd400640 1 --2- 172.21.15.117:0/2983054344 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0cc00a2800 0x7f0cc00a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80095f0 tx=0x7f0cc8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:32.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.357+0000 7f0cd2a00640 1 -- 172.21.15.117:0/2983054344 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0ccc04d020 con 0x7f0cd81022a0 2024-03-19T03:30:32.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:32.575+0000 7f0ce0800640 1 -- 172.21.15.117:0/2983054344 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f0cd802d1f0 con 0x7f0cd81022a0 2024-03-19T03:30:32.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:32 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:32.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:32 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:32.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:32 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:32.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:32 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:30:32.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:32 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:32.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:32 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1124055614' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:30:32.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:32 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:30:32.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:32 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3657150897' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-19T03:30:32.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:32 smithi117 ceph-mon[31838]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:32.838 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:32 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:32.838 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:32 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:32.838 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:32 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:32.838 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:32 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:30:32.838 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:32 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:32.838 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:32 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1124055614' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:30:32.838 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:32 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:30:32.838 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:32 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3657150897' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-19T03:30:32.839 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:32 smithi119 ceph-mon[66861]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:32.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:32.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:32.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:32.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:30:32.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:32.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:32 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1124055614' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:30:32.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:30:32.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:32 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3657150897' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-19T03:30:32.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:32 smithi178 ceph-mon[67578]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:33.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.575+0000 7f0cd2a00640 1 -- 172.21.15.117:0/2983054344 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7f0ccc0bdf00 con 0x7f0cd81022a0 2024-03-19T03:30:33.576 INFO:teuthology.orchestra.run.smithi117.stderr:pool 'cephfs_metadata' created 2024-03-19T03:30:33.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.582+0000 7f0ce0800640 1 -- 172.21.15.117:0/2983054344 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0cc00a2800 msgr2=0x7f0cc00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:33.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.582+0000 7f0ce0800640 1 --2- 172.21.15.117:0/2983054344 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0cc00a2800 0x7f0cc00a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80095f0 tx=0x7f0cc8005f90 comp rx=0 tx=0).stop 2024-03-19T03:30:33.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.582+0000 7f0ce0800640 1 -- 172.21.15.117:0/2983054344 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0cd81022a0 msgr2=0x7f0cd8197bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:33.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.582+0000 7f0ce0800640 1 --2- 172.21.15.117:0/2983054344 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0cd81022a0 0x7f0cd8197bb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc000c00 tx=0x7f0ccc02c650 comp rx=0 tx=0).stop 2024-03-19T03:30:33.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.583+0000 7f0ce0800640 1 -- 172.21.15.117:0/2983054344 shutdown_connections 2024-03-19T03:30:33.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.583+0000 7f0ce0800640 1 --2- 172.21.15.117:0/2983054344 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cd8107500 0x7f0cd819f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:33.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.583+0000 7f0ce0800640 1 --2- 172.21.15.117:0/2983054344 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0cd8104bd0 0x7f0cd81980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:33.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.583+0000 7f0ce0800640 1 --2- 172.21.15.117:0/2983054344 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0cc00a2800 0x7f0cc00a4cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:33.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.583+0000 7f0ce0800640 1 --2- 172.21.15.117:0/2983054344 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0cd81022a0 0x7f0cd8197bb0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:33.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.583+0000 7f0ce0800640 1 -- 172.21.15.117:0/2983054344 >> 172.21.15.117:0/2983054344 conn(0x7f0cd80fbcb0 msgr2=0x7f0cd8102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:33.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.583+0000 7f0ce0800640 1 -- 172.21.15.117:0/2983054344 shutdown_connections 2024-03-19T03:30:33.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.583+0000 7f0ce0800640 1 -- 172.21.15.117:0/2983054344 wait complete. 2024-03-19T03:30:33.603 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-19T03:30:33.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.798+0000 7fe6d1200640 1 -- 172.21.15.117:0/617400152 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6cc103a60 msgr2=0x7fe6cc103e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:33.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.798+0000 7fe6d1200640 1 --2- 172.21.15.117:0/617400152 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6cc103a60 0x7fe6cc103e80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc009860 tx=0x7fe6bc02bb70 comp rx=0 tx=0).stop 2024-03-19T03:30:33.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.798+0000 7fe6d1200640 1 -- 172.21.15.117:0/617400152 shutdown_connections 2024-03-19T03:30:33.799 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.798+0000 7fe6d1200640 1 --2- 172.21.15.117:0/617400152 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6cc105610 0x7fe6cc10bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:33.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.798+0000 7fe6d1200640 1 --2- 172.21.15.117:0/617400152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6cc104c50 0x7fe6cc1050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:33.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.798+0000 7fe6d1200640 1 --2- 172.21.15.117:0/617400152 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6cc103a60 0x7fe6cc103e80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:33.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.798+0000 7fe6d1200640 1 -- 172.21.15.117:0/617400152 >> 172.21.15.117:0/617400152 conn(0x7fe6cc0fefd0 msgr2=0x7fe6cc101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:33.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.798+0000 7fe6d1200640 1 -- 172.21.15.117:0/617400152 shutdown_connections 2024-03-19T03:30:33.800 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.798+0000 7fe6d1200640 1 -- 172.21.15.117:0/617400152 wait complete. 2024-03-19T03:30:33.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.799+0000 7fe6d1200640 1 Processor -- start 2024-03-19T03:30:33.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.799+0000 7fe6d1200640 1 -- start start 2024-03-19T03:30:33.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.799+0000 7fe6d1200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6cc103a60 0x7fe6cc199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:33.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.799+0000 7fe6d1200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6cc104c50 0x7fe6cc19a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:33.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.800+0000 7fe6d1200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6cc105610 0x7fe6cc1a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:33.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.800+0000 7fe6d1200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe6cc10f6e0 con 0x7fe6cc103a60 2024-03-19T03:30:33.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.800+0000 7fe6d1200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe6cc10f560 con 0x7fe6cc105610 2024-03-19T03:30:33.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.800+0000 7fe6d1200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fe6cc10f860 con 0x7fe6cc104c50 2024-03-19T03:30:33.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.800+0000 7fe6c9e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6cc103a60 0x7fe6cc199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:33.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.800+0000 7fe6c9400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6cc104c50 0x7fe6cc19a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:33.802 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.800+0000 7fe6c9e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6cc103a60 0x7fe6cc199d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:54440/0 (socket says 172.21.15.117:54440) 2024-03-19T03:30:33.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.800+0000 7fe6c9e00640 1 -- 172.21.15.117:0/3142979648 learned_addr learned my addr 172.21.15.117:0/3142979648 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:33.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.800+0000 7fe6ca800640 1 --2- 172.21.15.117:0/3142979648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6cc105610 0x7fe6cc1a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:33.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.800+0000 7fe6c9400640 1 -- 172.21.15.117:0/3142979648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6cc105610 msgr2=0x7fe6cc1a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:33.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.800+0000 7fe6c9400640 1 --2- 172.21.15.117:0/3142979648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6cc105610 0x7fe6cc1a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:33.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.800+0000 7fe6c9400640 1 -- 172.21.15.117:0/3142979648 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6cc103a60 msgr2=0x7fe6cc199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:33.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.800+0000 7fe6c9400640 1 --2- 172.21.15.117:0/3142979648 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6cc103a60 0x7fe6cc199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:33.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.800+0000 7fe6c9400640 1 -- 172.21.15.117:0/3142979648 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6cc1a1980 con 0x7fe6cc104c50 2024-03-19T03:30:33.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.800+0000 7fe6c9400640 1 --2- 172.21.15.117:0/3142979648 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6cc104c50 0x7fe6cc19a2a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe6b800e790 tx=0x7fe6b800ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:33.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.801+0000 7fe6c2a00640 1 -- 172.21.15.117:0/3142979648 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b800eea0 con 0x7fe6cc104c50 2024-03-19T03:30:33.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.801+0000 7fe6c2a00640 1 -- 172.21.15.117:0/3142979648 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6b800ccc0 con 0x7fe6cc104c50 2024-03-19T03:30:33.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.801+0000 7fe6d1200640 1 -- 172.21.15.117:0/3142979648 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6cc1a1c70 con 0x7fe6cc104c50 2024-03-19T03:30:33.803 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.801+0000 7fe6c2a00640 1 -- 172.21.15.117:0/3142979648 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8018780 con 0x7fe6cc104c50 2024-03-19T03:30:33.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.801+0000 7fe6d1200640 1 -- 172.21.15.117:0/3142979648 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6cc1a21c0 con 0x7fe6cc104c50 2024-03-19T03:30:33.804 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.802+0000 7fe6d1200640 1 -- 172.21.15.117:0/3142979648 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6cc10ae50 con 0x7fe6cc104c50 2024-03-19T03:30:33.805 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.804+0000 7fe6c2a00640 1 -- 172.21.15.117:0/3142979648 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fe6b8010430 con 0x7fe6cc104c50 2024-03-19T03:30:33.806 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.806+0000 7fe6c2a00640 1 --2- 172.21.15.117:0/3142979648 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe6b00a2880 0x7fe6b00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:33.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.806+0000 7fe6c2a00640 1 -- 172.21.15.117:0/3142979648 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7fe6b80ce8b0 con 0x7fe6cc104c50 2024-03-19T03:30:33.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.806+0000 7fe6c9e00640 1 --2- 172.21.15.117:0/3142979648 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe6b00a2880 0x7fe6b00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:33.807 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.807+0000 7fe6c9e00640 1 --2- 172.21.15.117:0/3142979648 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe6b00a2880 0x7fe6b00a4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc0095f0 tx=0x7fe6bc034ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:33.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:33.809+0000 7fe6c2a00640 1 -- 172.21.15.117:0/3142979648 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe6b809b590 con 0x7fe6cc104c50 2024-03-19T03:30:33.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:33 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2983054344' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-19T03:30:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:33 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-19T03:30:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:33 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:33.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:33 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2983054344' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-19T03:30:33.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:33 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-19T03:30:33.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:33.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:33.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:33.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:33.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:33 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:34.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.026+0000 7fe6d1200640 1 -- 172.21.15.117:0/3142979648 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fe6cc062420 con 0x7fe6cc104c50 2024-03-19T03:30:34.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:33 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2983054344' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-19T03:30:34.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:33 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-19T03:30:34.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:34.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:34.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:34.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:34.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:33 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:34.610 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.608+0000 7fe6c2a00640 1 -- 172.21.15.117:0/3142979648 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7fe6b809f630 con 0x7fe6cc104c50 2024-03-19T03:30:34.610 INFO:teuthology.orchestra.run.smithi117.stderr:pool 'cephfs_data' created 2024-03-19T03:30:34.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.613+0000 7fe6d1200640 1 -- 172.21.15.117:0/3142979648 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe6b00a2880 msgr2=0x7fe6b00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:34.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.613+0000 7fe6d1200640 1 --2- 172.21.15.117:0/3142979648 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe6b00a2880 0x7fe6b00a4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc0095f0 tx=0x7fe6bc034ad0 comp rx=0 tx=0).stop 2024-03-19T03:30:34.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.614+0000 7fe6d1200640 1 -- 172.21.15.117:0/3142979648 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6cc104c50 msgr2=0x7fe6cc19a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:34.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.614+0000 7fe6d1200640 1 --2- 172.21.15.117:0/3142979648 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6cc104c50 0x7fe6cc19a2a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe6b800e790 tx=0x7fe6b800ec50 comp rx=0 tx=0).stop 2024-03-19T03:30:34.615 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.614+0000 7fe6d1200640 1 -- 172.21.15.117:0/3142979648 shutdown_connections 2024-03-19T03:30:34.615 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.614+0000 7fe6d1200640 1 --2- 172.21.15.117:0/3142979648 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe6b00a2880 0x7fe6b00a4d40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:34.615 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.614+0000 7fe6d1200640 1 --2- 172.21.15.117:0/3142979648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6cc105610 0x7fe6cc1a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:34.615 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.614+0000 7fe6d1200640 1 --2- 172.21.15.117:0/3142979648 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6cc104c50 0x7fe6cc19a2a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:34.615 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.614+0000 7fe6d1200640 1 --2- 172.21.15.117:0/3142979648 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6cc103a60 0x7fe6cc199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:34.616 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.614+0000 7fe6d1200640 1 -- 172.21.15.117:0/3142979648 >> 172.21.15.117:0/3142979648 conn(0x7fe6cc0fefd0 msgr2=0x7fe6cc108020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:34.616 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.615+0000 7fe6d1200640 1 -- 172.21.15.117:0/3142979648 shutdown_connections 2024-03-19T03:30:34.616 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.616+0000 7fe6d1200640 1 -- 172.21.15.117:0/3142979648 wait complete. 2024-03-19T03:30:34.646 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-19T03:30:34.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.828+0000 7f4322600640 1 -- 172.21.15.117:0/2207937301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f431c1033e0 msgr2=0x7f431c109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:34.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.828+0000 7f4322600640 1 --2- 172.21.15.117:0/2207937301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f431c1033e0 0x7f431c109c70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4304009860 tx=0x7f430402bb70 comp rx=0 tx=0).stop 2024-03-19T03:30:34.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.828+0000 7f4322600640 1 -- 172.21.15.117:0/2207937301 shutdown_connections 2024-03-19T03:30:34.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.828+0000 7f4322600640 1 --2- 172.21.15.117:0/2207937301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f431c1033e0 0x7f431c109c70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:34.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.828+0000 7f4322600640 1 --2- 172.21.15.117:0/2207937301 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f431c102a20 0x7f431c102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:34.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.828+0000 7f4322600640 1 --2- 172.21.15.117:0/2207937301 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f431c101830 0x7f431c101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:34.829 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.828+0000 7f4322600640 1 -- 172.21.15.117:0/2207937301 >> 172.21.15.117:0/2207937301 conn(0x7f431c0fcdc0 msgr2=0x7f431c0ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:34.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.828+0000 7f4322600640 1 -- 172.21.15.117:0/2207937301 shutdown_connections 2024-03-19T03:30:34.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.828+0000 7f4322600640 1 -- 172.21.15.117:0/2207937301 wait complete. 2024-03-19T03:30:34.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.829+0000 7f4322600640 1 Processor -- start 2024-03-19T03:30:34.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.829+0000 7f4322600640 1 -- start start 2024-03-19T03:30:34.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.829+0000 7f4322600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f431c101830 0x7f431c199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:34.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.829+0000 7f4322600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f431c102a20 0x7f431c19a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:34.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.829+0000 7f4322600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f431c1033e0 0x7f431c1a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:34.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.829+0000 7f4322600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f431c10d4b0 con 0x7f431c101830 2024-03-19T03:30:34.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.829+0000 7f4322600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f431c10d330 con 0x7f431c1033e0 2024-03-19T03:30:34.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.829+0000 7f4322600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f431c10d630 con 0x7f431c102a20 2024-03-19T03:30:34.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.829+0000 7f431aa00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f431c102a20 0x7f431c19a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:34.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.829+0000 7f431b400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f431c101830 0x7f431c199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:34.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.829+0000 7f431aa00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f431c102a20 0x7f431c19a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:35100/0 (socket says 172.21.15.117:35100) 2024-03-19T03:30:34.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.829+0000 7f431aa00640 1 -- 172.21.15.117:0/3047923131 learned_addr learned my addr 172.21.15.117:0/3047923131 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:34.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.830+0000 7f431be00640 1 --2- 172.21.15.117:0/3047923131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f431c1033e0 0x7f431c1a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:34.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.830+0000 7f431aa00640 1 -- 172.21.15.117:0/3047923131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f431c1033e0 msgr2=0x7f431c1a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:34.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.830+0000 7f431aa00640 1 --2- 172.21.15.117:0/3047923131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f431c1033e0 0x7f431c1a1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:34.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.830+0000 7f431aa00640 1 -- 172.21.15.117:0/3047923131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f431c101830 msgr2=0x7f431c199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:34.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.830+0000 7f431aa00640 1 --2- 172.21.15.117:0/3047923131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f431c101830 0x7f431c199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:34.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.830+0000 7f431aa00640 1 -- 172.21.15.117:0/3047923131 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f431c1a1910 con 0x7f431c102a20 2024-03-19T03:30:34.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.830+0000 7f431b400640 1 --2- 172.21.15.117:0/3047923131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f431c101830 0x7f431c199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:30:34.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.830+0000 7f431be00640 1 --2- 172.21.15.117:0/3047923131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f431c1033e0 0x7f431c1a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:30:34.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.830+0000 7f431aa00640 1 --2- 172.21.15.117:0/3047923131 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f431c102a20 0x7f431c19a240 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f430c00e9c0 tx=0x7f430c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:34.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.830+0000 7f4313e00640 1 -- 172.21.15.117:0/3047923131 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430c00cde0 con 0x7f431c102a20 2024-03-19T03:30:34.834 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.830+0000 7f4313e00640 1 -- 172.21.15.117:0/3047923131 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f430c0035f0 con 0x7f431c102a20 2024-03-19T03:30:34.834 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.830+0000 7f4322600640 1 -- 172.21.15.117:0/3047923131 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f431c1a1c00 con 0x7f431c102a20 2024-03-19T03:30:34.834 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.830+0000 7f4313e00640 1 -- 172.21.15.117:0/3047923131 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430c010690 con 0x7f431c102a20 2024-03-19T03:30:34.834 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.831+0000 7f4322600640 1 -- 172.21.15.117:0/3047923131 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f431c1a2150 con 0x7f431c102a20 2024-03-19T03:30:34.834 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.831+0000 7f4322600640 1 -- 172.21.15.117:0/3047923131 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f431c1a2380 con 0x7f431c102a20 2024-03-19T03:30:34.834 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.833+0000 7f4313e00640 1 -- 172.21.15.117:0/3047923131 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f430c0037a0 con 0x7f431c102a20 2024-03-19T03:30:34.835 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.834+0000 7f4313e00640 1 --2- 172.21.15.117:0/3047923131 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f43000a2800 0x7f43000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:34.835 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.835+0000 7f4313e00640 1 -- 172.21.15.117:0/3047923131 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7f430c0cec40 con 0x7f431c102a20 2024-03-19T03:30:34.835 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.835+0000 7f431b400640 1 --2- 172.21.15.117:0/3047923131 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f43000a2800 0x7f43000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:34.836 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.835+0000 7f431b400640 1 --2- 172.21.15.117:0/3047923131 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f43000a2800 0x7f43000a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f431c19b1d0 tx=0x7f4314009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:34.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:34.839+0000 7f4313e00640 1 -- 172.21.15.117:0/3047923131 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f430c09be90 con 0x7f431c102a20 2024-03-19T03:30:34.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:34 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-19T03:30:34.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:34 smithi119 ceph-mon[66861]: osdmap e82: 12 total, 12 up, 12 in 2024-03-19T03:30:35.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:34 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-19T03:30:35.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:34 smithi117 ceph-mon[31838]: osdmap e82: 12 total, 12 up, 12 in 2024-03-19T03:30:35.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.077+0000 7f4322600640 1 -- 172.21.15.117:0/3047923131 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f431c1a25f0 con 0x7f431c102a20 2024-03-19T03:30:35.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:34 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-19T03:30:35.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:34 smithi178 ceph-mon[67578]: osdmap e82: 12 total, 12 up, 12 in 2024-03-19T03:30:35.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.668+0000 7f4313e00640 1 -- 172.21.15.117:0/3047923131 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f430c086110 con 0x7f431c102a20 2024-03-19T03:30:35.670 INFO:teuthology.orchestra.run.smithi117.stderr:new fs with metadata pool 2 and data pool 3 2024-03-19T03:30:35.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.674+0000 7f4322600640 1 -- 172.21.15.117:0/3047923131 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f43000a2800 msgr2=0x7f43000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:35.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.674+0000 7f4322600640 1 --2- 172.21.15.117:0/3047923131 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f43000a2800 0x7f43000a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f431c19b1d0 tx=0x7f4314009500 comp rx=0 tx=0).stop 2024-03-19T03:30:35.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.674+0000 7f4322600640 1 -- 172.21.15.117:0/3047923131 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f431c102a20 msgr2=0x7f431c19a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:35.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.674+0000 7f4322600640 1 --2- 172.21.15.117:0/3047923131 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f431c102a20 0x7f431c19a240 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f430c00e9c0 tx=0x7f430c00ee80 comp rx=0 tx=0).stop 2024-03-19T03:30:35.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.675+0000 7f4322600640 1 -- 172.21.15.117:0/3047923131 shutdown_connections 2024-03-19T03:30:35.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.675+0000 7f4322600640 1 --2- 172.21.15.117:0/3047923131 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f43000a2800 0x7f43000a4cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:35.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.675+0000 7f4322600640 1 --2- 172.21.15.117:0/3047923131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f431c1033e0 0x7f431c1a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:35.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.675+0000 7f4322600640 1 --2- 172.21.15.117:0/3047923131 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f431c102a20 0x7f431c19a240 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:35.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.675+0000 7f4322600640 1 --2- 172.21.15.117:0/3047923131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f431c101830 0x7f431c199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:35.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.675+0000 7f4322600640 1 -- 172.21.15.117:0/3047923131 >> 172.21.15.117:0/3047923131 conn(0x7f431c0fcdc0 msgr2=0x7f431c073a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:35.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.676+0000 7f4322600640 1 -- 172.21.15.117:0/3047923131 shutdown_connections 2024-03-19T03:30:35.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.676+0000 7f4322600640 1 -- 172.21.15.117:0/3047923131 wait complete. 2024-03-19T03:30:35.702 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-19T03:30:35.702 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-19T03:30:35.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.887+0000 7fa441000640 1 -- 172.21.15.117:0/2759145076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa43c077dc0 msgr2=0x7fa43c078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:35.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.887+0000 7fa441000640 1 --2- 172.21.15.117:0/2759145076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa43c077dc0 0x7fa43c078660 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa434009ca0 tx=0x7fa43402c0f0 comp rx=0 tx=0).stop 2024-03-19T03:30:35.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.887+0000 7fa441000640 1 -- 172.21.15.117:0/2759145076 shutdown_connections 2024-03-19T03:30:35.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.887+0000 7fa441000640 1 --2- 172.21.15.117:0/2759145076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa43c077dc0 0x7fa43c078660 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:35.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.887+0000 7fa441000640 1 --2- 172.21.15.117:0/2759145076 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa43c077420 0x7fa43c077880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:35.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.887+0000 7fa441000640 1 --2- 172.21.15.117:0/2759145076 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa43c0761e0 0x7fa43c076600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:35.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.887+0000 7fa441000640 1 -- 172.21.15.117:0/2759145076 >> 172.21.15.117:0/2759145076 conn(0x7fa43c0fefe0 msgr2=0x7fa43c101400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:35.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.887+0000 7fa441000640 1 -- 172.21.15.117:0/2759145076 shutdown_connections 2024-03-19T03:30:35.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.887+0000 7fa441000640 1 -- 172.21.15.117:0/2759145076 wait complete. 2024-03-19T03:30:35.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.888+0000 7fa441000640 1 Processor -- start 2024-03-19T03:30:35.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.888+0000 7fa441000640 1 -- start start 2024-03-19T03:30:35.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.888+0000 7fa441000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa43c0761e0 0x7fa43c19e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:35.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.889+0000 7fa441000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa43c077420 0x7fa43c19e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:35.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.889+0000 7fa441000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa43c077dc0 0x7fa43c1a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:35.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.889+0000 7fa441000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa43c113ae0 con 0x7fa43c077dc0 2024-03-19T03:30:35.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.889+0000 7fa441000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa43c113960 con 0x7fa43c077420 2024-03-19T03:30:35.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.889+0000 7fa441000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fa43c113c60 con 0x7fa43c0761e0 2024-03-19T03:30:35.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.889+0000 7fa43a800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa43c077dc0 0x7fa43c1a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:35.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.889+0000 7fa43a800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa43c077dc0 0x7fa43c1a5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39644/0 (socket says 172.21.15.117:39644) 2024-03-19T03:30:35.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.889+0000 7fa43a800640 1 -- 172.21.15.117:0/3484525870 learned_addr learned my addr 172.21.15.117:0/3484525870 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:35.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.889+0000 7fa439e00640 1 --2- 172.21.15.117:0/3484525870 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa43c0761e0 0x7fa43c19e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:35.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.889+0000 7fa439400640 1 --2- 172.21.15.117:0/3484525870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa43c077420 0x7fa43c19e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:35.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.889+0000 7fa43a800640 1 -- 172.21.15.117:0/3484525870 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa43c0761e0 msgr2=0x7fa43c19e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:35.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.889+0000 7fa43a800640 1 --2- 172.21.15.117:0/3484525870 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa43c0761e0 0x7fa43c19e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:35.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.889+0000 7fa43a800640 1 -- 172.21.15.117:0/3484525870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa43c077420 msgr2=0x7fa43c19e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:35.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.889+0000 7fa43a800640 1 --2- 172.21.15.117:0/3484525870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa43c077420 0x7fa43c19e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:35.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.889+0000 7fa43a800640 1 -- 172.21.15.117:0/3484525870 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa43c1a5cc0 con 0x7fa43c077dc0 2024-03-19T03:30:35.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.890+0000 7fa43a800640 1 --2- 172.21.15.117:0/3484525870 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa43c077dc0 0x7fa43c1a5650 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa43402c630 tx=0x7fa434032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:35.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.890+0000 7fa432a00640 1 -- 172.21.15.117:0/3484525870 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa434041020 con 0x7fa43c077dc0 2024-03-19T03:30:35.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.890+0000 7fa432a00640 1 -- 172.21.15.117:0/3484525870 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa434034cd0 con 0x7fa43c077dc0 2024-03-19T03:30:35.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.890+0000 7fa441000640 1 -- 172.21.15.117:0/3484525870 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa43c1a5f50 con 0x7fa43c077dc0 2024-03-19T03:30:35.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.890+0000 7fa432a00640 1 -- 172.21.15.117:0/3484525870 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa434039040 con 0x7fa43c077dc0 2024-03-19T03:30:35.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.890+0000 7fa441000640 1 -- 172.21.15.117:0/3484525870 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa43c1a6410 con 0x7fa43c077dc0 2024-03-19T03:30:35.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.891+0000 7fa441000640 1 -- 172.21.15.117:0/3484525870 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa43c1104d0 con 0x7fa43c077dc0 2024-03-19T03:30:35.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.893+0000 7fa432a00640 1 -- 172.21.15.117:0/3484525870 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fa434044020 con 0x7fa43c077dc0 2024-03-19T03:30:35.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.895+0000 7fa432a00640 1 --2- 172.21.15.117:0/3484525870 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa4200a2880 0x7fa4200a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:35.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.895+0000 7fa432a00640 1 -- 172.21.15.117:0/3484525870 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7fa4340f06b0 con 0x7fa43c077dc0 2024-03-19T03:30:35.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.895+0000 7fa439e00640 1 --2- 172.21.15.117:0/3484525870 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa4200a2880 0x7fa4200a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:35.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.896+0000 7fa439e00640 1 --2- 172.21.15.117:0/3484525870 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa4200a2880 0x7fa4200a4d40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa42c009bd0 tx=0x7fa42c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:35.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:35.900+0000 7fa432a00640 1 -- 172.21.15.117:0/3484525870 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa43404b020 con 0x7fa43c077dc0 2024-03-19T03:30:35.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:35 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3142979648' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-19T03:30:35.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:35 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-19T03:30:35.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:35 smithi119 ceph-mon[66861]: pgmap v310: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:35.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:35 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-19T03:30:35.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:35 smithi119 ceph-mon[66861]: osdmap e83: 12 total, 12 up, 12 in 2024-03-19T03:30:35.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:35 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:35.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:35 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:35.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:35 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:35.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:35 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:30:35.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:35 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:35.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:35 smithi119 ceph-mon[66861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T03:30:35.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:35 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3047923131' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-19T03:30:35.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:35 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-19T03:30:35.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:35 smithi119 ceph-mon[66861]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-19T03:30:36.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:35 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3142979648' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-19T03:30:36.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:35 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-19T03:30:36.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:35 smithi117 ceph-mon[31838]: pgmap v310: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:36.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:35 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-19T03:30:36.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:35 smithi117 ceph-mon[31838]: osdmap e83: 12 total, 12 up, 12 in 2024-03-19T03:30:36.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:35 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:36.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:35 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:36.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:35 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:36.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:35 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:30:36.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:35 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:36.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:35 smithi117 ceph-mon[31838]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T03:30:36.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:35 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3047923131' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-19T03:30:36.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:35 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-19T03:30:36.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:35 smithi117 ceph-mon[31838]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-19T03:30:36.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:35 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3142979648' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-19T03:30:36.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:35 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-19T03:30:36.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:35 smithi178 ceph-mon[67578]: pgmap v310: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:30:36.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:35 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-19T03:30:36.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:35 smithi178 ceph-mon[67578]: osdmap e83: 12 total, 12 up, 12 in 2024-03-19T03:30:36.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:35 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:36.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:35 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:36.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:35 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:30:36.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:35 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:30:36.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:35 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:30:36.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:35 smithi178 ceph-mon[67578]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T03:30:36.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:35 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3047923131' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-19T03:30:36.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:35 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-19T03:30:36.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:35 smithi178 ceph-mon[67578]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-19T03:30:36.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.108+0000 7fa441000640 1 -- 172.21.15.117:0/3484525870 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7fa43c02d3f0 con 0x7fa43c077dc0 2024-03-19T03:30:36.628 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.627+0000 7fa432a00640 1 -- 172.21.15.117:0/3484525870 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) ==== 157+0+0 (secure 0 0 0) 0x7fa4340bd930 con 0x7fa43c077dc0 2024-03-19T03:30:36.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.630+0000 7fa441000640 1 -- 172.21.15.117:0/3484525870 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa4200a2880 msgr2=0x7fa4200a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:36.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.630+0000 7fa441000640 1 --2- 172.21.15.117:0/3484525870 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa4200a2880 0x7fa4200a4d40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa42c009bd0 tx=0x7fa42c005f90 comp rx=0 tx=0).stop 2024-03-19T03:30:36.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.630+0000 7fa441000640 1 -- 172.21.15.117:0/3484525870 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa43c077dc0 msgr2=0x7fa43c1a5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:36.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.630+0000 7fa441000640 1 --2- 172.21.15.117:0/3484525870 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa43c077dc0 0x7fa43c1a5650 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa43402c630 tx=0x7fa434032ea0 comp rx=0 tx=0).stop 2024-03-19T03:30:36.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.631+0000 7fa441000640 1 -- 172.21.15.117:0/3484525870 shutdown_connections 2024-03-19T03:30:36.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.631+0000 7fa441000640 1 --2- 172.21.15.117:0/3484525870 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa43c077dc0 0x7fa43c1a5650 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:36.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.631+0000 7fa441000640 1 --2- 172.21.15.117:0/3484525870 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa4200a2880 0x7fa4200a4d40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:36.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.631+0000 7fa441000640 1 --2- 172.21.15.117:0/3484525870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa43c077420 0x7fa43c19e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:36.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.631+0000 7fa441000640 1 --2- 172.21.15.117:0/3484525870 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa43c0761e0 0x7fa43c19e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:36.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.631+0000 7fa441000640 1 -- 172.21.15.117:0/3484525870 >> 172.21.15.117:0/3484525870 conn(0x7fa43c0fefe0 msgr2=0x7fa43c100620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:36.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.631+0000 7fa441000640 1 -- 172.21.15.117:0/3484525870 shutdown_connections 2024-03-19T03:30:36.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.631+0000 7fa441000640 1 -- 172.21.15.117:0/3484525870 wait complete. 2024-03-19T03:30:36.654 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-19T03:30:36.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.865+0000 7fd1d3e00640 1 -- 172.21.15.117:0/1524086689 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1d4104a20 msgr2=0x7fd1d4106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:36.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.865+0000 7fd1d3e00640 1 --2- 172.21.15.117:0/1524086689 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1d4104a20 0x7fd1d4106e10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0009900 tx=0x7fd1c002bc30 comp rx=0 tx=0).stop 2024-03-19T03:30:36.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.866+0000 7fd1d3e00640 1 -- 172.21.15.117:0/1524086689 shutdown_connections 2024-03-19T03:30:36.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.866+0000 7fd1d3e00640 1 --2- 172.21.15.117:0/1524086689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1d4107350 0x7fd1d41097a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:36.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.866+0000 7fd1d3e00640 1 --2- 172.21.15.117:0/1524086689 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1d4104a20 0x7fd1d4106e10 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:36.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.866+0000 7fd1d3e00640 1 --2- 172.21.15.117:0/1524086689 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1d41020f0 0x7fd1d41044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:36.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.866+0000 7fd1d3e00640 1 -- 172.21.15.117:0/1524086689 >> 172.21.15.117:0/1524086689 conn(0x7fd1d40fbb00 msgr2=0x7fd1d40fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:36.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.866+0000 7fd1d3e00640 1 -- 172.21.15.117:0/1524086689 shutdown_connections 2024-03-19T03:30:36.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.866+0000 7fd1d3e00640 1 -- 172.21.15.117:0/1524086689 wait complete. 2024-03-19T03:30:36.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.867+0000 7fd1d3e00640 1 Processor -- start 2024-03-19T03:30:36.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.867+0000 7fd1d3e00640 1 -- start start 2024-03-19T03:30:36.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.867+0000 7fd1d3e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1d41020f0 0x7fd1d4199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:36.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.867+0000 7fd1d3e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1d4104a20 0x7fd1d419a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:36.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.867+0000 7fd1d3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1d4107350 0x7fd1d41a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:36.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.867+0000 7fd1d3e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d410d350 con 0x7fd1d4104a20 2024-03-19T03:30:36.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.867+0000 7fd1d3e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d410d1d0 con 0x7fd1d4107350 2024-03-19T03:30:36.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.867+0000 7fd1d3e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d410d4d0 con 0x7fd1d41020f0 2024-03-19T03:30:36.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.867+0000 7fd1d2a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1d41020f0 0x7fd1d4199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:36.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.867+0000 7fd1d2000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1d4104a20 0x7fd1d419a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:36.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.867+0000 7fd1d2a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1d41020f0 0x7fd1d4199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:51282/0 (socket says 172.21.15.117:51282) 2024-03-19T03:30:36.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.867+0000 7fd1d2a00640 1 -- 172.21.15.117:0/2542182041 learned_addr learned my addr 172.21.15.117:0/2542182041 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:36.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.867+0000 7fd1d3400640 1 --2- 172.21.15.117:0/2542182041 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1d4107350 0x7fd1d41a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:36.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.868+0000 7fd1d3400640 1 -- 172.21.15.117:0/2542182041 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1d41020f0 msgr2=0x7fd1d4199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:36.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.868+0000 7fd1d3400640 1 --2- 172.21.15.117:0/2542182041 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1d41020f0 0x7fd1d4199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:36.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.868+0000 7fd1d3400640 1 -- 172.21.15.117:0/2542182041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1d4104a20 msgr2=0x7fd1d419a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:36.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.868+0000 7fd1d3400640 1 --2- 172.21.15.117:0/2542182041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1d4104a20 0x7fd1d419a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:36.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.868+0000 7fd1d3400640 1 -- 172.21.15.117:0/2542182041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d41a1800 con 0x7fd1d4107350 2024-03-19T03:30:36.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.868+0000 7fd1d2000640 1 --2- 172.21.15.117:0/2542182041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1d4104a20 0x7fd1d419a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:30:36.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.868+0000 7fd1d3400640 1 --2- 172.21.15.117:0/2542182041 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1d4107350 0x7fd1d41a1190 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00ec50 tx=0x7fd1cc00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:36.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.868+0000 7fd1cb400640 1 -- 172.21.15.117:0/2542182041 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc003b70 con 0x7fd1d4107350 2024-03-19T03:30:36.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.868+0000 7fd1cb400640 1 -- 172.21.15.117:0/2542182041 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1cc003d10 con 0x7fd1d4107350 2024-03-19T03:30:36.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.868+0000 7fd1d3e00640 1 -- 172.21.15.117:0/2542182041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1d41a1af0 con 0x7fd1d4107350 2024-03-19T03:30:36.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.868+0000 7fd1cb400640 1 -- 172.21.15.117:0/2542182041 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc010890 con 0x7fd1d4107350 2024-03-19T03:30:36.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.868+0000 7fd1d3e00640 1 -- 172.21.15.117:0/2542182041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1d41a1fc0 con 0x7fd1d4107350 2024-03-19T03:30:36.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.870+0000 7fd1d3e00640 1 -- 172.21.15.117:0/2542182041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1d4108ea0 con 0x7fd1d4107350 2024-03-19T03:30:36.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.871+0000 7fd1cb400640 1 -- 172.21.15.117:0/2542182041 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fd1cc003fc0 con 0x7fd1d4107350 2024-03-19T03:30:36.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.872+0000 7fd1cb400640 1 --2- 172.21.15.117:0/2542182041 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd1b80a2800 0x7fd1b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:36.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.873+0000 7fd1cb400640 1 -- 172.21.15.117:0/2542182041 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8277+0+0 (secure 0 0 0) 0x7fd1cc0cf010 con 0x7fd1d4107350 2024-03-19T03:30:36.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.873+0000 7fd1d2a00640 1 --2- 172.21.15.117:0/2542182041 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd1b80a2800 0x7fd1b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:36.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.873+0000 7fd1d2a00640 1 --2- 172.21.15.117:0/2542182041 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd1b80a2800 0x7fd1b80a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4009cd0 tx=0x7fd1c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:36.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:36.876+0000 7fd1cb400640 1 -- 172.21.15.117:0/2542182041 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd1cc09c080 con 0x7fd1d4107350 2024-03-19T03:30:36.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:36 smithi119 ceph-mon[66861]: osdmap e84: 12 total, 12 up, 12 in 2024-03-19T03:30:36.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:36 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-19T03:30:36.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:36 smithi119 ceph-mon[66861]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-19T03:30:36.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:36 smithi119 ceph-mon[66861]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-19T03:30:36.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:36 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3484525870' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-19T03:30:36.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:36 smithi119 ceph-mon[66861]: pgmap v313: 97 pgs: 2 creating+activating, 25 creating+peering, 37 active+clean, 33 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-19T03:30:37.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:36 smithi117 ceph-mon[31838]: osdmap e84: 12 total, 12 up, 12 in 2024-03-19T03:30:37.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:36 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-19T03:30:37.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:36 smithi117 ceph-mon[31838]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-19T03:30:37.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:36 smithi117 ceph-mon[31838]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-19T03:30:37.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:36 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3484525870' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-19T03:30:37.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:36 smithi117 ceph-mon[31838]: pgmap v313: 97 pgs: 2 creating+activating, 25 creating+peering, 37 active+clean, 33 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-19T03:30:37.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:37.074+0000 7fd1d3e00640 1 -- 172.21.15.117:0/2542182041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fd1d4062420 con 0x7fd1d4107350 2024-03-19T03:30:37.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:36 smithi178 ceph-mon[67578]: osdmap e84: 12 total, 12 up, 12 in 2024-03-19T03:30:37.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:36 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-19T03:30:37.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:36 smithi178 ceph-mon[67578]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-19T03:30:37.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:36 smithi178 ceph-mon[67578]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-19T03:30:37.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:36 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3484525870' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-19T03:30:37.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:36 smithi178 ceph-mon[67578]: pgmap v313: 97 pgs: 2 creating+activating, 25 creating+peering, 37 active+clean, 33 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-19T03:30:37.944 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:37 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3484525870' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-19T03:30:37.944 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:37 smithi117 ceph-mon[31838]: osdmap e85: 12 total, 12 up, 12 in 2024-03-19T03:30:37.944 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:37 smithi117 ceph-mon[31838]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-19T03:30:37.944 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:37 smithi117 ceph-mon[31838]: Cluster is now healthy 2024-03-19T03:30:37.944 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:37 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] up:active 2024-03-19T03:30:37.944 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:37 smithi117 ceph-mon[31838]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T03:30:37.944 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:37 smithi117 ceph-mon[31838]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T03:30:37.944 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:37 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2542182041' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-19T03:30:37.945 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:37 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-19T03:30:37.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:37 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3484525870' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-19T03:30:37.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:37 smithi119 ceph-mon[66861]: osdmap e85: 12 total, 12 up, 12 in 2024-03-19T03:30:37.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:37 smithi119 ceph-mon[66861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-19T03:30:37.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:37 smithi119 ceph-mon[66861]: Cluster is now healthy 2024-03-19T03:30:37.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:37 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] up:active 2024-03-19T03:30:37.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:37 smithi119 ceph-mon[66861]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T03:30:37.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:37 smithi119 ceph-mon[66861]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T03:30:37.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:37 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2542182041' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-19T03:30:37.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:37 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-19T03:30:38.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:37 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3484525870' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-19T03:30:38.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:37 smithi178 ceph-mon[67578]: osdmap e85: 12 total, 12 up, 12 in 2024-03-19T03:30:38.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:37 smithi178 ceph-mon[67578]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-19T03:30:38.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:37 smithi178 ceph-mon[67578]: Cluster is now healthy 2024-03-19T03:30:38.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:37 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] up:active 2024-03-19T03:30:38.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:37 smithi178 ceph-mon[67578]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T03:30:38.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:37 smithi178 ceph-mon[67578]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T03:30:38.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:37 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2542182041' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-19T03:30:38.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:37 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-19T03:30:38.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.705+0000 7fd1cb400640 1 -- 172.21.15.117:0/2542182041 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) ==== 228+0+0 (secure 0 0 0) 0x7fd1cc09b980 con 0x7fd1d4107350 2024-03-19T03:30:38.706 INFO:teuthology.orchestra.run.smithi117.stderr:pool 'cephfs_data_ec' created 2024-03-19T03:30:38.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.711+0000 7fd1d3e00640 1 -- 172.21.15.117:0/2542182041 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd1b80a2800 msgr2=0x7fd1b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:38.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.711+0000 7fd1d3e00640 1 --2- 172.21.15.117:0/2542182041 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd1b80a2800 0x7fd1b80a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4009cd0 tx=0x7fd1c4005f90 comp rx=0 tx=0).stop 2024-03-19T03:30:38.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.711+0000 7fd1d3e00640 1 -- 172.21.15.117:0/2542182041 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1d4107350 msgr2=0x7fd1d41a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:38.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.711+0000 7fd1d3e00640 1 --2- 172.21.15.117:0/2542182041 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1d4107350 0x7fd1d41a1190 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00ec50 tx=0x7fd1cc00c6a0 comp rx=0 tx=0).stop 2024-03-19T03:30:38.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.712+0000 7fd1d3e00640 1 -- 172.21.15.117:0/2542182041 shutdown_connections 2024-03-19T03:30:38.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.712+0000 7fd1d3e00640 1 --2- 172.21.15.117:0/2542182041 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd1b80a2800 0x7fd1b80a4cc0 secure :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4009cd0 tx=0x7fd1c4005f90 comp rx=0 tx=0).stop 2024-03-19T03:30:38.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.712+0000 7fd1d3e00640 1 --2- 172.21.15.117:0/2542182041 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd1d4107350 0x7fd1d41a1190 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:38.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.712+0000 7fd1d3e00640 1 --2- 172.21.15.117:0/2542182041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1d4104a20 0x7fd1d419a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:38.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.712+0000 7fd1d3e00640 1 --2- 172.21.15.117:0/2542182041 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1d41020f0 0x7fd1d4199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:38.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.712+0000 7fd1d3e00640 1 -- 172.21.15.117:0/2542182041 >> 172.21.15.117:0/2542182041 conn(0x7fd1d40fbb00 msgr2=0x7fd1d41017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:38.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.713+0000 7fd1d3e00640 1 -- 172.21.15.117:0/2542182041 shutdown_connections 2024-03-19T03:30:38.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.713+0000 7fd1d3e00640 1 -- 172.21.15.117:0/2542182041 wait complete. 2024-03-19T03:30:38.739 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-19T03:30:38.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.934+0000 7fb9c4600640 1 -- 172.21.15.117:0/3685574956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9bc100a50 msgr2=0x7fb9bc100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:38.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.934+0000 7fb9c4600640 1 --2- 172.21.15.117:0/3685574956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9bc100a50 0x7fb9bc100eb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8009900 tx=0x7fb9a802bc30 comp rx=0 tx=0).stop 2024-03-19T03:30:38.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.934+0000 7fb9c4600640 1 -- 172.21.15.117:0/3685574956 shutdown_connections 2024-03-19T03:30:38.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.934+0000 7fb9c4600640 1 --2- 172.21.15.117:0/3685574956 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb9bc1013f0 0x7fb9bc101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:38.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.934+0000 7fb9c4600640 1 --2- 172.21.15.117:0/3685574956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9bc100a50 0x7fb9bc100eb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:38.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.934+0000 7fb9c4600640 1 --2- 172.21.15.117:0/3685574956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9bc1000f0 0x7fb9bc100510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:38.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.934+0000 7fb9c4600640 1 -- 172.21.15.117:0/3685574956 >> 172.21.15.117:0/3685574956 conn(0x7fb9bc0fbcb0 msgr2=0x7fb9bc0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:38.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.934+0000 7fb9c4600640 1 -- 172.21.15.117:0/3685574956 shutdown_connections 2024-03-19T03:30:38.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.934+0000 7fb9c4600640 1 -- 172.21.15.117:0/3685574956 wait complete. 2024-03-19T03:30:38.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.935+0000 7fb9c4600640 1 Processor -- start 2024-03-19T03:30:38.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.935+0000 7fb9c4600640 1 -- start start 2024-03-19T03:30:38.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.935+0000 7fb9c4600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9bc1000f0 0x7fb9bc199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:38.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.935+0000 7fb9c4600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb9bc100a50 0x7fb9bc19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:38.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.935+0000 7fb9c4600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9bc1013f0 0x7fb9bc1a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:38.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.935+0000 7fb9c4600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc10f6e0 con 0x7fb9bc1013f0 2024-03-19T03:30:38.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.935+0000 7fb9c4600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc10f560 con 0x7fb9bc1000f0 2024-03-19T03:30:38.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.935+0000 7fb9c4600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fb9bc10f860 con 0x7fb9bc100a50 2024-03-19T03:30:38.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.936+0000 7fb9c2600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9bc1013f0 0x7fb9bc1a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:38.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.936+0000 7fb9c1c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9bc1000f0 0x7fb9bc199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:38.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.936+0000 7fb9c2600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9bc1013f0 0x7fb9bc1a1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39672/0 (socket says 172.21.15.117:39672) 2024-03-19T03:30:38.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.936+0000 7fb9c1c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9bc1000f0 0x7fb9bc199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57558/0 (socket says 172.21.15.117:57558) 2024-03-19T03:30:38.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.936+0000 7fb9c2600640 1 -- 172.21.15.117:0/1746093521 learned_addr learned my addr 172.21.15.117:0/1746093521 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:38.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.936+0000 7fb9c1200640 1 --2- 172.21.15.117:0/1746093521 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb9bc100a50 0x7fb9bc19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:38.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.936+0000 7fb9c1c00640 1 -- 172.21.15.117:0/1746093521 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb9bc100a50 msgr2=0x7fb9bc19a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:38.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.936+0000 7fb9c1c00640 1 --2- 172.21.15.117:0/1746093521 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb9bc100a50 0x7fb9bc19a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:38.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.936+0000 7fb9c1c00640 1 -- 172.21.15.117:0/1746093521 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9bc1013f0 msgr2=0x7fb9bc1a1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:38.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.936+0000 7fb9c1c00640 1 --2- 172.21.15.117:0/1746093521 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9bc1013f0 0x7fb9bc1a1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:38.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.936+0000 7fb9c1c00640 1 -- 172.21.15.117:0/1746093521 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9bc1a1970 con 0x7fb9bc1000f0 2024-03-19T03:30:38.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.936+0000 7fb9c2600640 1 --2- 172.21.15.117:0/1746093521 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9bc1013f0 0x7fb9bc1a1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:30:38.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.937+0000 7fb9c1c00640 1 --2- 172.21.15.117:0/1746093521 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9bc1000f0 0x7fb9bc199d50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb9b000e9f0 tx=0x7fb9b000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:38.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.937+0000 7fb9ba000640 1 -- 172.21.15.117:0/1746093521 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b000ccc0 con 0x7fb9bc1000f0 2024-03-19T03:30:38.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.937+0000 7fb9ba000640 1 -- 172.21.15.117:0/1746093521 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9b0010910 con 0x7fb9bc1000f0 2024-03-19T03:30:38.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.937+0000 7fb9c4600640 1 -- 172.21.15.117:0/1746093521 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9bc1a1c60 con 0x7fb9bc1000f0 2024-03-19T03:30:38.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.937+0000 7fb9ba000640 1 -- 172.21.15.117:0/1746093521 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b0018960 con 0x7fb9bc1000f0 2024-03-19T03:30:38.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.937+0000 7fb9c4600640 1 -- 172.21.15.117:0/1746093521 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9bc1a21d0 con 0x7fb9bc1000f0 2024-03-19T03:30:38.940 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.938+0000 7fb9c4600640 1 -- 172.21.15.117:0/1746093521 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9bc10af70 con 0x7fb9bc1000f0 2024-03-19T03:30:38.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.941+0000 7fb9ba000640 1 -- 172.21.15.117:0/1746093521 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fb9b0010ac0 con 0x7fb9bc1000f0 2024-03-19T03:30:38.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.943+0000 7fb9ba000640 1 --2- 172.21.15.117:0/1746093521 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fb9a40a2880 0x7fb9a40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:38.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.943+0000 7fb9ba000640 1 -- 172.21.15.117:0/1746093521 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7fb9b00cefd0 con 0x7fb9bc1000f0 2024-03-19T03:30:38.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.943+0000 7fb9c1200640 1 --2- 172.21.15.117:0/1746093521 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fb9a40a2880 0x7fb9a40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:38.944 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.944+0000 7fb9c1200640 1 --2- 172.21.15.117:0/1746093521 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fb9a40a2880 0x7fb9a40a4d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb9a802c240 tx=0x7fb9a802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:38.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:38.947+0000 7fb9ba000640 1 -- 172.21.15.117:0/1746093521 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb9b009b7a0 con 0x7fb9bc1000f0 2024-03-19T03:30:38.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:38 smithi119 ceph-mon[66861]: osdmap e86: 12 total, 12 up, 12 in 2024-03-19T03:30:38.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:38 smithi119 ceph-mon[66861]: pgmap v316: 97 pgs: 2 creating+activating, 25 creating+peering, 52 active+clean, 18 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 7 op/s 2024-03-19T03:30:39.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:38 smithi117 ceph-mon[31838]: osdmap e86: 12 total, 12 up, 12 in 2024-03-19T03:30:39.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:38 smithi117 ceph-mon[31838]: pgmap v316: 97 pgs: 2 creating+activating, 25 creating+peering, 52 active+clean, 18 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 7 op/s 2024-03-19T03:30:39.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:38 smithi178 ceph-mon[67578]: osdmap e86: 12 total, 12 up, 12 in 2024-03-19T03:30:39.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:38 smithi178 ceph-mon[67578]: pgmap v316: 97 pgs: 2 creating+activating, 25 creating+peering, 52 active+clean, 18 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 7 op/s 2024-03-19T03:30:39.140 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.139+0000 7fb9c4600640 1 -- 172.21.15.117:0/1746093521 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fb9bc0624c0 con 0x7fb9bc1000f0 2024-03-19T03:30:39.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.708+0000 7fb9ba000640 1 -- 172.21.15.117:0/1746093521 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7fb9b009f840 con 0x7fb9bc1000f0 2024-03-19T03:30:39.713 INFO:teuthology.orchestra.run.smithi117.stderr:set pool 4 allow_ec_overwrites to true 2024-03-19T03:30:39.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.713+0000 7fb9c4600640 1 -- 172.21.15.117:0/1746093521 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fb9a40a2880 msgr2=0x7fb9a40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:39.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.713+0000 7fb9c4600640 1 --2- 172.21.15.117:0/1746093521 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fb9a40a2880 0x7fb9a40a4d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb9a802c240 tx=0x7fb9a802df90 comp rx=0 tx=0).stop 2024-03-19T03:30:39.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.713+0000 7fb9c4600640 1 -- 172.21.15.117:0/1746093521 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9bc1000f0 msgr2=0x7fb9bc199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:39.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.713+0000 7fb9c4600640 1 --2- 172.21.15.117:0/1746093521 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9bc1000f0 0x7fb9bc199d50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb9b000e9f0 tx=0x7fb9b000eeb0 comp rx=0 tx=0).stop 2024-03-19T03:30:39.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.714+0000 7fb9c4600640 1 -- 172.21.15.117:0/1746093521 shutdown_connections 2024-03-19T03:30:39.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.714+0000 7fb9c4600640 1 --2- 172.21.15.117:0/1746093521 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9bc1013f0 0x7fb9bc1a1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:39.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.714+0000 7fb9c4600640 1 --2- 172.21.15.117:0/1746093521 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb9bc100a50 0x7fb9bc19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:39.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.714+0000 7fb9c4600640 1 --2- 172.21.15.117:0/1746093521 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fb9a40a2880 0x7fb9a40a4d40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:39.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.714+0000 7fb9c4600640 1 --2- 172.21.15.117:0/1746093521 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9bc1000f0 0x7fb9bc199d50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:39.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.714+0000 7fb9c4600640 1 -- 172.21.15.117:0/1746093521 >> 172.21.15.117:0/1746093521 conn(0x7fb9bc0fbcb0 msgr2=0x7fb9bc0fd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:39.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.715+0000 7fb9c4600640 1 -- 172.21.15.117:0/1746093521 shutdown_connections 2024-03-19T03:30:39.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.715+0000 7fb9c4600640 1 -- 172.21.15.117:0/1746093521 wait complete. 2024-03-19T03:30:39.718 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 19 03:30:39 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-b[97021]: 2024-03-19T03:30:39.434+0000 7f2c847ae700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-19T03:30:39.751 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-19T03:30:39.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.929+0000 7f2615a00640 1 -- 172.21.15.117:0/2404681261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2610101700 msgr2=0x7f2610101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:39.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.929+0000 7f2615a00640 1 --2- 172.21.15.117:0/2404681261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2610101700 0x7f2610101b20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2600009900 tx=0x7f260002bc30 comp rx=0 tx=0).stop 2024-03-19T03:30:39.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.929+0000 7f2615a00640 1 -- 172.21.15.117:0/2404681261 shutdown_connections 2024-03-19T03:30:39.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.929+0000 7f2615a00640 1 --2- 172.21.15.117:0/2404681261 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26101032b0 0x7f2610109b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:39.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.929+0000 7f2615a00640 1 --2- 172.21.15.117:0/2404681261 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26101028f0 0x7f2610102d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:39.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.929+0000 7f2615a00640 1 --2- 172.21.15.117:0/2404681261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2610101700 0x7f2610101b20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:39.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.929+0000 7f2615a00640 1 -- 172.21.15.117:0/2404681261 >> 172.21.15.117:0/2404681261 conn(0x7f26100fcc90 msgr2=0x7f26100ff0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:39.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.929+0000 7f2615a00640 1 -- 172.21.15.117:0/2404681261 shutdown_connections 2024-03-19T03:30:39.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.929+0000 7f2615a00640 1 -- 172.21.15.117:0/2404681261 wait complete. 2024-03-19T03:30:39.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.930+0000 7f2615a00640 1 Processor -- start 2024-03-19T03:30:39.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.930+0000 7f2615a00640 1 -- start start 2024-03-19T03:30:39.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.930+0000 7f2615a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2610101700 0x7f2610197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:39.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.930+0000 7f2615a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26101028f0 0x7f2610197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:39.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.930+0000 7f2615a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26101032b0 0x7f261019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:39.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.931+0000 7f2615a00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f261010d3c0 con 0x7f2610101700 2024-03-19T03:30:39.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.931+0000 7f2615a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f261010d240 con 0x7f26101028f0 2024-03-19T03:30:39.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.931+0000 7f2615a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f261010d540 con 0x7f26101032b0 2024-03-19T03:30:39.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.931+0000 7f260fe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2610101700 0x7f2610197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:39.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.931+0000 7f260fe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2610101700 0x7f2610197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39692/0 (socket says 172.21.15.117:39692) 2024-03-19T03:30:39.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.931+0000 7f260fe00640 1 -- 172.21.15.117:0/3613367261 learned_addr learned my addr 172.21.15.117:0/3613367261 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:39.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.931+0000 7f260f400640 1 --2- 172.21.15.117:0/3613367261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26101028f0 0x7f2610197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:39.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.931+0000 7f2615000640 1 --2- 172.21.15.117:0/3613367261 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26101032b0 0x7f261019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:39.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.931+0000 7f260f400640 1 -- 172.21.15.117:0/3613367261 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26101032b0 msgr2=0x7f261019efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:39.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.931+0000 7f260f400640 1 --2- 172.21.15.117:0/3613367261 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26101032b0 0x7f261019efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:39.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.931+0000 7f260f400640 1 -- 172.21.15.117:0/3613367261 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2610101700 msgr2=0x7f2610197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:39.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.931+0000 7f260f400640 1 --2- 172.21.15.117:0/3613367261 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2610101700 0x7f2610197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:39.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.931+0000 7f260f400640 1 -- 172.21.15.117:0/3613367261 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f261019f650 con 0x7f26101028f0 2024-03-19T03:30:39.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.931+0000 7f260fe00640 1 --2- 172.21.15.117:0/3613367261 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2610101700 0x7f2610197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:30:39.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.931+0000 7f260f400640 1 --2- 172.21.15.117:0/3613367261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26101028f0 0x7f2610197f70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f25fc00e9f0 tx=0x7f25fc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:39.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.932+0000 7f260cc00640 1 -- 172.21.15.117:0/3613367261 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25fc00ccc0 con 0x7f26101028f0 2024-03-19T03:30:39.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.932+0000 7f260cc00640 1 -- 172.21.15.117:0/3613367261 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25fc005020 con 0x7f26101028f0 2024-03-19T03:30:39.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.932+0000 7f260cc00640 1 -- 172.21.15.117:0/3613367261 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25fc0049e0 con 0x7f26101028f0 2024-03-19T03:30:39.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.932+0000 7f2615a00640 1 -- 172.21.15.117:0/3613367261 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f261019f940 con 0x7f26101028f0 2024-03-19T03:30:39.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.932+0000 7f2615a00640 1 -- 172.21.15.117:0/3613367261 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f261019fe40 con 0x7f26101028f0 2024-03-19T03:30:39.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.933+0000 7f2615a00640 1 -- 172.21.15.117:0/3613367261 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2610108a20 con 0x7f26101028f0 2024-03-19T03:30:39.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.935+0000 7f260cc00640 1 -- 172.21.15.117:0/3613367261 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f25fc005560 con 0x7f26101028f0 2024-03-19T03:30:39.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.936+0000 7f260cc00640 1 --2- 172.21.15.117:0/3613367261 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f25f40a2880 0x7f25f40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:39.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.937+0000 7f260cc00640 1 -- 172.21.15.117:0/3613367261 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f25fc0cebd0 con 0x7f26101028f0 2024-03-19T03:30:39.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.937+0000 7f260fe00640 1 --2- 172.21.15.117:0/3613367261 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f25f40a2880 0x7f25f40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:39.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.937+0000 7f260fe00640 1 --2- 172.21.15.117:0/3613367261 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f25f40a2880 0x7f25f40a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2600006010 tx=0x7f260002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:39.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:39.940+0000 7f260cc00640 1 -- 172.21.15.117:0/3613367261 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f25fc0051d0 con 0x7f26101028f0 2024-03-19T03:30:39.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:39 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-19T03:30:39.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:39 smithi119 ceph-mon[66861]: osdmap e87: 12 total, 12 up, 12 in 2024-03-19T03:30:39.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:39 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1746093521' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-19T03:30:39.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:39 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-19T03:30:40.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:39 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-19T03:30:40.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:39 smithi117 ceph-mon[31838]: osdmap e87: 12 total, 12 up, 12 in 2024-03-19T03:30:40.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:39 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1746093521' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-19T03:30:40.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:39 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-19T03:30:40.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:39 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-19T03:30:40.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:39 smithi178 ceph-mon[67578]: osdmap e87: 12 total, 12 up, 12 in 2024-03-19T03:30:40.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:39 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1746093521' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-19T03:30:40.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:39 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-19T03:30:40.178 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.178+0000 7f2615a00640 1 -- 172.21.15.117:0/3613367261 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f2610062420 con 0x7f26101028f0 2024-03-19T03:30:40.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.739+0000 7f260cc00640 1 -- 172.21.15.117:0/3613367261 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f25fc09b9f0 con 0x7f26101028f0 2024-03-19T03:30:40.740 INFO:teuthology.orchestra.run.smithi117.stderr:added data pool 4 to fsmap 2024-03-19T03:30:40.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.742+0000 7f2615a00640 1 -- 172.21.15.117:0/3613367261 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f25f40a2880 msgr2=0x7f25f40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:40.744 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.742+0000 7f2615a00640 1 --2- 172.21.15.117:0/3613367261 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f25f40a2880 0x7f25f40a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2600006010 tx=0x7f260002df90 comp rx=0 tx=0).stop 2024-03-19T03:30:40.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.742+0000 7f2615a00640 1 -- 172.21.15.117:0/3613367261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26101028f0 msgr2=0x7f2610197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:40.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.742+0000 7f2615a00640 1 --2- 172.21.15.117:0/3613367261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26101028f0 0x7f2610197f70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f25fc00e9f0 tx=0x7f25fc00eeb0 comp rx=0 tx=0).stop 2024-03-19T03:30:40.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.742+0000 7f2615a00640 1 -- 172.21.15.117:0/3613367261 shutdown_connections 2024-03-19T03:30:40.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.742+0000 7f2615a00640 1 --2- 172.21.15.117:0/3613367261 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f26101032b0 0x7f261019efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:40.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.742+0000 7f2615a00640 1 --2- 172.21.15.117:0/3613367261 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f25f40a2880 0x7f25f40a4d40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:40.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.743+0000 7f2615a00640 1 --2- 172.21.15.117:0/3613367261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26101028f0 0x7f2610197f70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:40.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.743+0000 7f2615a00640 1 --2- 172.21.15.117:0/3613367261 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2610101700 0x7f2610197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:40.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.743+0000 7f2615a00640 1 -- 172.21.15.117:0/3613367261 >> 172.21.15.117:0/3613367261 conn(0x7f26100fcc90 msgr2=0x7f2610105bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:40.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.743+0000 7f2615a00640 1 -- 172.21.15.117:0/3613367261 shutdown_connections 2024-03-19T03:30:40.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.743+0000 7f2615a00640 1 -- 172.21.15.117:0/3613367261 wait complete. 2024-03-19T03:30:40.763 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:30:40.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.962+0000 7f66e3200640 1 -- 172.21.15.117:0/448041961 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66e4108e70 msgr2=0x7f66e41092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:40.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.962+0000 7f66e3200640 1 --2- 172.21.15.117:0/448041961 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66e4108e70 0x7f66e41092f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f66cc009900 tx=0x7f66cc02bc40 comp rx=0 tx=0).stop 2024-03-19T03:30:40.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.962+0000 7f66e3200640 1 -- 172.21.15.117:0/448041961 shutdown_connections 2024-03-19T03:30:40.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.962+0000 7f66e3200640 1 --2- 172.21.15.117:0/448041961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f66e4109830 0x7f66e41100c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:40.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.962+0000 7f66e3200640 1 --2- 172.21.15.117:0/448041961 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66e4108e70 0x7f66e41092f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:40.964 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.962+0000 7f66e3200640 1 --2- 172.21.15.117:0/448041961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66e4107c80 0x7f66e41080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:40.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.962+0000 7f66e3200640 1 -- 172.21.15.117:0/448041961 >> 172.21.15.117:0/448041961 conn(0x7f66e4076c30 msgr2=0x7f66e4079050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:40.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.962+0000 7f66e3200640 1 -- 172.21.15.117:0/448041961 shutdown_connections 2024-03-19T03:30:40.965 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.962+0000 7f66e3200640 1 -- 172.21.15.117:0/448041961 wait complete. 2024-03-19T03:30:40.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.963+0000 7f66e3200640 1 Processor -- start 2024-03-19T03:30:40.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.964+0000 7f66e3200640 1 -- start start 2024-03-19T03:30:40.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.964+0000 7f66e3200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66e4107c80 0x7f66e419df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:40.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.964+0000 7f66e3200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f66e4108e70 0x7f66e419e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:40.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.964+0000 7f66e3200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66e4109830 0x7f66e41a5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:40.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.964+0000 7f66e3200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f66e4113900 con 0x7f66e4108e70 2024-03-19T03:30:40.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.964+0000 7f66e3200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f66e4113780 con 0x7f66e4107c80 2024-03-19T03:30:40.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.964+0000 7f66e3200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f66e4113a80 con 0x7f66e4109830 2024-03-19T03:30:40.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.964+0000 7f66e1e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66e4107c80 0x7f66e419df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:40.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.964+0000 7f66e1e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66e4107c80 0x7f66e419df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57590/0 (socket says 172.21.15.117:57590) 2024-03-19T03:30:40.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.964+0000 7f66e1e00640 1 -- 172.21.15.117:0/3216611845 learned_addr learned my addr 172.21.15.117:0/3216611845 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:40.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.964+0000 7f66dbe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f66e4108e70 0x7f66e419e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:40.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.964+0000 7f66e2800640 1 --2- 172.21.15.117:0/3216611845 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66e4109830 0x7f66e41a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:40.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.964+0000 7f66e1e00640 1 -- 172.21.15.117:0/3216611845 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66e4109830 msgr2=0x7f66e41a5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:40.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.964+0000 7f66e1e00640 1 --2- 172.21.15.117:0/3216611845 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66e4109830 0x7f66e41a5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:40.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.964+0000 7f66e1e00640 1 -- 172.21.15.117:0/3216611845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f66e4108e70 msgr2=0x7f66e419e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:40.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.964+0000 7f66e1e00640 1 --2- 172.21.15.117:0/3216611845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f66e4108e70 0x7f66e419e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:40.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.965+0000 7f66e1e00640 1 -- 172.21.15.117:0/3216611845 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66e41a5b90 con 0x7f66e4107c80 2024-03-19T03:30:40.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.965+0000 7f66e2800640 1 --2- 172.21.15.117:0/3216611845 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66e4109830 0x7f66e41a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:30:40.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.965+0000 7f66dbe00640 1 --2- 172.21.15.117:0/3216611845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f66e4108e70 0x7f66e419e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:30:40.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.965+0000 7f66e1e00640 1 --2- 172.21.15.117:0/3216611845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66e4107c80 0x7f66e419df70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f66d400d950 tx=0x7f66d400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:40.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.965+0000 7f66daa00640 1 -- 172.21.15.117:0/3216611845 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66d4017070 con 0x7f66e4107c80 2024-03-19T03:30:40.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.965+0000 7f66daa00640 1 -- 172.21.15.117:0/3216611845 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66d4009c30 con 0x7f66e4107c80 2024-03-19T03:30:40.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.965+0000 7f66daa00640 1 -- 172.21.15.117:0/3216611845 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66d400f600 con 0x7f66e4107c80 2024-03-19T03:30:40.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.965+0000 7f66e3200640 1 -- 172.21.15.117:0/3216611845 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66e41a5e80 con 0x7f66e4107c80 2024-03-19T03:30:40.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.965+0000 7f66e3200640 1 -- 172.21.15.117:0/3216611845 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66e41a6290 con 0x7f66e4107c80 2024-03-19T03:30:40.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.966+0000 7f66e3200640 1 -- 172.21.15.117:0/3216611845 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66e410efa0 con 0x7f66e4107c80 2024-03-19T03:30:40.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.968+0000 7f66daa00640 1 -- 172.21.15.117:0/3216611845 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f66d401a070 con 0x7f66e4107c80 2024-03-19T03:30:40.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.969+0000 7f66daa00640 1 --2- 172.21.15.117:0/3216611845 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f66c80a2800 0x7f66c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:40.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.970+0000 7f66daa00640 1 -- 172.21.15.117:0/3216611845 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f66d40ce750 con 0x7f66e4107c80 2024-03-19T03:30:40.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.970+0000 7f66dbe00640 1 --2- 172.21.15.117:0/3216611845 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f66c80a2800 0x7f66c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:40.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.970+0000 7f66dbe00640 1 --2- 172.21.15.117:0/3216611845 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f66c80a2800 0x7f66c80a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f66cc02c7c0 tx=0x7f66cc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:40.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:40.973+0000 7f66daa00640 1 -- 172.21.15.117:0/3216611845 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f66d4009de0 con 0x7f66e4107c80 2024-03-19T03:30:40.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:40 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-19T03:30:40.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:40 smithi119 ceph-mon[66861]: osdmap e88: 12 total, 12 up, 12 in 2024-03-19T03:30:40.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:40 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3613367261' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-19T03:30:40.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:40 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-19T03:30:40.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:40 smithi119 ceph-mon[66861]: pgmap v319: 129 pgs: 20 creating+peering, 83 active+clean, 26 unknown; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 7 op/s 2024-03-19T03:30:41.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:40 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-19T03:30:41.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:40 smithi117 ceph-mon[31838]: osdmap e88: 12 total, 12 up, 12 in 2024-03-19T03:30:41.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:40 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3613367261' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-19T03:30:41.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:40 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-19T03:30:41.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:40 smithi117 ceph-mon[31838]: pgmap v319: 129 pgs: 20 creating+peering, 83 active+clean, 26 unknown; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 7 op/s 2024-03-19T03:30:41.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:40 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-19T03:30:41.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:40 smithi178 ceph-mon[67578]: osdmap e88: 12 total, 12 up, 12 in 2024-03-19T03:30:41.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:40 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3613367261' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-19T03:30:41.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:40 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-19T03:30:41.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:40 smithi178 ceph-mon[67578]: pgmap v319: 129 pgs: 20 creating+peering, 83 active+clean, 26 unknown; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 7 op/s 2024-03-19T03:30:41.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.213+0000 7f66e3200640 1 -- 172.21.15.117:0/3216611845 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66e4062450 con 0x7f66e4107c80 2024-03-19T03:30:41.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.214+0000 7f66daa00640 1 -- 172.21.15.117:0/3216611845 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11818 (secure 0 0 0) 0x7f66d409b550 con 0x7f66e4107c80 2024-03-19T03:30:41.215 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:30:41.216 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/54638259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":54638259},{"type":"v1","addr":"172.21.15.117:6835","nonce":54638259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3608185039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3608185039},{"type":"v1","addr":"172.21.15.117:6837","nonce":3608185039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/1291833214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1291833214},{"type":"v1","addr":"172.21.15.119:6833","nonce":1291833214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1579860804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1579860804},{"type":"v1","addr":"172.21.15.178:6835","nonce":1579860804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24515,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/1686938941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1686938941},{"type":"v1","addr":"172.21.15.119:6835","nonce":1686938941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1770913538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":1770913538},{"type":"v1","addr":"172.21.15.178:6837","nonce":1770913538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6839/2655593383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2655593383},{"type":"v1","addr":"172.21.15.178:6839","nonce":2655593383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2054992738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2054992738},{"type":"v1","addr":"172.21.15.119:6837","nonce":2054992738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24569,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6839/684335078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":684335078},{"type":"v1","addr":"172.21.15.117:6839","nonce":684335078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24581,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6841/1329933892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6840","nonce":1329933892},{"type":"v1","addr":"172.21.15.117:6841","nonce":1329933892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T03:30:35.081740+0000","modified":"2024-03-19T03:30:40.181111+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/2383354833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2383354833},{"type":"v1","addr":"172.21.15.119:6839","nonce":2383354833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24599,"qdb_cluster":[24599]},"id":1}]} 2024-03-19T03:30:41.216 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 17 2024-03-19T03:30:41.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.217+0000 7f66e3200640 1 -- 172.21.15.117:0/3216611845 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f66c80a2800 msgr2=0x7f66c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:41.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.217+0000 7f66e3200640 1 --2- 172.21.15.117:0/3216611845 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f66c80a2800 0x7f66c80a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f66cc02c7c0 tx=0x7f66cc005fd0 comp rx=0 tx=0).stop 2024-03-19T03:30:41.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.217+0000 7f66e3200640 1 -- 172.21.15.117:0/3216611845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66e4107c80 msgr2=0x7f66e419df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:41.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.217+0000 7f66e3200640 1 --2- 172.21.15.117:0/3216611845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66e4107c80 0x7f66e419df70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f66d400d950 tx=0x7f66d400de10 comp rx=0 tx=0).stop 2024-03-19T03:30:41.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.217+0000 7f66e3200640 1 -- 172.21.15.117:0/3216611845 shutdown_connections 2024-03-19T03:30:41.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.217+0000 7f66e3200640 1 --2- 172.21.15.117:0/3216611845 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66e4109830 0x7f66e41a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:41.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.217+0000 7f66e3200640 1 --2- 172.21.15.117:0/3216611845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f66e4108e70 0x7f66e419e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:41.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.217+0000 7f66e3200640 1 --2- 172.21.15.117:0/3216611845 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f66c80a2800 0x7f66c80a4cc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:41.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.217+0000 7f66e3200640 1 --2- 172.21.15.117:0/3216611845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66e4107c80 0x7f66e419df70 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:41.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.217+0000 7f66e3200640 1 -- 172.21.15.117:0/3216611845 >> 172.21.15.117:0/3216611845 conn(0x7f66e4076c30 msgr2=0x7f66e410c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:41.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.217+0000 7f66e3200640 1 -- 172.21.15.117:0/3216611845 shutdown_connections 2024-03-19T03:30:41.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.217+0000 7f66e3200640 1 -- 172.21.15.117:0/3216611845 wait complete. 2024-03-19T03:30:41.238 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:30:41.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.443+0000 7f6a29600640 1 -- 172.21.15.117:0/1136086417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6a24102a20 msgr2=0x7f6a24102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:41.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.443+0000 7f6a29600640 1 --2- 172.21.15.117:0/1136086417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6a24102a20 0x7f6a24102ea0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6a10009900 tx=0x7f6a1002bc30 comp rx=0 tx=0).stop 2024-03-19T03:30:41.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.443+0000 7f6a29600640 1 -- 172.21.15.117:0/1136086417 shutdown_connections 2024-03-19T03:30:41.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.443+0000 7f6a29600640 1 --2- 172.21.15.117:0/1136086417 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a241033e0 0x7f6a24109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:41.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.443+0000 7f6a29600640 1 --2- 172.21.15.117:0/1136086417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6a24102a20 0x7f6a24102ea0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:41.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.443+0000 7f6a29600640 1 --2- 172.21.15.117:0/1136086417 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a24101830 0x7f6a24101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:41.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.443+0000 7f6a29600640 1 -- 172.21.15.117:0/1136086417 >> 172.21.15.117:0/1136086417 conn(0x7f6a240fce00 msgr2=0x7f6a240ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:41.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.443+0000 7f6a29600640 1 -- 172.21.15.117:0/1136086417 shutdown_connections 2024-03-19T03:30:41.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.443+0000 7f6a29600640 1 -- 172.21.15.117:0/1136086417 wait complete. 2024-03-19T03:30:41.447 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.444+0000 7f6a29600640 1 Processor -- start 2024-03-19T03:30:41.447 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.444+0000 7f6a29600640 1 -- start start 2024-03-19T03:30:41.447 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.445+0000 7f6a29600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a24101830 0x7f6a24197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:41.447 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.445+0000 7f6a29600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a24102a20 0x7f6a241980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:41.447 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.445+0000 7f6a29600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6a241033e0 0x7f6a2419f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:41.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.445+0000 7f6a29600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6a2410d4b0 con 0x7f6a24101830 2024-03-19T03:30:41.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.445+0000 7f6a29600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6a2410d330 con 0x7f6a241033e0 2024-03-19T03:30:41.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.445+0000 7f6a29600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f6a2410d630 con 0x7f6a24102a20 2024-03-19T03:30:41.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.445+0000 7f6a22a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a24101830 0x7f6a24197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:41.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.445+0000 7f6a22a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a24101830 0x7f6a24197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39732/0 (socket says 172.21.15.117:39732) 2024-03-19T03:30:41.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.445+0000 7f6a23400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6a241033e0 0x7f6a2419f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:41.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.445+0000 7f6a22a00640 1 -- 172.21.15.117:0/4062759704 learned_addr learned my addr 172.21.15.117:0/4062759704 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:41.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.445+0000 7f6a22000640 1 --2- 172.21.15.117:0/4062759704 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a24102a20 0x7f6a241980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:41.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.446+0000 7f6a23400640 1 -- 172.21.15.117:0/4062759704 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a24102a20 msgr2=0x7f6a241980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:41.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.446+0000 7f6a23400640 1 --2- 172.21.15.117:0/4062759704 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a24102a20 0x7f6a241980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:41.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.446+0000 7f6a23400640 1 -- 172.21.15.117:0/4062759704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a24101830 msgr2=0x7f6a24197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:41.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.446+0000 7f6a23400640 1 --2- 172.21.15.117:0/4062759704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a24101830 0x7f6a24197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:41.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.446+0000 7f6a23400640 1 -- 172.21.15.117:0/4062759704 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a2419f7a0 con 0x7f6a241033e0 2024-03-19T03:30:41.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.446+0000 7f6a22a00640 1 --2- 172.21.15.117:0/4062759704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a24101830 0x7f6a24197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:30:41.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.446+0000 7f6a23400640 1 --2- 172.21.15.117:0/4062759704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6a241033e0 0x7f6a2419f130 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c00ec50 tx=0x7f6a1c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:41.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.446+0000 7f6a1b400640 1 -- 172.21.15.117:0/4062759704 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c003b70 con 0x7f6a241033e0 2024-03-19T03:30:41.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.446+0000 7f6a1b400640 1 -- 172.21.15.117:0/4062759704 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a1c003d10 con 0x7f6a241033e0 2024-03-19T03:30:41.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.446+0000 7f6a1b400640 1 -- 172.21.15.117:0/4062759704 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c010890 con 0x7f6a241033e0 2024-03-19T03:30:41.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.446+0000 7f6a29600640 1 -- 172.21.15.117:0/4062759704 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a2419fa90 con 0x7f6a241033e0 2024-03-19T03:30:41.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.446+0000 7f6a29600640 1 -- 172.21.15.117:0/4062759704 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a2419ff60 con 0x7f6a241033e0 2024-03-19T03:30:41.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.447+0000 7f6a29600640 1 -- 172.21.15.117:0/4062759704 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a24108ba0 con 0x7f6a241033e0 2024-03-19T03:30:41.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.449+0000 7f6a1b400640 1 -- 172.21.15.117:0/4062759704 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f6a1c003fc0 con 0x7f6a241033e0 2024-03-19T03:30:41.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.450+0000 7f6a1b400640 1 --2- 172.21.15.117:0/4062759704 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f6a080a2800 0x7f6a080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:41.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.451+0000 7f6a1b400640 1 -- 172.21.15.117:0/4062759704 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f6a1c0cf280 con 0x7f6a241033e0 2024-03-19T03:30:41.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.451+0000 7f6a22a00640 1 --2- 172.21.15.117:0/4062759704 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f6a080a2800 0x7f6a080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:41.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.451+0000 7f6a22a00640 1 --2- 172.21.15.117:0/4062759704 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f6a080a2800 0x7f6a080a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6a140058c0 tx=0x7f6a14005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:41.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.455+0000 7f6a1b400640 1 -- 172.21.15.117:0/4062759704 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6a1c09c080 con 0x7f6a241033e0 2024-03-19T03:30:41.657 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.656+0000 7f6a29600640 1 -- 172.21.15.117:0/4062759704 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6a24062420 con 0x7f6a241033e0 2024-03-19T03:30:41.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.658+0000 7f6a1b400640 1 -- 172.21.15.117:0/4062759704 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24210 (secure 0 0 0) 0x7f6a1c09b980 con 0x7f6a241033e0 2024-03-19T03:30:41.659 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:30:41.659 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":89,"fsid":"a3a82456-e59f-11ee-95c9-87774f69a715","created":"2024-03-19T03:21:30.427136+0000","modified":"2024-03-19T03:30:40.180745+0000","last_up_change":"2024-03-19T03:29:32.967505+0000","last_in_change":"2024-03-19T03:29:12.848740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:25:18.824270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T03:30:33.137170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T03:30:34.598838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T03:30:38.230023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b32eb0ee-9d4b-4748-87ec-eda02c907415","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6803","nonce":3014934900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6805","nonce":3014934900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6809","nonce":3014934900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6807","nonce":3014934900}]},"public_addr":"172.21.15.117:6803/3014934900","cluster_addr":"172.21.15.117:6805/3014934900","heartbeat_back_addr":"172.21.15.117:6809/3014934900","heartbeat_front_addr":"172.21.15.117:6807/3014934900","state":["exists","up"]},{"osd":1,"uuid":"e07585be-c016-4466-b07c-abbc22a58de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6801","nonce":2425927341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6803","nonce":2425927341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6807","nonce":2425927341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6805","nonce":2425927341}]},"public_addr":"172.21.15.119:6801/2425927341","cluster_addr":"172.21.15.119:6803/2425927341","heartbeat_back_addr":"172.21.15.119:6807/2425927341","heartbeat_front_addr":"172.21.15.119:6805/2425927341","state":["exists","up"]},{"osd":2,"uuid":"ede9edda-3b9f-41ca-ad04-5dcb738ab757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6801","nonce":3733812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6803","nonce":3733812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6807","nonce":3733812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6805","nonce":3733812989}]},"public_addr":"172.21.15.178:6801/3733812989","cluster_addr":"172.21.15.178:6803/3733812989","heartbeat_back_addr":"172.21.15.178:6807/3733812989","heartbeat_front_addr":"172.21.15.178:6805/3733812989","state":["exists","up"]},{"osd":3,"uuid":"f81c41e1-3a69-40db-a90f-58151286b081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6811","nonce":2178310135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6813","nonce":2178310135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6817","nonce":2178310135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6815","nonce":2178310135}]},"public_addr":"172.21.15.117:6811/2178310135","cluster_addr":"172.21.15.117:6813/2178310135","heartbeat_back_addr":"172.21.15.117:6817/2178310135","heartbeat_front_addr":"172.21.15.117:6815/2178310135","state":["exists","up"]},{"osd":4,"uuid":"caf7ce86-caf0-4eaf-bbab-bd2d93e63c33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6809","nonce":3455398086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6811","nonce":3455398086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6815","nonce":3455398086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6813","nonce":3455398086}]},"public_addr":"172.21.15.119:6809/3455398086","cluster_addr":"172.21.15.119:6811/3455398086","heartbeat_back_addr":"172.21.15.119:6815/3455398086","heartbeat_front_addr":"172.21.15.119:6813/3455398086","state":["exists","up"]},{"osd":5,"uuid":"20ba0b8a-2e0f-4ea7-b435-430e44766200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6809","nonce":1259631912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6811","nonce":1259631912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6815","nonce":1259631912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6813","nonce":1259631912}]},"public_addr":"172.21.15.178:6809/1259631912","cluster_addr":"172.21.15.178:6811/1259631912","heartbeat_back_addr":"172.21.15.178:6815/1259631912","heartbeat_front_addr":"172.21.15.178:6813/1259631912","state":["exists","up"]},{"osd":6,"uuid":"75c2ddd7-f837-42e0-ac47-8cfbcf97a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6819","nonce":864146757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6821","nonce":864146757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6825","nonce":864146757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6823","nonce":864146757}]},"public_addr":"172.21.15.117:6819/864146757","cluster_addr":"172.21.15.117:6821/864146757","heartbeat_back_addr":"172.21.15.117:6825/864146757","heartbeat_front_addr":"172.21.15.117:6823/864146757","state":["exists","up"]},{"osd":7,"uuid":"929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6817","nonce":1273365545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6819","nonce":1273365545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6823","nonce":1273365545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6821","nonce":1273365545}]},"public_addr":"172.21.15.119:6817/1273365545","cluster_addr":"172.21.15.119:6819/1273365545","heartbeat_back_addr":"172.21.15.119:6823/1273365545","heartbeat_front_addr":"172.21.15.119:6821/1273365545","state":["exists","up"]},{"osd":8,"uuid":"fd371461-a538-4781-92e1-5eead4ca3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6817","nonce":1059587501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6819","nonce":1059587501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6823","nonce":1059587501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6821","nonce":1059587501}]},"public_addr":"172.21.15.178:6817/1059587501","cluster_addr":"172.21.15.178:6819/1059587501","heartbeat_back_addr":"172.21.15.178:6823/1059587501","heartbeat_front_addr":"172.21.15.178:6821/1059587501","state":["exists","up"]},{"osd":9,"uuid":"0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6827","nonce":3653875600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6829","nonce":3653875600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6833","nonce":3653875600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6831","nonce":3653875600}]},"public_addr":"172.21.15.117:6827/3653875600","cluster_addr":"172.21.15.117:6829/3653875600","heartbeat_back_addr":"172.21.15.117:6833/3653875600","heartbeat_front_addr":"172.21.15.117:6831/3653875600","state":["exists","up"]},{"osd":10,"uuid":"de2049da-d19e-4386-afbf-fd6b5fddbcba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6825","nonce":1050976194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6827","nonce":1050976194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6831","nonce":1050976194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6829","nonce":1050976194}]},"public_addr":"172.21.15.119:6825/1050976194","cluster_addr":"172.21.15.119:6827/1050976194","heartbeat_back_addr":"172.21.15.119:6831/1050976194","heartbeat_front_addr":"172.21.15.119:6829/1050976194","state":["exists","up"]},{"osd":11,"uuid":"3cccc0f9-d273-4b6b-9b8e-c926371472dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6825","nonce":2915080368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6827","nonce":2915080368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6831","nonce":2915080368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6829","nonce":2915080368}]},"public_addr":"172.21.15.178:6825/2915080368","cluster_addr":"172.21.15.178:6827/2915080368","heartbeat_back_addr":"172.21.15.178:6831/2915080368","heartbeat_front_addr":"172.21.15.178:6829/2915080368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:27.876274+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:53.094706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:16.598813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:48.160419+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:14.048925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:38.814304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:12.542328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:38.942747+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:04.488963+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:38.196053+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:05.254728+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:30.874915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.117:0/185684447":"2024-03-20T03:22:38.134578+0000","172.21.15.117:0/361367101":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/2788067603":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3506200675":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6800/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3283471541":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3750920169":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/2780813298":"2024-03-20T03:22:03.905461+0000","172.21.15.117:6800/2788067603":"2024-03-20T03:22:38.134578+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:30:41.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.661+0000 7f6a29600640 1 -- 172.21.15.117:0/4062759704 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f6a080a2800 msgr2=0x7f6a080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:41.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.661+0000 7f6a29600640 1 --2- 172.21.15.117:0/4062759704 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f6a080a2800 0x7f6a080a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6a140058c0 tx=0x7f6a14005f90 comp rx=0 tx=0).stop 2024-03-19T03:30:41.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.661+0000 7f6a29600640 1 -- 172.21.15.117:0/4062759704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6a241033e0 msgr2=0x7f6a2419f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:41.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.661+0000 7f6a29600640 1 --2- 172.21.15.117:0/4062759704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6a241033e0 0x7f6a2419f130 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c00ec50 tx=0x7f6a1c00c6a0 comp rx=0 tx=0).stop 2024-03-19T03:30:41.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.661+0000 7f6a29600640 1 -- 172.21.15.117:0/4062759704 shutdown_connections 2024-03-19T03:30:41.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.661+0000 7f6a29600640 1 --2- 172.21.15.117:0/4062759704 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f6a080a2800 0x7f6a080a4cc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:41.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.661+0000 7f6a29600640 1 --2- 172.21.15.117:0/4062759704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6a241033e0 0x7f6a2419f130 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:41.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.661+0000 7f6a29600640 1 --2- 172.21.15.117:0/4062759704 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a24102a20 0x7f6a241980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:41.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.661+0000 7f6a29600640 1 --2- 172.21.15.117:0/4062759704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a24101830 0x7f6a24197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:41.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.661+0000 7f6a29600640 1 -- 172.21.15.117:0/4062759704 >> 172.21.15.117:0/4062759704 conn(0x7f6a240fce00 msgr2=0x7f6a24105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:41.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.661+0000 7f6a29600640 1 -- 172.21.15.117:0/4062759704 shutdown_connections 2024-03-19T03:30:41.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.661+0000 7f6a29600640 1 -- 172.21.15.117:0/4062759704 wait complete. 2024-03-19T03:30:41.682 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:30:41.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.896+0000 7fcab5600640 1 -- 172.21.15.117:0/1489543506 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcab0102b00 msgr2=0x7fcab0102f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:41.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.896+0000 7fcab5600640 1 --2- 172.21.15.117:0/1489543506 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcab0102b00 0x7fcab0102f80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fca98009900 tx=0x7fca9802bc30 comp rx=0 tx=0).stop 2024-03-19T03:30:41.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.896+0000 7fcab5600640 1 -- 172.21.15.117:0/1489543506 shutdown_connections 2024-03-19T03:30:41.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.896+0000 7fcab5600640 1 --2- 172.21.15.117:0/1489543506 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcab01034c0 0x7fcab0109d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:41.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.896+0000 7fcab5600640 1 --2- 172.21.15.117:0/1489543506 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcab0102b00 0x7fcab0102f80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:41.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.896+0000 7fcab5600640 1 --2- 172.21.15.117:0/1489543506 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcab00ffb30 0x7fcab00fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:41.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.896+0000 7fcab5600640 1 -- 172.21.15.117:0/1489543506 >> 172.21.15.117:0/1489543506 conn(0x7fcab00fb460 msgr2=0x7fcab00fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:41.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.896+0000 7fcab5600640 1 -- 172.21.15.117:0/1489543506 shutdown_connections 2024-03-19T03:30:41.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.896+0000 7fcab5600640 1 -- 172.21.15.117:0/1489543506 wait complete. 2024-03-19T03:30:41.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.897+0000 7fcab5600640 1 Processor -- start 2024-03-19T03:30:41.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.897+0000 7fcab5600640 1 -- start start 2024-03-19T03:30:41.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.898+0000 7fcab5600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcab00ffb30 0x7fcab0199de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:41.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.898+0000 7fcab5600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcab0102b00 0x7fcab019a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:41.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.898+0000 7fcab5600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcab01034c0 0x7fcab01a1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:41.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.898+0000 7fcab5600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcab010d510 con 0x7fcab00ffb30 2024-03-19T03:30:41.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.898+0000 7fcab5600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fcab010d390 con 0x7fcab0102b00 2024-03-19T03:30:41.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.898+0000 7fcab5600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fcab010d690 con 0x7fcab01034c0 2024-03-19T03:30:41.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.898+0000 7fcaa6000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcab0102b00 0x7fcab019a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:41.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.898+0000 7fcaaea00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcab00ffb30 0x7fcab0199de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:41.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.898+0000 7fcaa6000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcab0102b00 0x7fcab019a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57624/0 (socket says 172.21.15.117:57624) 2024-03-19T03:30:41.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.898+0000 7fcaa6000640 1 -- 172.21.15.117:0/517581102 learned_addr learned my addr 172.21.15.117:0/517581102 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:41.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.898+0000 7fcaaf400640 1 --2- 172.21.15.117:0/517581102 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcab01034c0 0x7fcab01a1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:41.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.898+0000 7fcaa6000640 1 -- 172.21.15.117:0/517581102 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcab01034c0 msgr2=0x7fcab01a1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:41.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.898+0000 7fcaa6000640 1 --2- 172.21.15.117:0/517581102 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcab01034c0 0x7fcab01a1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:41.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.898+0000 7fcaa6000640 1 -- 172.21.15.117:0/517581102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcab00ffb30 msgr2=0x7fcab0199de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:41.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.898+0000 7fcaa6000640 1 --2- 172.21.15.117:0/517581102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcab00ffb30 0x7fcab0199de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:41.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.898+0000 7fcaa6000640 1 -- 172.21.15.117:0/517581102 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcab01a1a00 con 0x7fcab0102b00 2024-03-19T03:30:41.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.899+0000 7fcaa6000640 1 --2- 172.21.15.117:0/517581102 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcab0102b00 0x7fcab019a320 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fca98005ab0 tx=0x7fca9802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:41.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.899+0000 7fcaa7e00640 1 -- 172.21.15.117:0/517581102 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca98043020 con 0x7fcab0102b00 2024-03-19T03:30:41.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.899+0000 7fcaa7e00640 1 -- 172.21.15.117:0/517581102 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca980054e0 con 0x7fcab0102b00 2024-03-19T03:30:41.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.899+0000 7fcaa7e00640 1 -- 172.21.15.117:0/517581102 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca98003bc0 con 0x7fcab0102b00 2024-03-19T03:30:41.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.899+0000 7fcab5600640 1 -- 172.21.15.117:0/517581102 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcab01a1c90 con 0x7fcab0102b00 2024-03-19T03:30:41.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.899+0000 7fcab5600640 1 -- 172.21.15.117:0/517581102 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcab01a2190 con 0x7fcab0102b00 2024-03-19T03:30:41.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.900+0000 7fcab5600640 1 -- 172.21.15.117:0/517581102 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcab01a2360 con 0x7fcab0102b00 2024-03-19T03:30:41.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.902+0000 7fcaa7e00640 1 -- 172.21.15.117:0/517581102 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fca98046020 con 0x7fcab0102b00 2024-03-19T03:30:41.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.903+0000 7fcaa7e00640 1 --2- 172.21.15.117:0/517581102 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fca940a2880 0x7fca940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:41.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.903+0000 7fcaa7e00640 1 -- 172.21.15.117:0/517581102 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fca980f0d80 con 0x7fcab0102b00 2024-03-19T03:30:41.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.903+0000 7fcaaea00640 1 --2- 172.21.15.117:0/517581102 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fca940a2880 0x7fca940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:41.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.904+0000 7fcaaea00640 1 --2- 172.21.15.117:0/517581102 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fca940a2880 0x7fca940a4d40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcab0069c90 tx=0x7fcaa0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:41.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:41.907+0000 7fcaa7e00640 1 -- 172.21.15.117:0/517581102 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fca9804d020 con 0x7fcab0102b00 2024-03-19T03:30:41.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:41 smithi119 ceph-mon[66861]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T03:30:41.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:41 smithi119 ceph-mon[66861]: osdmap e89: 12 total, 12 up, 12 in 2024-03-19T03:30:41.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:41 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-19T03:30:41.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:41 smithi119 ceph-mon[66861]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T03:30:41.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:41 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3216611845' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:41.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:41 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/4062759704' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:30:42.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:41 smithi117 ceph-mon[31838]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T03:30:42.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:41 smithi117 ceph-mon[31838]: osdmap e89: 12 total, 12 up, 12 in 2024-03-19T03:30:42.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:41 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-19T03:30:42.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:41 smithi117 ceph-mon[31838]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T03:30:42.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:41 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3216611845' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:42.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:41 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/4062759704' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:30:42.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:41 smithi178 ceph-mon[67578]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T03:30:42.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:41 smithi178 ceph-mon[67578]: osdmap e89: 12 total, 12 up, 12 in 2024-03-19T03:30:42.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:41 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-19T03:30:42.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:41 smithi178 ceph-mon[67578]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T03:30:42.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:41 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3216611845' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:42.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:41 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/4062759704' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:30:42.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.106+0000 7fcab5600640 1 -- 172.21.15.117:0/517581102 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcab01a2a30 con 0x7fcab0102b00 2024-03-19T03:30:42.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.109+0000 7fcaa7e00640 1 -- 172.21.15.117:0/517581102 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24210 (secure 0 0 0) 0x7fca980bdbe0 con 0x7fcab0102b00 2024-03-19T03:30:42.109 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:30:42.110 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":89,"fsid":"a3a82456-e59f-11ee-95c9-87774f69a715","created":"2024-03-19T03:21:30.427136+0000","modified":"2024-03-19T03:30:40.180745+0000","last_up_change":"2024-03-19T03:29:32.967505+0000","last_in_change":"2024-03-19T03:29:12.848740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:25:18.824270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T03:30:33.137170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T03:30:34.598838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T03:30:38.230023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b32eb0ee-9d4b-4748-87ec-eda02c907415","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6803","nonce":3014934900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6805","nonce":3014934900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6809","nonce":3014934900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6807","nonce":3014934900}]},"public_addr":"172.21.15.117:6803/3014934900","cluster_addr":"172.21.15.117:6805/3014934900","heartbeat_back_addr":"172.21.15.117:6809/3014934900","heartbeat_front_addr":"172.21.15.117:6807/3014934900","state":["exists","up"]},{"osd":1,"uuid":"e07585be-c016-4466-b07c-abbc22a58de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6801","nonce":2425927341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6803","nonce":2425927341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6807","nonce":2425927341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6805","nonce":2425927341}]},"public_addr":"172.21.15.119:6801/2425927341","cluster_addr":"172.21.15.119:6803/2425927341","heartbeat_back_addr":"172.21.15.119:6807/2425927341","heartbeat_front_addr":"172.21.15.119:6805/2425927341","state":["exists","up"]},{"osd":2,"uuid":"ede9edda-3b9f-41ca-ad04-5dcb738ab757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6801","nonce":3733812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6803","nonce":3733812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6807","nonce":3733812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6805","nonce":3733812989}]},"public_addr":"172.21.15.178:6801/3733812989","cluster_addr":"172.21.15.178:6803/3733812989","heartbeat_back_addr":"172.21.15.178:6807/3733812989","heartbeat_front_addr":"172.21.15.178:6805/3733812989","state":["exists","up"]},{"osd":3,"uuid":"f81c41e1-3a69-40db-a90f-58151286b081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6811","nonce":2178310135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6813","nonce":2178310135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6817","nonce":2178310135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6815","nonce":2178310135}]},"public_addr":"172.21.15.117:6811/2178310135","cluster_addr":"172.21.15.117:6813/2178310135","heartbeat_back_addr":"172.21.15.117:6817/2178310135","heartbeat_front_addr":"172.21.15.117:6815/2178310135","state":["exists","up"]},{"osd":4,"uuid":"caf7ce86-caf0-4eaf-bbab-bd2d93e63c33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6809","nonce":3455398086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6811","nonce":3455398086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6815","nonce":3455398086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6813","nonce":3455398086}]},"public_addr":"172.21.15.119:6809/3455398086","cluster_addr":"172.21.15.119:6811/3455398086","heartbeat_back_addr":"172.21.15.119:6815/3455398086","heartbeat_front_addr":"172.21.15.119:6813/3455398086","state":["exists","up"]},{"osd":5,"uuid":"20ba0b8a-2e0f-4ea7-b435-430e44766200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6809","nonce":1259631912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6811","nonce":1259631912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6815","nonce":1259631912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6813","nonce":1259631912}]},"public_addr":"172.21.15.178:6809/1259631912","cluster_addr":"172.21.15.178:6811/1259631912","heartbeat_back_addr":"172.21.15.178:6815/1259631912","heartbeat_front_addr":"172.21.15.178:6813/1259631912","state":["exists","up"]},{"osd":6,"uuid":"75c2ddd7-f837-42e0-ac47-8cfbcf97a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6819","nonce":864146757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6821","nonce":864146757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6825","nonce":864146757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6823","nonce":864146757}]},"public_addr":"172.21.15.117:6819/864146757","cluster_addr":"172.21.15.117:6821/864146757","heartbeat_back_addr":"172.21.15.117:6825/864146757","heartbeat_front_addr":"172.21.15.117:6823/864146757","state":["exists","up"]},{"osd":7,"uuid":"929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6817","nonce":1273365545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6819","nonce":1273365545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6823","nonce":1273365545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6821","nonce":1273365545}]},"public_addr":"172.21.15.119:6817/1273365545","cluster_addr":"172.21.15.119:6819/1273365545","heartbeat_back_addr":"172.21.15.119:6823/1273365545","heartbeat_front_addr":"172.21.15.119:6821/1273365545","state":["exists","up"]},{"osd":8,"uuid":"fd371461-a538-4781-92e1-5eead4ca3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6817","nonce":1059587501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6819","nonce":1059587501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6823","nonce":1059587501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6821","nonce":1059587501}]},"public_addr":"172.21.15.178:6817/1059587501","cluster_addr":"172.21.15.178:6819/1059587501","heartbeat_back_addr":"172.21.15.178:6823/1059587501","heartbeat_front_addr":"172.21.15.178:6821/1059587501","state":["exists","up"]},{"osd":9,"uuid":"0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6827","nonce":3653875600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6829","nonce":3653875600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6833","nonce":3653875600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6831","nonce":3653875600}]},"public_addr":"172.21.15.117:6827/3653875600","cluster_addr":"172.21.15.117:6829/3653875600","heartbeat_back_addr":"172.21.15.117:6833/3653875600","heartbeat_front_addr":"172.21.15.117:6831/3653875600","state":["exists","up"]},{"osd":10,"uuid":"de2049da-d19e-4386-afbf-fd6b5fddbcba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6825","nonce":1050976194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6827","nonce":1050976194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6831","nonce":1050976194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6829","nonce":1050976194}]},"public_addr":"172.21.15.119:6825/1050976194","cluster_addr":"172.21.15.119:6827/1050976194","heartbeat_back_addr":"172.21.15.119:6831/1050976194","heartbeat_front_addr":"172.21.15.119:6829/1050976194","state":["exists","up"]},{"osd":11,"uuid":"3cccc0f9-d273-4b6b-9b8e-c926371472dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6825","nonce":2915080368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6827","nonce":2915080368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6831","nonce":2915080368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6829","nonce":2915080368}]},"public_addr":"172.21.15.178:6825/2915080368","cluster_addr":"172.21.15.178:6827/2915080368","heartbeat_back_addr":"172.21.15.178:6831/2915080368","heartbeat_front_addr":"172.21.15.178:6829/2915080368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:27.876274+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:53.094706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:16.598813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:48.160419+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:14.048925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:38.814304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:12.542328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:38.942747+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:04.488963+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:38.196053+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:05.254728+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:30.874915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.117:0/185684447":"2024-03-20T03:22:38.134578+0000","172.21.15.117:0/361367101":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/2788067603":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3506200675":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6800/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3283471541":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3750920169":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/2780813298":"2024-03-20T03:22:03.905461+0000","172.21.15.117:6800/2788067603":"2024-03-20T03:22:38.134578+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:30:42.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.111+0000 7fcab5600640 1 -- 172.21.15.117:0/517581102 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fca940a2880 msgr2=0x7fca940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:42.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.111+0000 7fcab5600640 1 --2- 172.21.15.117:0/517581102 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fca940a2880 0x7fca940a4d40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcab0069c90 tx=0x7fcaa0009500 comp rx=0 tx=0).stop 2024-03-19T03:30:42.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.111+0000 7fcab5600640 1 -- 172.21.15.117:0/517581102 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcab0102b00 msgr2=0x7fcab019a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:42.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.111+0000 7fcab5600640 1 --2- 172.21.15.117:0/517581102 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcab0102b00 0x7fcab019a320 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fca98005ab0 tx=0x7fca9802c650 comp rx=0 tx=0).stop 2024-03-19T03:30:42.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.111+0000 7fcab5600640 1 -- 172.21.15.117:0/517581102 shutdown_connections 2024-03-19T03:30:42.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.111+0000 7fcab5600640 1 --2- 172.21.15.117:0/517581102 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcab01034c0 0x7fcab01a1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:42.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.111+0000 7fcab5600640 1 --2- 172.21.15.117:0/517581102 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fca940a2880 0x7fca940a4d40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:42.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.111+0000 7fcab5600640 1 --2- 172.21.15.117:0/517581102 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcab0102b00 0x7fcab019a320 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:42.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.111+0000 7fcab5600640 1 --2- 172.21.15.117:0/517581102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcab00ffb30 0x7fcab0199de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:42.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.111+0000 7fcab5600640 1 -- 172.21.15.117:0/517581102 >> 172.21.15.117:0/517581102 conn(0x7fcab00fb460 msgr2=0x7fcab01008f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:42.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.111+0000 7fcab5600640 1 -- 172.21.15.117:0/517581102 shutdown_connections 2024-03-19T03:30:42.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.111+0000 7fcab5600640 1 -- 172.21.15.117:0/517581102 wait complete. 2024-03-19T03:30:42.144 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:30:42.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.356+0000 7eff3bc00640 1 -- 172.21.15.117:0/1249770301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff34105610 msgr2=0x7eff3410bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:42.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.356+0000 7eff3bc00640 1 --2- 172.21.15.117:0/1249770301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff34105610 0x7eff3410bea0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7eff2c009ca0 tx=0x7eff2c02c100 comp rx=0 tx=0).stop 2024-03-19T03:30:42.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.356+0000 7eff3bc00640 1 -- 172.21.15.117:0/1249770301 shutdown_connections 2024-03-19T03:30:42.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.356+0000 7eff3bc00640 1 --2- 172.21.15.117:0/1249770301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff34105610 0x7eff3410bea0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:42.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.356+0000 7eff3bc00640 1 --2- 172.21.15.117:0/1249770301 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff34104c50 0x7eff341050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:42.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.356+0000 7eff3bc00640 1 --2- 172.21.15.117:0/1249770301 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7eff34103a60 0x7eff34103e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:42.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.356+0000 7eff3bc00640 1 -- 172.21.15.117:0/1249770301 >> 172.21.15.117:0/1249770301 conn(0x7eff340fefd0 msgr2=0x7eff34101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:42.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.356+0000 7eff3bc00640 1 -- 172.21.15.117:0/1249770301 shutdown_connections 2024-03-19T03:30:42.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.356+0000 7eff3bc00640 1 -- 172.21.15.117:0/1249770301 wait complete. 2024-03-19T03:30:42.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.357+0000 7eff3bc00640 1 Processor -- start 2024-03-19T03:30:42.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.358+0000 7eff3bc00640 1 -- start start 2024-03-19T03:30:42.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.358+0000 7eff3bc00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff34103a60 0x7eff34199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:42.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.358+0000 7eff3bc00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff34104c50 0x7eff3419a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:42.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.358+0000 7eff3bc00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7eff34105610 0x7eff341a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:42.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.358+0000 7eff3bc00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7eff3410f6c0 con 0x7eff34103a60 2024-03-19T03:30:42.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.358+0000 7eff3bc00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7eff3410f540 con 0x7eff34104c50 2024-03-19T03:30:42.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.358+0000 7eff3bc00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7eff3410f840 con 0x7eff34105610 2024-03-19T03:30:42.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.358+0000 7eff39200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff34103a60 0x7eff34199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:42.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.358+0000 7eff33e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff34104c50 0x7eff3419a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:42.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.358+0000 7eff39200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff34103a60 0x7eff34199c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39750/0 (socket says 172.21.15.117:39750) 2024-03-19T03:30:42.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.358+0000 7eff39200640 1 -- 172.21.15.117:0/2579736331 learned_addr learned my addr 172.21.15.117:0/2579736331 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:42.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.358+0000 7eff39c00640 1 --2- 172.21.15.117:0/2579736331 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7eff34105610 0x7eff341a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:42.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.359+0000 7eff33e00640 1 -- 172.21.15.117:0/2579736331 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7eff34105610 msgr2=0x7eff341a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:42.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.359+0000 7eff33e00640 1 --2- 172.21.15.117:0/2579736331 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7eff34105610 0x7eff341a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:42.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.359+0000 7eff33e00640 1 -- 172.21.15.117:0/2579736331 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff34103a60 msgr2=0x7eff34199c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:42.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.359+0000 7eff33e00640 1 --2- 172.21.15.117:0/2579736331 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff34103a60 0x7eff34199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:42.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.359+0000 7eff33e00640 1 -- 172.21.15.117:0/2579736331 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff341a1870 con 0x7eff34104c50 2024-03-19T03:30:42.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.359+0000 7eff33e00640 1 --2- 172.21.15.117:0/2579736331 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff34104c50 0x7eff3419a1a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7eff2000e9f0 tx=0x7eff2000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:42.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.359+0000 7eff31600640 1 -- 172.21.15.117:0/2579736331 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2000ccc0 con 0x7eff34104c50 2024-03-19T03:30:42.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.359+0000 7eff31600640 1 -- 172.21.15.117:0/2579736331 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff20010910 con 0x7eff34104c50 2024-03-19T03:30:42.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.359+0000 7eff3bc00640 1 -- 172.21.15.117:0/2579736331 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff341a1b60 con 0x7eff34104c50 2024-03-19T03:30:42.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.359+0000 7eff31600640 1 -- 172.21.15.117:0/2579736331 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff20018960 con 0x7eff34104c50 2024-03-19T03:30:42.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.359+0000 7eff3bc00640 1 -- 172.21.15.117:0/2579736331 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff341a20b0 con 0x7eff34104c50 2024-03-19T03:30:42.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.360+0000 7eff3bc00640 1 -- 172.21.15.117:0/2579736331 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff3410acb0 con 0x7eff34104c50 2024-03-19T03:30:42.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.362+0000 7eff31600640 1 -- 172.21.15.117:0/2579736331 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7eff20010ac0 con 0x7eff34104c50 2024-03-19T03:30:42.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.364+0000 7eff31600640 1 --2- 172.21.15.117:0/2579736331 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7eff180a2800 0x7eff180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:42.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.364+0000 7eff31600640 1 -- 172.21.15.117:0/2579736331 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7eff200cf120 con 0x7eff34104c50 2024-03-19T03:30:42.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.364+0000 7eff39200640 1 --2- 172.21.15.117:0/2579736331 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7eff180a2800 0x7eff180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:42.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.364+0000 7eff39200640 1 --2- 172.21.15.117:0/2579736331 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7eff180a2800 0x7eff180a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7eff3419b130 tx=0x7eff2400a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:42.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.366+0000 7eff31600640 1 -- 172.21.15.117:0/2579736331 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7eff2009b8d0 con 0x7eff34104c50 2024-03-19T03:30:42.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.602+0000 7eff3bc00640 1 -- 172.21.15.117:0/2579736331 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff3402d290 con 0x7eff34104c50 2024-03-19T03:30:42.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.603+0000 7eff31600640 1 -- 172.21.15.117:0/2579736331 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11818 (secure 0 0 0) 0x7eff2009f970 con 0x7eff34104c50 2024-03-19T03:30:42.605 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:30:42.605 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/54638259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":54638259},{"type":"v1","addr":"172.21.15.117:6835","nonce":54638259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3608185039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3608185039},{"type":"v1","addr":"172.21.15.117:6837","nonce":3608185039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/1291833214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1291833214},{"type":"v1","addr":"172.21.15.119:6833","nonce":1291833214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/1579860804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1579860804},{"type":"v1","addr":"172.21.15.178:6835","nonce":1579860804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24515,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/1686938941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1686938941},{"type":"v1","addr":"172.21.15.119:6835","nonce":1686938941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1770913538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":1770913538},{"type":"v1","addr":"172.21.15.178:6837","nonce":1770913538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6839/2655593383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2655593383},{"type":"v1","addr":"172.21.15.178:6839","nonce":2655593383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24545,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6837/2054992738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2054992738},{"type":"v1","addr":"172.21.15.119:6837","nonce":2054992738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24569,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6839/684335078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":684335078},{"type":"v1","addr":"172.21.15.117:6839","nonce":684335078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24581,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6841/1329933892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6840","nonce":1329933892},{"type":"v1","addr":"172.21.15.117:6841","nonce":1329933892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-19T03:30:35.081740+0000","modified":"2024-03-19T03:30:40.181111+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/2383354833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2383354833},{"type":"v1","addr":"172.21.15.119:6839","nonce":2383354833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24599,"qdb_cluster":[24599]},"id":1}]} 2024-03-19T03:30:42.605 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 17 2024-03-19T03:30:42.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.607+0000 7eff3bc00640 1 -- 172.21.15.117:0/2579736331 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7eff180a2800 msgr2=0x7eff180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:42.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.607+0000 7eff3bc00640 1 --2- 172.21.15.117:0/2579736331 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7eff180a2800 0x7eff180a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7eff3419b130 tx=0x7eff2400a3b0 comp rx=0 tx=0).stop 2024-03-19T03:30:42.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.607+0000 7eff3bc00640 1 -- 172.21.15.117:0/2579736331 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff34104c50 msgr2=0x7eff3419a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:42.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.607+0000 7eff3bc00640 1 --2- 172.21.15.117:0/2579736331 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff34104c50 0x7eff3419a1a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7eff2000e9f0 tx=0x7eff2000eeb0 comp rx=0 tx=0).stop 2024-03-19T03:30:42.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.607+0000 7eff3bc00640 1 -- 172.21.15.117:0/2579736331 shutdown_connections 2024-03-19T03:30:42.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.607+0000 7eff3bc00640 1 --2- 172.21.15.117:0/2579736331 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7eff34105610 0x7eff341a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:42.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.607+0000 7eff3bc00640 1 --2- 172.21.15.117:0/2579736331 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7eff180a2800 0x7eff180a4cc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:42.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.607+0000 7eff3bc00640 1 --2- 172.21.15.117:0/2579736331 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff34104c50 0x7eff3419a1a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:42.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.607+0000 7eff3bc00640 1 --2- 172.21.15.117:0/2579736331 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff34103a60 0x7eff34199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:42.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.607+0000 7eff3bc00640 1 -- 172.21.15.117:0/2579736331 >> 172.21.15.117:0/2579736331 conn(0x7eff340fefd0 msgr2=0x7eff34107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:42.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.607+0000 7eff3bc00640 1 -- 172.21.15.117:0/2579736331 shutdown_connections 2024-03-19T03:30:42.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.607+0000 7eff3bc00640 1 -- 172.21.15.117:0/2579736331 wait complete. 2024-03-19T03:30:42.628 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:30:42.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.827+0000 7f48f4400640 1 -- 172.21.15.117:0/3080775835 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48ec107e60 msgr2=0x7f48ec108280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:42.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.827+0000 7f48f4400640 1 --2- 172.21.15.117:0/3080775835 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48ec107e60 0x7f48ec108280 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f48e0009900 tx=0x7f48e002bc30 comp rx=0 tx=0).stop 2024-03-19T03:30:42.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.827+0000 7f48f4400640 1 -- 172.21.15.117:0/3080775835 shutdown_connections 2024-03-19T03:30:42.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.827+0000 7f48f4400640 1 --2- 172.21.15.117:0/3080775835 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f48ec109970 0x7f48ec110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:42.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.827+0000 7f48f4400640 1 --2- 172.21.15.117:0/3080775835 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48ec108fb0 0x7f48ec109430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:42.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.827+0000 7f48f4400640 1 --2- 172.21.15.117:0/3080775835 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48ec107e60 0x7f48ec108280 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:42.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.827+0000 7f48f4400640 1 -- 172.21.15.117:0/3080775835 >> 172.21.15.117:0/3080775835 conn(0x7f48ec076e30 msgr2=0x7f48ec079250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:42.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.827+0000 7f48f4400640 1 -- 172.21.15.117:0/3080775835 shutdown_connections 2024-03-19T03:30:42.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.827+0000 7f48f4400640 1 -- 172.21.15.117:0/3080775835 wait complete. 2024-03-19T03:30:42.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.827+0000 7f48f4400640 1 Processor -- start 2024-03-19T03:30:42.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.827+0000 7f48f4400640 1 -- start start 2024-03-19T03:30:42.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.827+0000 7f48f4400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48ec107e60 0x7f48ec19e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:42.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.827+0000 7f48f4400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f48ec108fb0 0x7f48ec19e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:42.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.827+0000 7f48f4400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48ec109970 0x7f48ec1a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:42.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.827+0000 7f48f4400640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f48ec113b30 con 0x7f48ec107e60 2024-03-19T03:30:42.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.827+0000 7f48f4400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f48ec1139b0 con 0x7f48ec109970 2024-03-19T03:30:42.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.827+0000 7f48f4400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f48ec113cb0 con 0x7f48ec108fb0 2024-03-19T03:30:42.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.828+0000 7f48f1a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48ec107e60 0x7f48ec19e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:42.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.828+0000 7f48f2400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48ec109970 0x7f48ec1a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:42.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.828+0000 7f48f1a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48ec107e60 0x7f48ec19e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39770/0 (socket says 172.21.15.117:39770) 2024-03-19T03:30:42.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.828+0000 7f48f2400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48ec109970 0x7f48ec1a56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57670/0 (socket says 172.21.15.117:57670) 2024-03-19T03:30:42.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.828+0000 7f48f1a00640 1 -- 172.21.15.117:0/3993877853 learned_addr learned my addr 172.21.15.117:0/3993877853 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:42.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.828+0000 7f48f1000640 1 --2- 172.21.15.117:0/3993877853 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f48ec108fb0 0x7f48ec19e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:42.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.828+0000 7f48f1a00640 1 -- 172.21.15.117:0/3993877853 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f48ec108fb0 msgr2=0x7f48ec19e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:42.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.828+0000 7f48f1a00640 1 --2- 172.21.15.117:0/3993877853 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f48ec108fb0 0x7f48ec19e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:42.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.828+0000 7f48f1a00640 1 -- 172.21.15.117:0/3993877853 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48ec109970 msgr2=0x7f48ec1a56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:42.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.828+0000 7f48f1a00640 1 --2- 172.21.15.117:0/3993877853 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48ec109970 0x7f48ec1a56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:42.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.828+0000 7f48f1a00640 1 -- 172.21.15.117:0/3993877853 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48ec1a5d60 con 0x7f48ec107e60 2024-03-19T03:30:42.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.828+0000 7f48f1000640 1 --2- 172.21.15.117:0/3993877853 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f48ec108fb0 0x7f48ec19e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:30:42.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.828+0000 7f48f2400640 1 --2- 172.21.15.117:0/3993877853 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48ec109970 0x7f48ec1a56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:30:42.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.828+0000 7f48f1a00640 1 --2- 172.21.15.117:0/3993877853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48ec107e60 0x7f48ec19e140 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f48e0005ab0 tx=0x7f48e002c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:42.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.829+0000 7f48ea000640 1 -- 172.21.15.117:0/3993877853 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e0043020 con 0x7f48ec107e60 2024-03-19T03:30:42.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.829+0000 7f48ea000640 1 -- 172.21.15.117:0/3993877853 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48e00054e0 con 0x7f48ec107e60 2024-03-19T03:30:42.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.829+0000 7f48f4400640 1 -- 172.21.15.117:0/3993877853 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48ec1a5ff0 con 0x7f48ec107e60 2024-03-19T03:30:42.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.829+0000 7f48ea000640 1 -- 172.21.15.117:0/3993877853 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e0003ce0 con 0x7f48ec107e60 2024-03-19T03:30:42.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.829+0000 7f48f4400640 1 -- 172.21.15.117:0/3993877853 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48ec1a6450 con 0x7f48ec107e60 2024-03-19T03:30:42.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.830+0000 7f48f4400640 1 -- 172.21.15.117:0/3993877853 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48ec10f110 con 0x7f48ec107e60 2024-03-19T03:30:42.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.833+0000 7f48ea000640 1 -- 172.21.15.117:0/3993877853 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f48e0046020 con 0x7f48ec107e60 2024-03-19T03:30:42.834 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.833+0000 7f48ea000640 1 --2- 172.21.15.117:0/3993877853 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f48d40a2800 0x7f48d40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:42.834 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.833+0000 7f48ea000640 1 -- 172.21.15.117:0/3993877853 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f48e00f1270 con 0x7f48ec107e60 2024-03-19T03:30:42.834 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.834+0000 7f48f1000640 1 --2- 172.21.15.117:0/3993877853 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f48d40a2800 0x7f48d40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:42.835 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.834+0000 7f48f1000640 1 --2- 172.21.15.117:0/3993877853 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f48d40a2800 0x7f48d40a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f48ec19f650 tx=0x7f48d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:42.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:42.836+0000 7f48ea000640 1 -- 172.21.15.117:0/3993877853 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f48e004d020 con 0x7f48ec107e60 2024-03-19T03:30:42.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:42 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/517581102' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:30:42.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:42 smithi119 ceph-mon[66861]: pgmap v321: 129 pgs: 6 creating+peering, 120 active+clean, 3 unknown; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 2 op/s 2024-03-19T03:30:42.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:42 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2579736331' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:43.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.035+0000 7f48f4400640 1 -- 172.21.15.117:0/3993877853 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f48ec02d3c0 con 0x7f48ec107e60 2024-03-19T03:30:43.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.038+0000 7f48ea000640 1 -- 172.21.15.117:0/3993877853 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24210 (secure 0 0 0) 0x7f48e00be070 con 0x7f48ec107e60 2024-03-19T03:30:43.039 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:30:43.039 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":89,"fsid":"a3a82456-e59f-11ee-95c9-87774f69a715","created":"2024-03-19T03:21:30.427136+0000","modified":"2024-03-19T03:30:40.180745+0000","last_up_change":"2024-03-19T03:29:32.967505+0000","last_in_change":"2024-03-19T03:29:12.848740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:25:18.824270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T03:30:33.137170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T03:30:34.598838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T03:30:38.230023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b32eb0ee-9d4b-4748-87ec-eda02c907415","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6803","nonce":3014934900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6805","nonce":3014934900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6809","nonce":3014934900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6807","nonce":3014934900}]},"public_addr":"172.21.15.117:6803/3014934900","cluster_addr":"172.21.15.117:6805/3014934900","heartbeat_back_addr":"172.21.15.117:6809/3014934900","heartbeat_front_addr":"172.21.15.117:6807/3014934900","state":["exists","up"]},{"osd":1,"uuid":"e07585be-c016-4466-b07c-abbc22a58de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6801","nonce":2425927341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6803","nonce":2425927341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6807","nonce":2425927341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6805","nonce":2425927341}]},"public_addr":"172.21.15.119:6801/2425927341","cluster_addr":"172.21.15.119:6803/2425927341","heartbeat_back_addr":"172.21.15.119:6807/2425927341","heartbeat_front_addr":"172.21.15.119:6805/2425927341","state":["exists","up"]},{"osd":2,"uuid":"ede9edda-3b9f-41ca-ad04-5dcb738ab757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6801","nonce":3733812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6803","nonce":3733812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6807","nonce":3733812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6805","nonce":3733812989}]},"public_addr":"172.21.15.178:6801/3733812989","cluster_addr":"172.21.15.178:6803/3733812989","heartbeat_back_addr":"172.21.15.178:6807/3733812989","heartbeat_front_addr":"172.21.15.178:6805/3733812989","state":["exists","up"]},{"osd":3,"uuid":"f81c41e1-3a69-40db-a90f-58151286b081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6811","nonce":2178310135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6813","nonce":2178310135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6817","nonce":2178310135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6815","nonce":2178310135}]},"public_addr":"172.21.15.117:6811/2178310135","cluster_addr":"172.21.15.117:6813/2178310135","heartbeat_back_addr":"172.21.15.117:6817/2178310135","heartbeat_front_addr":"172.21.15.117:6815/2178310135","state":["exists","up"]},{"osd":4,"uuid":"caf7ce86-caf0-4eaf-bbab-bd2d93e63c33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6809","nonce":3455398086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6811","nonce":3455398086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6815","nonce":3455398086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6813","nonce":3455398086}]},"public_addr":"172.21.15.119:6809/3455398086","cluster_addr":"172.21.15.119:6811/3455398086","heartbeat_back_addr":"172.21.15.119:6815/3455398086","heartbeat_front_addr":"172.21.15.119:6813/3455398086","state":["exists","up"]},{"osd":5,"uuid":"20ba0b8a-2e0f-4ea7-b435-430e44766200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6809","nonce":1259631912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6811","nonce":1259631912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6815","nonce":1259631912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6813","nonce":1259631912}]},"public_addr":"172.21.15.178:6809/1259631912","cluster_addr":"172.21.15.178:6811/1259631912","heartbeat_back_addr":"172.21.15.178:6815/1259631912","heartbeat_front_addr":"172.21.15.178:6813/1259631912","state":["exists","up"]},{"osd":6,"uuid":"75c2ddd7-f837-42e0-ac47-8cfbcf97a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6819","nonce":864146757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6821","nonce":864146757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6825","nonce":864146757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6823","nonce":864146757}]},"public_addr":"172.21.15.117:6819/864146757","cluster_addr":"172.21.15.117:6821/864146757","heartbeat_back_addr":"172.21.15.117:6825/864146757","heartbeat_front_addr":"172.21.15.117:6823/864146757","state":["exists","up"]},{"osd":7,"uuid":"929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6817","nonce":1273365545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6819","nonce":1273365545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6823","nonce":1273365545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6821","nonce":1273365545}]},"public_addr":"172.21.15.119:6817/1273365545","cluster_addr":"172.21.15.119:6819/1273365545","heartbeat_back_addr":"172.21.15.119:6823/1273365545","heartbeat_front_addr":"172.21.15.119:6821/1273365545","state":["exists","up"]},{"osd":8,"uuid":"fd371461-a538-4781-92e1-5eead4ca3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6817","nonce":1059587501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6819","nonce":1059587501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6823","nonce":1059587501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6821","nonce":1059587501}]},"public_addr":"172.21.15.178:6817/1059587501","cluster_addr":"172.21.15.178:6819/1059587501","heartbeat_back_addr":"172.21.15.178:6823/1059587501","heartbeat_front_addr":"172.21.15.178:6821/1059587501","state":["exists","up"]},{"osd":9,"uuid":"0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6827","nonce":3653875600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6829","nonce":3653875600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6833","nonce":3653875600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6831","nonce":3653875600}]},"public_addr":"172.21.15.117:6827/3653875600","cluster_addr":"172.21.15.117:6829/3653875600","heartbeat_back_addr":"172.21.15.117:6833/3653875600","heartbeat_front_addr":"172.21.15.117:6831/3653875600","state":["exists","up"]},{"osd":10,"uuid":"de2049da-d19e-4386-afbf-fd6b5fddbcba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6825","nonce":1050976194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6827","nonce":1050976194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6831","nonce":1050976194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6829","nonce":1050976194}]},"public_addr":"172.21.15.119:6825/1050976194","cluster_addr":"172.21.15.119:6827/1050976194","heartbeat_back_addr":"172.21.15.119:6831/1050976194","heartbeat_front_addr":"172.21.15.119:6829/1050976194","state":["exists","up"]},{"osd":11,"uuid":"3cccc0f9-d273-4b6b-9b8e-c926371472dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6825","nonce":2915080368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6827","nonce":2915080368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6831","nonce":2915080368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6829","nonce":2915080368}]},"public_addr":"172.21.15.178:6825/2915080368","cluster_addr":"172.21.15.178:6827/2915080368","heartbeat_back_addr":"172.21.15.178:6831/2915080368","heartbeat_front_addr":"172.21.15.178:6829/2915080368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:27.876274+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:53.094706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:16.598813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:48.160419+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:14.048925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:38.814304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:12.542328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:38.942747+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:04.488963+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:38.196053+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:05.254728+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:30.874915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.117:0/185684447":"2024-03-20T03:22:38.134578+0000","172.21.15.117:0/361367101":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/2788067603":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3506200675":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6800/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3283471541":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3750920169":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/2780813298":"2024-03-20T03:22:03.905461+0000","172.21.15.117:6800/2788067603":"2024-03-20T03:22:38.134578+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:30:43.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.040+0000 7f48f4400640 1 -- 172.21.15.117:0/3993877853 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f48d40a2800 msgr2=0x7f48d40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:43.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.040+0000 7f48f4400640 1 --2- 172.21.15.117:0/3993877853 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f48d40a2800 0x7f48d40a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f48ec19f650 tx=0x7f48d8005f90 comp rx=0 tx=0).stop 2024-03-19T03:30:43.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.041+0000 7f48f4400640 1 -- 172.21.15.117:0/3993877853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48ec107e60 msgr2=0x7f48ec19e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:43.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.041+0000 7f48f4400640 1 --2- 172.21.15.117:0/3993877853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48ec107e60 0x7f48ec19e140 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f48e0005ab0 tx=0x7f48e002c6c0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:42 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/517581102' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:30:43.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:42 smithi117 ceph-mon[31838]: pgmap v321: 129 pgs: 6 creating+peering, 120 active+clean, 3 unknown; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 2 op/s 2024-03-19T03:30:43.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:42 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2579736331' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:43.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.041+0000 7f48f4400640 1 -- 172.21.15.117:0/3993877853 shutdown_connections 2024-03-19T03:30:43.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.041+0000 7f48f4400640 1 --2- 172.21.15.117:0/3993877853 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f48d40a2800 0x7f48d40a4cc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.041+0000 7f48f4400640 1 --2- 172.21.15.117:0/3993877853 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48ec109970 0x7f48ec1a56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.041+0000 7f48f4400640 1 --2- 172.21.15.117:0/3993877853 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f48ec108fb0 0x7f48ec19e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.041+0000 7f48f4400640 1 --2- 172.21.15.117:0/3993877853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48ec107e60 0x7f48ec19e140 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.041+0000 7f48f4400640 1 -- 172.21.15.117:0/3993877853 >> 172.21.15.117:0/3993877853 conn(0x7f48ec076e30 msgr2=0x7f48ec10c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:43.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.041+0000 7f48f4400640 1 -- 172.21.15.117:0/3993877853 shutdown_connections 2024-03-19T03:30:43.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.041+0000 7f48f4400640 1 -- 172.21.15.117:0/3993877853 wait complete. 2024-03-19T03:30:43.060 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-19T03:30:35.081740+0000', 'modified': '2024-03-19T03:30:40.181111+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24599}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24599': {'gid': 24599, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.119:6839/2383354833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6838', 'nonce': 2383354833}, {'type': 'v1', 'addr': '172.21.15.119:6839', 'nonce': 2383354833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24599, 'qdb_cluster': [24599]} 2024-03-19T03:30:43.060 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-19T03:30:43.060 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-19T03:30:43.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:42 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/517581102' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:30:43.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:42 smithi178 ceph-mon[67578]: pgmap v321: 129 pgs: 6 creating+peering, 120 active+clean, 3 unknown; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 2 op/s 2024-03-19T03:30:43.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:42 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2579736331' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:30:43.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.262+0000 7fd3ff200640 1 -- 172.21.15.117:0/1551533371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3f8105400 msgr2=0x7fd3f810bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:43.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.262+0000 7fd3ff200640 1 --2- 172.21.15.117:0/1551533371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3f8105400 0x7fd3f810bc90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4009ca0 tx=0x7fd3f402c100 comp rx=0 tx=0).stop 2024-03-19T03:30:43.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.263+0000 7fd3ff200640 1 -- 172.21.15.117:0/1551533371 shutdown_connections 2024-03-19T03:30:43.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.263+0000 7fd3ff200640 1 --2- 172.21.15.117:0/1551533371 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3f8105400 0x7fd3f810bc90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.263+0000 7fd3ff200640 1 --2- 172.21.15.117:0/1551533371 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3f8104a40 0x7fd3f8104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.263+0000 7fd3ff200640 1 --2- 172.21.15.117:0/1551533371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd3f8103850 0x7fd3f8103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.263+0000 7fd3ff200640 1 -- 172.21.15.117:0/1551533371 >> 172.21.15.117:0/1551533371 conn(0x7fd3f80fedc0 msgr2=0x7fd3f8101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:43.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.263+0000 7fd3ff200640 1 -- 172.21.15.117:0/1551533371 shutdown_connections 2024-03-19T03:30:43.264 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.263+0000 7fd3ff200640 1 -- 172.21.15.117:0/1551533371 wait complete. 2024-03-19T03:30:43.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.264+0000 7fd3ff200640 1 Processor -- start 2024-03-19T03:30:43.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.264+0000 7fd3ff200640 1 -- start start 2024-03-19T03:30:43.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.264+0000 7fd3ff200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3f8103850 0x7fd3f8199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:43.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.264+0000 7fd3ff200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd3f8104a40 0x7fd3f8199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:43.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.264+0000 7fd3ff200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3f8105400 0x7fd3f81a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:43.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.264+0000 7fd3ff200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f810f4d0 con 0x7fd3f8104a40 2024-03-19T03:30:43.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.264+0000 7fd3ff200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f810f350 con 0x7fd3f8103850 2024-03-19T03:30:43.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.264+0000 7fd3ff200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f810f650 con 0x7fd3f8105400 2024-03-19T03:30:43.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.265+0000 7fd3fde00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3f8103850 0x7fd3f8199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:43.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.265+0000 7fd3fde00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3f8103850 0x7fd3f8199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57682/0 (socket says 172.21.15.117:57682) 2024-03-19T03:30:43.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.265+0000 7fd3fde00640 1 -- 172.21.15.117:0/2061628575 learned_addr learned my addr 172.21.15.117:0/2061628575 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:43.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.265+0000 7fd3fd400640 1 --2- 172.21.15.117:0/2061628575 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd3f8104a40 0x7fd3f8199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:43.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.265+0000 7fd3fe800640 1 --2- 172.21.15.117:0/2061628575 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3f8105400 0x7fd3f81a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:43.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.265+0000 7fd3fde00640 1 -- 172.21.15.117:0/2061628575 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3f8105400 msgr2=0x7fd3f81a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:43.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.265+0000 7fd3fde00640 1 --2- 172.21.15.117:0/2061628575 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3f8105400 0x7fd3f81a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.265+0000 7fd3fde00640 1 -- 172.21.15.117:0/2061628575 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd3f8104a40 msgr2=0x7fd3f8199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:43.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.265+0000 7fd3fde00640 1 --2- 172.21.15.117:0/2061628575 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd3f8104a40 0x7fd3f8199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.265+0000 7fd3fde00640 1 -- 172.21.15.117:0/2061628575 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3f81a16d0 con 0x7fd3f8103850 2024-03-19T03:30:43.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.265+0000 7fd3fde00640 1 --2- 172.21.15.117:0/2061628575 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3f8103850 0x7fd3f8199ab0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd3e800d770 tx=0x7fd3e800dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:43.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.265+0000 7fd3f2a00640 1 -- 172.21.15.117:0/2061628575 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e8017070 con 0x7fd3f8103850 2024-03-19T03:30:43.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.266+0000 7fd3f2a00640 1 -- 172.21.15.117:0/2061628575 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3e8009c30 con 0x7fd3f8103850 2024-03-19T03:30:43.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.266+0000 7fd3f2a00640 1 -- 172.21.15.117:0/2061628575 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e8003da0 con 0x7fd3f8103850 2024-03-19T03:30:43.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.266+0000 7fd3ff200640 1 -- 172.21.15.117:0/2061628575 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3f81a19c0 con 0x7fd3f8103850 2024-03-19T03:30:43.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.266+0000 7fd3ff200640 1 -- 172.21.15.117:0/2061628575 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3f81a1ec0 con 0x7fd3f8103850 2024-03-19T03:30:43.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.269+0000 7fd4033fb740 1 Processor -- start 2024-03-19T03:30:43.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.269+0000 7fd4033fb740 1 -- start start 2024-03-19T03:30:43.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.269+0000 7fd3f2a00640 1 -- 172.21.15.117:0/2061628575 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fd3e8003f40 con 0x7fd3f8103850 2024-03-19T03:30:43.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.269+0000 7fd4033fb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112e5c0 0x559ae112e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:43.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.269+0000 7fd4033fb740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x559ae112efb0 0x559ae1133ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:43.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.269+0000 7fd4033fb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ae1134120 0x559ae11365b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:43.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.269+0000 7fd4033fb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559ae1092d00 con 0x559ae1134120 2024-03-19T03:30:43.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.269+0000 7fd4033fb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x559ae103bb00 con 0x559ae112e5c0 2024-03-19T03:30:43.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.269+0000 7fd4033fb740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x559ae108eda0 con 0x559ae112efb0 2024-03-19T03:30:43.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.270+0000 7fd3fe800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112e5c0 0x559ae112e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:43.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.270+0000 7fd3fe800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112e5c0 0x559ae112e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57684/0 (socket says 172.21.15.117:57684) 2024-03-19T03:30:43.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.270+0000 7fd3fe800640 1 -- 172.21.15.117:0/1480182864 learned_addr learned my addr 172.21.15.117:0/1480182864 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:43.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.270+0000 7fd3fd400640 1 --2- 172.21.15.117:0/1480182864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ae1134120 0x559ae11365b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:43.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.270+0000 7fd3fde00640 1 --2- 172.21.15.117:0/1480182864 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x559ae112efb0 0x559ae1133ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:43.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.270+0000 7fd3fe800640 1 -- 172.21.15.117:0/1480182864 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x559ae112efb0 msgr2=0x559ae1133ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:30:43.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.270+0000 7fd3fe800640 1 --2- 172.21.15.117:0/1480182864 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x559ae112efb0 0x559ae1133ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.270+0000 7fd3fe800640 1 -- 172.21.15.117:0/1480182864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ae1134120 msgr2=0x559ae11365b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:30:43.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.270+0000 7fd3fe800640 1 --2- 172.21.15.117:0/1480182864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ae1134120 0x559ae11365b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.270+0000 7fd3fe800640 1 -- 172.21.15.117:0/1480182864 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ae0da72d0 con 0x559ae112e5c0 2024-03-19T03:30:43.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.270+0000 7fd3fe800640 1 --2- 172.21.15.117:0/1480182864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112e5c0 0x559ae112e9e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4005e30 tx=0x7fd3f40323d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4364fff8aa2f730f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:43.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.271+0000 7fd3e7000640 1 -- 172.21.15.117:0/1480182864 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f4043020 con 0x559ae112e5c0 2024-03-19T03:30:43.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.271+0000 7fd3e7000640 1 -- 172.21.15.117:0/1480182864 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3f4032c90 con 0x559ae112e5c0 2024-03-19T03:30:43.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.271+0000 7fd3e7000640 1 -- 172.21.15.117:0/1480182864 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f4039040 con 0x559ae112e5c0 2024-03-19T03:30:43.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.271+0000 7fd3f2a00640 1 --2- 172.21.15.117:0/2061628575 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd3e00a2880 0x7fd3e00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:43.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.271+0000 7fd3fde00640 1 --2- 172.21.15.117:0/2061628575 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd3e00a2880 0x7fd3e00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:43.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.271+0000 7fd3f2a00640 1 -- 172.21.15.117:0/2061628575 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd3e80ce620 con 0x7fd3f8103850 2024-03-19T03:30:43.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.271+0000 7fd4033fb740 1 -- 172.21.15.117:0/1480182864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112e5c0 msgr2=0x559ae112e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:43.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.271+0000 7fd4033fb740 1 --2- 172.21.15.117:0/1480182864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112e5c0 0x559ae112e9e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4005e30 tx=0x7fd3f40323d0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.272+0000 7fd4033fb740 1 -- 172.21.15.117:0/1480182864 shutdown_connections 2024-03-19T03:30:43.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.272+0000 7fd4033fb740 1 --2- 172.21.15.117:0/1480182864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ae1134120 0x559ae11365b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.272+0000 7fd4033fb740 1 --2- 172.21.15.117:0/1480182864 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x559ae112efb0 0x559ae1133ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.272+0000 7fd4033fb740 1 --2- 172.21.15.117:0/1480182864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112e5c0 0x559ae112e9e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.272+0000 7fd4033fb740 1 -- 172.21.15.117:0/1480182864 >> 172.21.15.117:0/1480182864 conn(0x559ae1129b30 msgr2=0x559ae1129f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:43.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.272+0000 7fd3fde00640 1 --2- 172.21.15.117:0/2061628575 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd3e00a2880 0x7fd3e00a4d40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8021000 tx=0x7fd3e80d40e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:43.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.272+0000 7fd4033fb740 1 -- 172.21.15.117:0/1480182864 shutdown_connections 2024-03-19T03:30:43.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.272+0000 7fd4033fb740 1 -- 172.21.15.117:0/1480182864 wait complete. 2024-03-19T03:30:43.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.272+0000 7fd4033fb740 1 -- start start 2024-03-19T03:30:43.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.272+0000 7fd4033fb740 1 Processor -- start 2024-03-19T03:30:43.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.273+0000 7fd4033fb740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x559ae112e5c0 0x559ae1138ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:43.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.273+0000 7fd4033fb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112efb0 0x559ae1139010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:43.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.273+0000 7fd4033fb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ae1134120 0x559ae1140050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:43.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.273+0000 7fd4033fb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559ae1137070 con 0x559ae1134120 2024-03-19T03:30:43.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.273+0000 7fd4033fb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x559ae104eed0 con 0x559ae112efb0 2024-03-19T03:30:43.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.273+0000 7fd4033fb740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x559ae11371f0 con 0x559ae112e5c0 2024-03-19T03:30:43.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.273+0000 7fd4033fb740 10 client.0 _ll_register_callbacks cb 0x559ae104cc10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:30:43.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.273+0000 7fd3fd400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112efb0 0x559ae1139010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:43.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.273+0000 7fd3fd400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112efb0 0x559ae1139010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57692/0 (socket says 172.21.15.117:57692) 2024-03-19T03:30:43.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.273+0000 7fd3fd400640 1 -- 172.21.15.117:0/80354244 learned_addr learned my addr 172.21.15.117:0/80354244 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:43.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.273+0000 7fd3fe800640 1 --2- 172.21.15.117:0/80354244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ae1134120 0x559ae1140050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:43.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.273+0000 7fd3fe800640 1 --2- 172.21.15.117:0/80354244 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x559ae112e5c0 0x559ae1138ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:43.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.273+0000 7fd3fd400640 1 -- 172.21.15.117:0/80354244 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x559ae112e5c0 msgr2=0x559ae1138ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:30:43.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.273+0000 7fd3fd400640 1 --2- 172.21.15.117:0/80354244 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x559ae112e5c0 0x559ae1138ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.273+0000 7fd3fd400640 1 -- 172.21.15.117:0/80354244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ae1134120 msgr2=0x559ae1140050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:30:43.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.273+0000 7fd3fd400640 1 --2- 172.21.15.117:0/80354244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ae1134120 0x559ae1140050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.273+0000 7fd3fd400640 1 -- 172.21.15.117:0/80354244 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ae0e716b0 con 0x559ae112efb0 2024-03-19T03:30:43.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.274+0000 7fd3fd400640 1 --2- 172.21.15.117:0/80354244 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112efb0 0x559ae1139010 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec012720 tx=0x7fd3ec012be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=114fc4c1d2be797d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:43.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.274+0000 7fd3db400640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-19T03:30:43.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.274+0000 7fd3db400640 1 -- 172.21.15.117:0/80354244 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec00f920 con 0x559ae112efb0 2024-03-19T03:30:43.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.274+0000 7fd3db400640 1 -- 172.21.15.117:0/80354244 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd3ec00fac0 con 0x559ae112efb0 2024-03-19T03:30:43.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.274+0000 7fd4033fb740 1 -- 172.21.15.117:0/80354244 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3f4032c90 con 0x559ae112efb0 2024-03-19T03:30:43.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.274+0000 7fd3db400640 1 -- 172.21.15.117:0/80354244 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fd3ec00fcc0 con 0x559ae112efb0 2024-03-19T03:30:43.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.274+0000 7fd4033fb740 10 client.24707 fetch_fsmap learned FSMap version 17 2024-03-19T03:30:43.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.274+0000 7fd4033fb740 1 -- 172.21.15.117:0/80354244 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x559ae1143460 con 0x559ae112efb0 2024-03-19T03:30:43.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.274+0000 7fd3db400640 1 -- 172.21.15.117:0/80354244 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3f4032c90 con 0x559ae112efb0 2024-03-19T03:30:43.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.275+0000 7fd3db400640 1 -- 172.21.15.117:0/80354244 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fd3ec01b450 con 0x559ae112efb0 2024-03-19T03:30:43.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.275+0000 7fd4033fb740 10 client.24707 fetch_fsmap finished waiting for FSMap version 17 2024-03-19T03:30:43.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.275+0000 7fd4033fb740 10 client.24707 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-19T03:30:43.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.275+0000 7fd4033fb740 20 client.24707 populate_metadata read hostname 'smithi117' 2024-03-19T03:30:43.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.275+0000 7fd4033fb740 1 --2- 172.21.15.117:0/80354244 >> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] conn(0x559ae1145b30 0x559ae1146fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:43.282 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.275+0000 7fd4033fb740 4 client.24707 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:30:43.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.275+0000 7fd4033fb740 1 -- 172.21.15.117:0/80354244 --> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ae10a0240 con 0x559ae1145b30 2024-03-19T03:30:43.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.275+0000 7fd3fe800640 1 --2- 172.21.15.117:0/80354244 >> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] conn(0x559ae1145b30 0x559ae1146fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:43.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.275+0000 7fd3fe800640 1 --2- 172.21.15.117:0/80354244 >> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] conn(0x559ae1145b30 0x559ae1146fa0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d84009d147a2c8d server_cookie=35fdb73870199a83 in_seq=0 out_seq=0 2024-03-19T03:30:43.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.275+0000 7fd3db400640 10 client.24707 ms_handle_connect on v2:172.21.15.119:6838/2383354833 2024-03-19T03:30:43.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.276+0000 7fd3db400640 1 -- 172.21.15.117:0/80354244 <== mds.0 v2:172.21.15.119:6838/2383354833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559ae10a0240 con 0x559ae1145b30 2024-03-19T03:30:43.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.276+0000 7fd3db400640 10 client.24707 handle_command_reply: tid=0 2024-03-19T03:30:43.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.277+0000 7fd4033fb740 1 client.24707 shutdown 2024-03-19T03:30:43.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.277+0000 7fd4033fb740 1 -- 172.21.15.117:0/80354244 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112efb0 msgr2=0x559ae1139010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:43.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.277+0000 7fd4033fb740 1 --2- 172.21.15.117:0/80354244 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112efb0 0x559ae1139010 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec012720 tx=0x7fd3ec012be0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.277+0000 7fd4033fb740 1 -- 172.21.15.117:0/80354244 shutdown_connections 2024-03-19T03:30:43.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.277+0000 7fd4033fb740 1 --2- 172.21.15.117:0/80354244 >> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] conn(0x559ae1145b30 0x559ae1146fa0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.277+0000 7fd4033fb740 1 --2- 172.21.15.117:0/80354244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ae1134120 0x559ae1140050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.277+0000 7fd4033fb740 1 --2- 172.21.15.117:0/80354244 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112efb0 0x559ae1139010 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.277+0000 7fd3db400640 0 client.24707 ms_handle_reset on v2:172.21.15.119:6838/2383354833 2024-03-19T03:30:43.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.277+0000 7fd4033fb740 1 --2- 172.21.15.117:0/80354244 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x559ae112e5c0 0x559ae1138ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.277+0000 7fd4033fb740 1 -- 172.21.15.117:0/80354244 >> 172.21.15.117:0/80354244 conn(0x559ae1129b30 msgr2=0x559ae1128d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:43.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.277+0000 7fd4033fb740 1 -- 172.21.15.117:0/80354244 shutdown_connections 2024-03-19T03:30:43.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.277+0000 7fd4033fb740 1 -- 172.21.15.117:0/80354244 wait complete. 2024-03-19T03:30:43.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.277+0000 7fd4033fb740 20 client.24707 trim_cache size 0 max 16384 2024-03-19T03:30:43.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.296+0000 7fd4033fb740 1 Processor -- start 2024-03-19T03:30:43.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.296+0000 7fd4033fb740 1 -- start start 2024-03-19T03:30:43.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.296+0000 7fd4033fb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112e5c0 0x559ae1137da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:43.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.296+0000 7fd4033fb740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x559ae112efb0 0x559ae1140010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:43.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.297+0000 7fd4033fb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ae1134120 0x559ae11382e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:43.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.297+0000 7fd4033fb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559ae11452e0 con 0x559ae1134120 2024-03-19T03:30:43.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.297+0000 7fd4033fb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x559ae11459b0 con 0x559ae112e5c0 2024-03-19T03:30:43.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.297+0000 7fd4033fb740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x559ae1146080 con 0x559ae112efb0 2024-03-19T03:30:43.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.297+0000 7fd3fe800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112e5c0 0x559ae1137da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:43.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.297+0000 7fd3fe800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112e5c0 0x559ae1137da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57702/0 (socket says 172.21.15.117:57702) 2024-03-19T03:30:43.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.297+0000 7fd3fe800640 1 -- 172.21.15.117:0/995620138 learned_addr learned my addr 172.21.15.117:0/995620138 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:43.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.297+0000 7fd3fe800640 1 --2- 172.21.15.117:0/995620138 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ae1134120 0x559ae11382e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:43.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.297+0000 7fd3fd400640 1 --2- 172.21.15.117:0/995620138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x559ae112efb0 0x559ae1140010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:43.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.297+0000 7fd3fe800640 1 -- 172.21.15.117:0/995620138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x559ae112efb0 msgr2=0x559ae1140010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:30:43.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.297+0000 7fd3fe800640 1 --2- 172.21.15.117:0/995620138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x559ae112efb0 0x559ae1140010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.297+0000 7fd3fe800640 1 -- 172.21.15.117:0/995620138 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ae1134120 msgr2=0x559ae11382e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:30:43.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.297+0000 7fd3fe800640 1 --2- 172.21.15.117:0/995620138 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ae1134120 0x559ae11382e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.297+0000 7fd3fe800640 1 -- 172.21.15.117:0/995620138 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ae1139a30 con 0x559ae112e5c0 2024-03-19T03:30:43.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.298+0000 7fd3fe800640 1 --2- 172.21.15.117:0/995620138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112e5c0 0x559ae1137da0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd3f40325f0 tx=0x7fd3f4036040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fd11d1a3458620b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:43.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.298+0000 7fd3e6600640 1 -- 172.21.15.117:0/995620138 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f4039070 con 0x559ae112e5c0 2024-03-19T03:30:43.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.298+0000 7fd3e6600640 1 -- 172.21.15.117:0/995620138 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3f4032710 con 0x559ae112e5c0 2024-03-19T03:30:43.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.298+0000 7fd3e6600640 1 -- 172.21.15.117:0/995620138 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f4043040 con 0x559ae112e5c0 2024-03-19T03:30:43.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.298+0000 7fd4033fb740 1 -- 172.21.15.117:0/995620138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112e5c0 msgr2=0x559ae1137da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:43.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.298+0000 7fd4033fb740 1 --2- 172.21.15.117:0/995620138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112e5c0 0x559ae1137da0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd3f40325f0 tx=0x7fd3f4036040 comp rx=0 tx=0).stop 2024-03-19T03:30:43.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.298+0000 7fd4033fb740 1 -- 172.21.15.117:0/995620138 shutdown_connections 2024-03-19T03:30:43.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.298+0000 7fd4033fb740 1 --2- 172.21.15.117:0/995620138 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ae1134120 0x559ae11382e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.298+0000 7fd4033fb740 1 --2- 172.21.15.117:0/995620138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x559ae112efb0 0x559ae1140010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.298+0000 7fd4033fb740 1 --2- 172.21.15.117:0/995620138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112e5c0 0x559ae1137da0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.298+0000 7fd4033fb740 1 -- 172.21.15.117:0/995620138 >> 172.21.15.117:0/995620138 conn(0x559ae1129b30 msgr2=0x559ae1138aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:43.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.299+0000 7fd4033fb740 1 -- 172.21.15.117:0/995620138 shutdown_connections 2024-03-19T03:30:43.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.299+0000 7fd4033fb740 1 -- 172.21.15.117:0/995620138 wait complete. 2024-03-19T03:30:43.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.299+0000 7fd4033fb740 1 -- start start 2024-03-19T03:30:43.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.299+0000 7fd4033fb740 1 Processor -- start 2024-03-19T03:30:43.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.299+0000 7fd4033fb740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x559ae112e5c0 0x559ae113bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:43.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.299+0000 7fd4033fb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112efb0 0x559ae113c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:43.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.300+0000 7fd4033fb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ae1134120 0x559ae113f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:43.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.300+0000 7fd4033fb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559ae1146e20 con 0x559ae1134120 2024-03-19T03:30:43.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.300+0000 7fd4033fb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x559ae1146750 con 0x559ae112efb0 2024-03-19T03:30:43.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.300+0000 7fd4033fb740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x559ae11474f0 con 0x559ae112e5c0 2024-03-19T03:30:43.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.300+0000 7fd4033fb740 10 client.0 _ll_register_callbacks cb 0x7fd3ec009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:30:43.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.300+0000 7fd3fd400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112efb0 0x559ae113c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:43.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.300+0000 7fd3fd400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112efb0 0x559ae113c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57712/0 (socket says 172.21.15.117:57712) 2024-03-19T03:30:43.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.300+0000 7fd3fd400640 1 -- 172.21.15.117:0/814598183 learned_addr learned my addr 172.21.15.117:0/814598183 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:43.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.300+0000 7fd3fe800640 1 --2- 172.21.15.117:0/814598183 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x559ae112e5c0 0x559ae113bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:43.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.300+0000 7fd3fd400640 1 -- 172.21.15.117:0/814598183 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x559ae112e5c0 msgr2=0x559ae113bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:30:43.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.300+0000 7fd3fd400640 1 --2- 172.21.15.117:0/814598183 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x559ae112e5c0 0x559ae113bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.300+0000 7fd3fe800640 1 --2- 172.21.15.117:0/814598183 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ae1134120 0x559ae113f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:43.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.300+0000 7fd3fd400640 1 -- 172.21.15.117:0/814598183 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ae1134120 msgr2=0x559ae113f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:30:43.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.300+0000 7fd3fd400640 1 --2- 172.21.15.117:0/814598183 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ae1134120 0x559ae113f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.300+0000 7fd3fd400640 1 -- 172.21.15.117:0/814598183 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ae113f8b0 con 0x559ae112efb0 2024-03-19T03:30:43.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.300+0000 7fd3fe800640 1 --2- 172.21.15.117:0/814598183 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ae1134120 0x559ae113f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T03:30:43.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.300+0000 7fd3fd400640 1 --2- 172.21.15.117:0/814598183 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112efb0 0x559ae113c2a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec01b960 tx=0x7fd3ec01be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94623069a32a5ebe server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:43.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.300+0000 7fd3db400640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-19T03:30:43.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.301+0000 7fd3db400640 1 -- 172.21.15.117:0/814598183 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec00fe60 con 0x559ae112efb0 2024-03-19T03:30:43.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.301+0000 7fd4033fb740 1 -- 172.21.15.117:0/814598183 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3f4032710 con 0x559ae112efb0 2024-03-19T03:30:43.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.301+0000 7fd3db400640 1 -- 172.21.15.117:0/814598183 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd3ec01b600 con 0x559ae112efb0 2024-03-19T03:30:43.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.301+0000 7fd3db400640 1 -- 172.21.15.117:0/814598183 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fd3ec0050d0 con 0x559ae112efb0 2024-03-19T03:30:43.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.301+0000 7fd4033fb740 10 client.24713 fetch_fsmap learned FSMap version 17 2024-03-19T03:30:43.305 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.301+0000 7fd4033fb740 1 -- 172.21.15.117:0/814598183 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x559ae11b4720 con 0x559ae112efb0 2024-03-19T03:30:43.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.301+0000 7fd3db400640 1 -- 172.21.15.117:0/814598183 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3f4032710 con 0x559ae112efb0 2024-03-19T03:30:43.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.301+0000 7fd3db400640 1 -- 172.21.15.117:0/814598183 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fd3ec003c90 con 0x559ae112efb0 2024-03-19T03:30:43.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.301+0000 7fd4033fb740 10 client.24713 fetch_fsmap finished waiting for FSMap version 17 2024-03-19T03:30:43.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.301+0000 7fd4033fb740 10 client.24713 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-19T03:30:43.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.301+0000 7fd4033fb740 20 client.24713 populate_metadata read hostname 'smithi117' 2024-03-19T03:30:43.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.301+0000 7fd4033fb740 1 --2- 172.21.15.117:0/814598183 >> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] conn(0x559ae11b9350 0x559ae11b5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:43.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.301+0000 7fd4033fb740 4 client.24713 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T03:30:43.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.301+0000 7fd4033fb740 1 -- 172.21.15.117:0/814598183 --> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] -- command(tid 0: {"prefix": "status"}) -- 0x559ae109a590 con 0x559ae11b9350 2024-03-19T03:30:43.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.302+0000 7fd3fe800640 1 --2- 172.21.15.117:0/814598183 >> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] conn(0x559ae11b9350 0x559ae11b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:43.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.302+0000 7fd3fe800640 1 --2- 172.21.15.117:0/814598183 >> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] conn(0x559ae11b9350 0x559ae11b5790 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57a6d57f357b55dd server_cookie=6e1134ca1efe5989 in_seq=0 out_seq=0 2024-03-19T03:30:43.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.302+0000 7fd3db400640 10 client.24713 ms_handle_connect on v2:172.21.15.119:6838/2383354833 2024-03-19T03:30:43.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.302+0000 7fd3db400640 1 -- 172.21.15.117:0/814598183 <== mds.0 v2:172.21.15.119:6838/2383354833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x559ae109a590 con 0x559ae11b9350 2024-03-19T03:30:43.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.302+0000 7fd3db400640 10 client.24713 handle_command_reply: tid=0 2024-03-19T03:30:43.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.302+0000 7fd4033fb740 1 client.24713 shutdown 2024-03-19T03:30:43.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.302+0000 7fd4033fb740 1 -- 172.21.15.117:0/814598183 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112efb0 msgr2=0x559ae113c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:43.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.302+0000 7fd4033fb740 1 --2- 172.21.15.117:0/814598183 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112efb0 0x559ae113c2a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec01b960 tx=0x7fd3ec01be00 comp rx=0 tx=0).stop 2024-03-19T03:30:43.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.302+0000 7fd4033fb740 1 -- 172.21.15.117:0/814598183 shutdown_connections 2024-03-19T03:30:43.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.302+0000 7fd4033fb740 1 --2- 172.21.15.117:0/814598183 >> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] conn(0x559ae11b9350 0x559ae11b5790 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.302+0000 7fd4033fb740 1 --2- 172.21.15.117:0/814598183 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559ae1134120 0x559ae113f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.302+0000 7fd4033fb740 1 --2- 172.21.15.117:0/814598183 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x559ae112efb0 0x559ae113c2a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.302+0000 7fd4033fb740 1 --2- 172.21.15.117:0/814598183 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x559ae112e5c0 0x559ae113bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.302+0000 7fd3db400640 0 client.24713 ms_handle_reset on v2:172.21.15.119:6838/2383354833 2024-03-19T03:30:43.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.302+0000 7fd4033fb740 1 -- 172.21.15.117:0/814598183 >> 172.21.15.117:0/814598183 conn(0x559ae1129b30 msgr2=0x559ae1137da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:43.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.303+0000 7fd4033fb740 1 -- 172.21.15.117:0/814598183 shutdown_connections 2024-03-19T03:30:43.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.303+0000 7fd4033fb740 1 -- 172.21.15.117:0/814598183 wait complete. 2024-03-19T03:30:43.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.303+0000 7fd4033fb740 20 client.24713 trim_cache size 0 max 16384 2024-03-19T03:30:43.309 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-19T03:30:43.309 INFO:teuthology.orchestra.run.smithi117.stdout: "cluster_fsid": "a3a82456-e59f-11ee-95c9-87774f69a715", 2024-03-19T03:30:43.309 INFO:teuthology.orchestra.run.smithi117.stdout: "whoami": 0, 2024-03-19T03:30:43.309 INFO:teuthology.orchestra.run.smithi117.stdout: "id": 24599, 2024-03-19T03:30:43.309 INFO:teuthology.orchestra.run.smithi117.stdout: "want_state": "up:active", 2024-03-19T03:30:43.309 INFO:teuthology.orchestra.run.smithi117.stdout: "state": "up:active", 2024-03-19T03:30:43.309 INFO:teuthology.orchestra.run.smithi117.stdout: "fs_name": "cephfs", 2024-03-19T03:30:43.310 INFO:teuthology.orchestra.run.smithi117.stdout: "rank_uptime": 7.6357007220000002, 2024-03-19T03:30:43.310 INFO:teuthology.orchestra.run.smithi117.stdout: "mdsmap_epoch": 17, 2024-03-19T03:30:43.310 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch": 89, 2024-03-19T03:30:43.310 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch_barrier": 89, 2024-03-19T03:30:43.310 INFO:teuthology.orchestra.run.smithi117.stdout: "uptime": 14.979503488000001 2024-03-19T03:30:43.310 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-19T03:30:43.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.303+0000 7fd3ff200640 1 -- 172.21.15.117:0/2061628575 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd3e00a2880 msgr2=0x7fd3e00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:43.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.303+0000 7fd3ff200640 1 --2- 172.21.15.117:0/2061628575 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd3e00a2880 0x7fd3e00a4d40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8021000 tx=0x7fd3e80d40e0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.303+0000 7fd3ff200640 1 -- 172.21.15.117:0/2061628575 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3f8103850 msgr2=0x7fd3f8199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:43.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.303+0000 7fd3ff200640 1 --2- 172.21.15.117:0/2061628575 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3f8103850 0x7fd3f8199ab0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd3e800d770 tx=0x7fd3e800dc30 comp rx=0 tx=0).stop 2024-03-19T03:30:43.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.303+0000 7fd3ff200640 1 -- 172.21.15.117:0/2061628575 shutdown_connections 2024-03-19T03:30:43.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.303+0000 7fd3ff200640 1 --2- 172.21.15.117:0/2061628575 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3f8105400 0x7fd3f81a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.303+0000 7fd3ff200640 1 --2- 172.21.15.117:0/2061628575 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd3f8104a40 0x7fd3f8199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.303+0000 7fd3ff200640 1 --2- 172.21.15.117:0/2061628575 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd3e00a2880 0x7fd3e00a4d40 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.303+0000 7fd3ff200640 1 --2- 172.21.15.117:0/2061628575 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3f8103850 0x7fd3f8199ab0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:43.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.303+0000 7fd3ff200640 1 -- 172.21.15.117:0/2061628575 >> 172.21.15.117:0/2061628575 conn(0x7fd3f80fedc0 msgr2=0x7fd3f8107d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:43.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.304+0000 7fd3ff200640 1 -- 172.21.15.117:0/2061628575 shutdown_connections 2024-03-19T03:30:43.311 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:43.304+0000 7fd3ff200640 1 -- 172.21.15.117:0/2061628575 wait complete. 2024-03-19T03:30:43.321 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-19T03:30:43.321 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-19T03:30:43.321 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-19T03:30:43.743 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:30:43.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:43 smithi119 ceph-mon[66861]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-19T03:30:43.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:43 smithi119 ceph-mon[66861]: Cluster is now healthy 2024-03-19T03:30:43.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:43 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3993877853' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:30:44.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:43 smithi117 ceph-mon[31838]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-19T03:30:44.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:43 smithi117 ceph-mon[31838]: Cluster is now healthy 2024-03-19T03:30:44.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:43 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3993877853' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:30:44.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:43 smithi178 ceph-mon[67578]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-19T03:30:44.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:43 smithi178 ceph-mon[67578]: Cluster is now healthy 2024-03-19T03:30:44.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:43 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3993877853' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:30:45.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:44 smithi117 ceph-mon[31838]: pgmap v322: 129 pgs: 6 creating+peering, 123 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 2 op/s 2024-03-19T03:30:45.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:44 smithi178 ceph-mon[67578]: pgmap v322: 129 pgs: 6 creating+peering, 123 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 2 op/s 2024-03-19T03:30:45.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:44 smithi119 ceph-mon[66861]: pgmap v322: 129 pgs: 6 creating+peering, 123 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 2 op/s 2024-03-19T03:30:46.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.017+0000 7f7a41000700 1 -- 172.21.15.117:0/2448849916 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a3c077160 msgr2=0x7f7a3c112c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:46.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.017+0000 7f7a41000700 1 --2- 172.21.15.117:0/2448849916 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a3c077160 0x7f7a3c112c50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7a28009990 tx=0x7f7a28009ca0 comp rx=0 tx=0).stop 2024-03-19T03:30:46.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.017+0000 7f7a41000700 1 -- 172.21.15.117:0/2448849916 shutdown_connections 2024-03-19T03:30:46.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.017+0000 7f7a41000700 1 --2- 172.21.15.117:0/2448849916 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a3c113190 0x7f7a3c115580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:46.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.017+0000 7f7a41000700 1 --2- 172.21.15.117:0/2448849916 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a3c076840 0x7f7a3c076c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:46.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.017+0000 7f7a41000700 1 --2- 172.21.15.117:0/2448849916 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a3c077160 0x7f7a3c112c50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:46.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.017+0000 7f7a41000700 1 -- 172.21.15.117:0/2448849916 >> 172.21.15.117:0/2448849916 conn(0x7f7a3c100020 msgr2=0x7f7a3c102440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:46.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.017+0000 7f7a41000700 1 -- 172.21.15.117:0/2448849916 shutdown_connections 2024-03-19T03:30:46.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.017+0000 7f7a41000700 1 -- 172.21.15.117:0/2448849916 wait complete. 2024-03-19T03:30:46.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.018+0000 7f7a41000700 1 Processor -- start 2024-03-19T03:30:46.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.019+0000 7f7a41000700 1 -- start start 2024-03-19T03:30:46.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.019+0000 7f7a41000700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a3c076840 0x7f7a3c19f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:46.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.019+0000 7f7a41000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a3c077160 0x7f7a3c19f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:46.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.019+0000 7f7a41000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a3c113190 0x7f7a3c1a3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:46.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.019+0000 7f7a41000700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7a3c118c70 con 0x7f7a3c113190 2024-03-19T03:30:46.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.019+0000 7f7a41000700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7a3c118af0 con 0x7f7a3c077160 2024-03-19T03:30:46.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.019+0000 7f7a41000700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f7a3c118df0 con 0x7f7a3c076840 2024-03-19T03:30:46.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.019+0000 7f7a39e00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a3c076840 0x7f7a3c19f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:46.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.019+0000 7f7a39400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a3c077160 0x7f7a3c19f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:46.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.020+0000 7f7a39400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a3c077160 0x7f7a3c19f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:47532/0 (socket says 172.21.15.117:47532) 2024-03-19T03:30:46.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.020+0000 7f7a39400700 1 -- 172.21.15.117:0/3261615728 learned_addr learned my addr 172.21.15.117:0/3261615728 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:46.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.020+0000 7f7a39e00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a3c076840 0x7f7a3c19f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:39136/0 (socket says 172.21.15.117:39136) 2024-03-19T03:30:46.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.020+0000 7f7a3a800700 1 --2- 172.21.15.117:0/3261615728 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a3c113190 0x7f7a3c1a3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:46.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.020+0000 7f7a39400700 1 -- 172.21.15.117:0/3261615728 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a3c076840 msgr2=0x7f7a3c19f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:46.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.020+0000 7f7a39400700 1 --2- 172.21.15.117:0/3261615728 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a3c076840 0x7f7a3c19f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:46.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.020+0000 7f7a39400700 1 -- 172.21.15.117:0/3261615728 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a3c113190 msgr2=0x7f7a3c1a3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:46.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.020+0000 7f7a39400700 1 --2- 172.21.15.117:0/3261615728 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a3c113190 0x7f7a3c1a3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:46.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.020+0000 7f7a39400700 1 -- 172.21.15.117:0/3261615728 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a3c1a40d0 con 0x7f7a3c077160 2024-03-19T03:30:46.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.020+0000 7f7a39400700 1 --2- 172.21.15.117:0/3261615728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a3c077160 0x7f7a3c19f600 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7a2800bae0 tx=0x7f7a2800bb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:46.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.020+0000 7f7a32a00700 1 -- 172.21.15.117:0/3261615728 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a28026070 con 0x7f7a3c077160 2024-03-19T03:30:46.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.020+0000 7f7a41000700 1 -- 172.21.15.117:0/3261615728 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a3c1a4360 con 0x7f7a3c077160 2024-03-19T03:30:46.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.020+0000 7f7a32a00700 1 -- 172.21.15.117:0/3261615728 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a28003d70 con 0x7f7a3c077160 2024-03-19T03:30:46.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.020+0000 7f7a32a00700 1 -- 172.21.15.117:0/3261615728 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a28021410 con 0x7f7a3c077160 2024-03-19T03:30:46.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.021+0000 7f7a41000700 1 -- 172.21.15.117:0/3261615728 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a3c1abc40 con 0x7f7a3c077160 2024-03-19T03:30:46.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.022+0000 7f7a41000700 1 -- 172.21.15.117:0/3261615728 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a3c068fd0 con 0x7f7a3c077160 2024-03-19T03:30:46.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.024+0000 7f7a32a00700 1 -- 172.21.15.117:0/3261615728 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f7a28029070 con 0x7f7a3c077160 2024-03-19T03:30:46.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.025+0000 7f7a32a00700 1 --2- 172.21.15.117:0/3261615728 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7a200a2890 0x7f7a200a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:46.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.025+0000 7f7a32a00700 1 -- 172.21.15.117:0/3261615728 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7a280cfd50 con 0x7f7a3c077160 2024-03-19T03:30:46.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.025+0000 7f7a39e00700 1 --2- 172.21.15.117:0/3261615728 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7a200a2890 0x7f7a200a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:46.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.026+0000 7f7a39e00700 1 --2- 172.21.15.117:0/3261615728 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7a200a2890 0x7f7a200a4d50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c009c60 tx=0x7f7a2c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:46.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.027+0000 7f7a32a00700 1 -- 172.21.15.117:0/3261615728 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7a28013070 con 0x7f7a3c077160 2024-03-19T03:30:46.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.300+0000 7f7a41000700 1 -- 172.21.15.117:0/3261615728 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f7a3c02dbb0 con 0x7f7a3c077160 2024-03-19T03:30:46.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.300+0000 7f7a32a00700 1 -- 172.21.15.117:0/3261615728 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f7a2809cbe0 con 0x7f7a3c077160 2024-03-19T03:30:46.301 INFO:teuthology.orchestra.run.smithi117.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-19T03:30:46.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.303+0000 7f7a41000700 1 -- 172.21.15.117:0/3261615728 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7a200a2890 msgr2=0x7f7a200a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:46.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.303+0000 7f7a41000700 1 --2- 172.21.15.117:0/3261615728 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7a200a2890 0x7f7a200a4d50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c009c60 tx=0x7f7a2c009260 comp rx=0 tx=0).stop 2024-03-19T03:30:46.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.303+0000 7f7a41000700 1 -- 172.21.15.117:0/3261615728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a3c077160 msgr2=0x7f7a3c19f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:46.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.303+0000 7f7a41000700 1 --2- 172.21.15.117:0/3261615728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a3c077160 0x7f7a3c19f600 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7a2800bae0 tx=0x7f7a2800bb10 comp rx=0 tx=0).stop 2024-03-19T03:30:46.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.303+0000 7f7a41000700 1 -- 172.21.15.117:0/3261615728 shutdown_connections 2024-03-19T03:30:46.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.303+0000 7f7a41000700 1 --2- 172.21.15.117:0/3261615728 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7a200a2890 0x7f7a200a4d50 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:46.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.303+0000 7f7a41000700 1 --2- 172.21.15.117:0/3261615728 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a3c113190 0x7f7a3c1a3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:46.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.303+0000 7f7a41000700 1 --2- 172.21.15.117:0/3261615728 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a3c076840 0x7f7a3c19f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:46.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.303+0000 7f7a41000700 1 --2- 172.21.15.117:0/3261615728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a3c077160 0x7f7a3c19f600 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:46.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.303+0000 7f7a41000700 1 -- 172.21.15.117:0/3261615728 >> 172.21.15.117:0/3261615728 conn(0x7f7a3c100020 msgr2=0x7f7a3c102410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:46.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.304+0000 7f7a41000700 1 -- 172.21.15.117:0/3261615728 shutdown_connections 2024-03-19T03:30:46.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:46.304+0000 7f7a41000700 1 -- 172.21.15.117:0/3261615728 wait complete. 2024-03-19T03:30:46.686 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:46 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3261615728' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-19T03:30:46.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:46 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3261615728' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-19T03:30:46.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:46 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3261615728' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-19T03:30:47.059 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-19T03:30:47.060 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-19T03:30:47.060 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi117.front.sepia.ceph.com 2024-03-19T03:30:47.060 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-19T03:30:47.060 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-19T03:30:47.060 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-19T03:30:47.060 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-19T03:30:47.060 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-19T03:30:47.060 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-19T03:30:47.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:30:47.060 DEBUG:teuthology.orchestra.run.smithi117:> ip addr 2024-03-19T03:30:47.114 INFO:teuthology.orchestra.run.smithi117.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-19T03:30:47.114 INFO:teuthology.orchestra.run.smithi117.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-19T03:30:47.114 INFO:teuthology.orchestra.run.smithi117.stdout: inet 127.0.0.1/8 scope host lo 2024-03-19T03:30:47.114 INFO:teuthology.orchestra.run.smithi117.stdout: valid_lft forever preferred_lft forever 2024-03-19T03:30:47.114 INFO:teuthology.orchestra.run.smithi117.stdout: inet6 ::1/128 scope host 2024-03-19T03:30:47.115 INFO:teuthology.orchestra.run.smithi117.stdout: valid_lft forever preferred_lft forever 2024-03-19T03:30:47.115 INFO:teuthology.orchestra.run.smithi117.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T03:30:47.115 INFO:teuthology.orchestra.run.smithi117.stdout: link/ether 0c:c4:7a:d8:a8:8a brd ff:ff:ff:ff:ff:ff 2024-03-19T03:30:47.115 INFO:teuthology.orchestra.run.smithi117.stdout: altname enp7s0f0 2024-03-19T03:30:47.115 INFO:teuthology.orchestra.run.smithi117.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T03:30:47.115 INFO:teuthology.orchestra.run.smithi117.stdout: link/ether 0c:c4:7a:d8:a8:8b brd ff:ff:ff:ff:ff:ff 2024-03-19T03:30:47.115 INFO:teuthology.orchestra.run.smithi117.stdout: altname enp7s0f1 2024-03-19T03:30:47.115 INFO:teuthology.orchestra.run.smithi117.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T03:30:47.115 INFO:teuthology.orchestra.run.smithi117.stdout: link/ether 0c:c4:7a:8f:44:62 brd ff:ff:ff:ff:ff:ff 2024-03-19T03:30:47.115 INFO:teuthology.orchestra.run.smithi117.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-19T03:30:47.115 INFO:teuthology.orchestra.run.smithi117.stdout: link/ether 0c:c4:7a:8f:44:63 brd ff:ff:ff:ff:ff:ff 2024-03-19T03:30:47.115 INFO:teuthology.orchestra.run.smithi117.stdout: inet 172.21.15.117/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-19T03:30:47.115 INFO:teuthology.orchestra.run.smithi117.stdout: valid_lft 40951sec preferred_lft 40951sec 2024-03-19T03:30:47.116 INFO:teuthology.orchestra.run.smithi117.stdout: inet6 fe80::ec4:7aff:fe8f:4463/64 scope link 2024-03-19T03:30:47.116 INFO:teuthology.orchestra.run.smithi117.stdout: valid_lft forever preferred_lft forever 2024-03-19T03:30:47.116 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-19T03:30:47.116 DEBUG:teuthology.orchestra.run.smithi117:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T03:30:47.116 DEBUG:teuthology.orchestra.run.smithi117:> set -e 2024-03-19T03:30:47.117 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip link add name ceph-brx type bridge 2024-03-19T03:30:47.117 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip addr flush dev ceph-brx 2024-03-19T03:30:47.117 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip link set ceph-brx up 2024-03-19T03:30:47.117 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-19T03:30:47.117 DEBUG:teuthology.orchestra.run.smithi117:> ') 2024-03-19T03:30:47.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T03:30:47.516 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:47 smithi117 ceph-mon[31838]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 946 B/s wr, 0 op/s 2024-03-19T03:30:47.612 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:30:47.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:30:47.623 DEBUG:teuthology.orchestra.run.smithi117:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-19T03:30:47.706 INFO:teuthology.orchestra.run.smithi117.stdout:1 2024-03-19T03:30:47.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:30:47.710 DEBUG:teuthology.orchestra.run.smithi117:> route 2024-03-19T03:30:47.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:47 smithi119 ceph-mon[66861]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 946 B/s wr, 0 op/s 2024-03-19T03:30:47.772 INFO:teuthology.orchestra.run.smithi117.stdout:Kernel IP routing table 2024-03-19T03:30:47.772 INFO:teuthology.orchestra.run.smithi117.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-19T03:30:47.772 INFO:teuthology.orchestra.run.smithi117.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-19T03:30:47.772 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-19T03:30:47.772 INFO:teuthology.orchestra.run.smithi117.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-19T03:30:47.774 DEBUG:teuthology.orchestra.run.smithi117:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T03:30:47.774 DEBUG:teuthology.orchestra.run.smithi117:> set -e 2024-03-19T03:30:47.774 DEBUG:teuthology.orchestra.run.smithi117:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-19T03:30:47.774 DEBUG:teuthology.orchestra.run.smithi117:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-19T03:30:47.774 DEBUG:teuthology.orchestra.run.smithi117:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-19T03:30:47.774 DEBUG:teuthology.orchestra.run.smithi117:> ') 2024-03-19T03:30:47.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:47 smithi178 ceph-mon[67578]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 946 B/s wr, 0 op/s 2024-03-19T03:30:47.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T03:30:48.542 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:30:48.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:30:48.552 DEBUG:teuthology.orchestra.run.smithi117:> ip netns list 2024-03-19T03:30:48.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:30:48.607 DEBUG:teuthology.orchestra.run.smithi117:> ip netns list-id 2024-03-19T03:30:48.660 DEBUG:teuthology.orchestra.run.smithi117:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T03:30:48.660 DEBUG:teuthology.orchestra.run.smithi117:> set -e 2024-03-19T03:30:48.660 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T03:30:48.660 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-19T03:30:48.660 DEBUG:teuthology.orchestra.run.smithi117:> ') 2024-03-19T03:30:48.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T03:30:48.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:30:48.859 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-19T03:30:48.860 DEBUG:teuthology.orchestra.run.smithi117:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T03:30:48.860 DEBUG:teuthology.orchestra.run.smithi117:> set -e 2024-03-19T03:30:48.860 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-19T03:30:48.860 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-19T03:30:48.860 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-19T03:30:48.861 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-19T03:30:48.861 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-19T03:30:48.861 DEBUG:teuthology.orchestra.run.smithi117:> ') 2024-03-19T03:30:48.962 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T03:30:49.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:30:49.207 DEBUG:teuthology.orchestra.run.smithi117:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T03:30:49.207 DEBUG:teuthology.orchestra.run.smithi117:> set -e 2024-03-19T03:30:49.208 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip link set brx.0 up 2024-03-19T03:30:49.208 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip link set dev brx.0 master ceph-brx 2024-03-19T03:30:49.208 DEBUG:teuthology.orchestra.run.smithi117:> ') 2024-03-19T03:30:49.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T03:30:49.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:30:49.437 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-19T03:30:49.437 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T03:30:49.437 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-19T03:30:49.480 INFO:teuthology.orchestra.run.smithi117.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-19T03:30:49.480 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T03:30:49.481 DEBUG:teuthology.orchestra.run.smithi117:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-19T03:30:49.571 DEBUG:teuthology.orchestra.run.smithi117:> sudo modprobe fuse 2024-03-19T03:30:49.645 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:49 smithi117 ceph-mon[31838]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-19T03:30:49.661 DEBUG:teuthology.orchestra.run.smithi117:> cat /proc/self/mounts | awk '{print $2}' 2024-03-19T03:30:49.715 INFO:teuthology.orchestra.run.smithi117.stdout:/proc 2024-03-19T03:30:49.715 INFO:teuthology.orchestra.run.smithi117.stdout:/sys 2024-03-19T03:30:49.716 INFO:teuthology.orchestra.run.smithi117.stdout:/dev 2024-03-19T03:30:49.716 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/kernel/security 2024-03-19T03:30:49.716 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/shm 2024-03-19T03:30:49.716 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/pts 2024-03-19T03:30:49.716 INFO:teuthology.orchestra.run.smithi117.stdout:/run 2024-03-19T03:30:49.716 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/cgroup 2024-03-19T03:30:49.716 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/pstore 2024-03-19T03:30:49.716 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/bpf 2024-03-19T03:30:49.716 INFO:teuthology.orchestra.run.smithi117.stdout:/ 2024-03-19T03:30:49.716 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/selinux 2024-03-19T03:30:49.716 INFO:teuthology.orchestra.run.smithi117.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T03:30:49.716 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/mqueue 2024-03-19T03:30:49.716 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/hugepages 2024-03-19T03:30:49.716 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/kernel/debug 2024-03-19T03:30:49.717 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/kernel/tracing 2024-03-19T03:30:49.717 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/fuse/connections 2024-03-19T03:30:49.717 INFO:teuthology.orchestra.run.smithi117.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-19T03:30:49.717 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/kernel/config 2024-03-19T03:30:49.717 INFO:teuthology.orchestra.run.smithi117.stdout:/run/credentials/systemd-sysctl.service 2024-03-19T03:30:49.717 INFO:teuthology.orchestra.run.smithi117.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-19T03:30:49.717 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/nfs/rpc_pipefs 2024-03-19T03:30:49.717 INFO:teuthology.orchestra.run.smithi117.stdout:/run/user/1000 2024-03-19T03:30:49.717 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/ceph 2024-03-19T03:30:49.717 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay 2024-03-19T03:30:49.717 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/ffcb41e49d8a60d7438e8a70df807eb4d007e0b5849e6f3849397d53059129ae/merged 2024-03-19T03:30:49.717 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/46a4dc4127c2edbd670054f0315c040b89c4fe10a7dc1007d6d72c17f7b56dc2/merged 2024-03-19T03:30:49.717 INFO:teuthology.orchestra.run.smithi117.stdout:/run/user/0 2024-03-19T03:30:49.717 INFO:teuthology.orchestra.run.smithi117.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T03:30:49.718 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/b7948c96b409834b2d64e8652bf66a3d40c48d875a9367792d6ffdcd2b345b2c/merged 2024-03-19T03:30:49.718 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/a5678fb3e75e2613880448e4f088d5e68abbc851d0312377caa2f00ba257195e/merged 2024-03-19T03:30:49.718 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/c1cd9cb5777c985125a799d940052cf0f7f1abf4a9a5faf71ade6b1fa0ee5c6d/merged 2024-03-19T03:30:49.718 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/a77a26bb231ab360a3837039fde2434861acbb9b112875e882101e02bfa7dc3b/merged 2024-03-19T03:30:49.718 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/0d18c46d2dc20ab64abe0cfa724931dd6b4b9459a0a02e9630fae6d4a0d48068/merged 2024-03-19T03:30:49.718 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/4b36ddb9a4bdf1b9e02f367ef94385c2257ea8a4ec1cf83f6f2e0b0622d0030f/merged 2024-03-19T03:30:49.718 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/4f23a87d11ff137837a44bb90099357250b84015f62a0232753c933468bc6b7a/merged 2024-03-19T03:30:49.718 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/00807ca34357a6b73f371350940d94662cee6a0dbf6c17cfcfb32380f1a9b7fc/merged 2024-03-19T03:30:49.718 INFO:teuthology.orchestra.run.smithi117.stdout:/run/netns 2024-03-19T03:30:49.718 INFO:teuthology.orchestra.run.smithi117.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T03:30:49.718 INFO:teuthology.orchestra.run.smithi117.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T03:30:49.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:30:49.719 DEBUG:teuthology.orchestra.run.smithi117:> ls /sys/fs/fuse/connections 2024-03-19T03:30:49.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:49 smithi119 ceph-mon[66861]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-19T03:30:49.767 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-19T03:30:49.767 DEBUG:teuthology.orchestra.run.smithi117:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-19T03:30:49.772 DEBUG:teuthology.orchestra.run.smithi117:> sudo modprobe fuse 2024-03-19T03:30:49.846 DEBUG:teuthology.orchestra.run.smithi117:> cat /proc/self/mounts | awk '{print $2}' 2024-03-19T03:30:49.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:49 smithi178 ceph-mon[67578]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-19T03:30:49.888 INFO:teuthology.orchestra.run.smithi117.stdout:/proc 2024-03-19T03:30:49.888 INFO:teuthology.orchestra.run.smithi117.stdout:/sys 2024-03-19T03:30:49.888 INFO:teuthology.orchestra.run.smithi117.stdout:/dev 2024-03-19T03:30:49.888 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/kernel/security 2024-03-19T03:30:49.889 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/shm 2024-03-19T03:30:49.889 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/pts 2024-03-19T03:30:49.889 INFO:teuthology.orchestra.run.smithi117.stdout:/run 2024-03-19T03:30:49.889 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/cgroup 2024-03-19T03:30:49.889 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/pstore 2024-03-19T03:30:49.889 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/bpf 2024-03-19T03:30:49.889 INFO:teuthology.orchestra.run.smithi117.stdout:/ 2024-03-19T03:30:49.889 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/selinux 2024-03-19T03:30:49.889 INFO:teuthology.orchestra.run.smithi117.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T03:30:49.889 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/mqueue 2024-03-19T03:30:49.889 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/hugepages 2024-03-19T03:30:49.889 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/kernel/debug 2024-03-19T03:30:49.889 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/kernel/tracing 2024-03-19T03:30:49.889 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/fuse/connections 2024-03-19T03:30:49.889 INFO:teuthology.orchestra.run.smithi117.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-19T03:30:49.890 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/kernel/config 2024-03-19T03:30:49.890 INFO:teuthology.orchestra.run.smithi117.stdout:/run/credentials/systemd-sysctl.service 2024-03-19T03:30:49.890 INFO:teuthology.orchestra.run.smithi117.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-19T03:30:49.890 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/nfs/rpc_pipefs 2024-03-19T03:30:49.890 INFO:teuthology.orchestra.run.smithi117.stdout:/run/user/1000 2024-03-19T03:30:49.890 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/ceph 2024-03-19T03:30:49.890 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay 2024-03-19T03:30:49.890 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/ffcb41e49d8a60d7438e8a70df807eb4d007e0b5849e6f3849397d53059129ae/merged 2024-03-19T03:30:49.890 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/46a4dc4127c2edbd670054f0315c040b89c4fe10a7dc1007d6d72c17f7b56dc2/merged 2024-03-19T03:30:49.890 INFO:teuthology.orchestra.run.smithi117.stdout:/run/user/0 2024-03-19T03:30:49.891 INFO:teuthology.orchestra.run.smithi117.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T03:30:49.891 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/b7948c96b409834b2d64e8652bf66a3d40c48d875a9367792d6ffdcd2b345b2c/merged 2024-03-19T03:30:49.891 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/a5678fb3e75e2613880448e4f088d5e68abbc851d0312377caa2f00ba257195e/merged 2024-03-19T03:30:49.891 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/c1cd9cb5777c985125a799d940052cf0f7f1abf4a9a5faf71ade6b1fa0ee5c6d/merged 2024-03-19T03:30:49.891 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/a77a26bb231ab360a3837039fde2434861acbb9b112875e882101e02bfa7dc3b/merged 2024-03-19T03:30:49.891 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/0d18c46d2dc20ab64abe0cfa724931dd6b4b9459a0a02e9630fae6d4a0d48068/merged 2024-03-19T03:30:49.891 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/4b36ddb9a4bdf1b9e02f367ef94385c2257ea8a4ec1cf83f6f2e0b0622d0030f/merged 2024-03-19T03:30:49.891 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/4f23a87d11ff137837a44bb90099357250b84015f62a0232753c933468bc6b7a/merged 2024-03-19T03:30:49.891 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/00807ca34357a6b73f371350940d94662cee6a0dbf6c17cfcfb32380f1a9b7fc/merged 2024-03-19T03:30:49.891 INFO:teuthology.orchestra.run.smithi117.stdout:/run/netns 2024-03-19T03:30:49.891 INFO:teuthology.orchestra.run.smithi117.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T03:30:49.891 INFO:teuthology.orchestra.run.smithi117.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T03:30:49.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:30:49.892 DEBUG:teuthology.orchestra.run.smithi117:> ls /sys/fs/fuse/connections 2024-03-19T03:30:49.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi117.stderr:2024-03-19T03:30:49.976+0000 7f1234445480 -1 init, newargv = 0x55d07729ba00 newargc=15 2024-03-19T03:30:49.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi117.stderr:ceph-fuse[76168]: starting ceph client 2024-03-19T03:30:49.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi117.stderr:ceph-fuse[76168]: starting fuse 2024-03-19T03:30:51.082 DEBUG:teuthology.orchestra.run.smithi117:> sudo modprobe fuse 2024-03-19T03:30:51.184 DEBUG:teuthology.orchestra.run.smithi117:> cat /proc/self/mounts | awk '{print $2}' 2024-03-19T03:30:51.237 INFO:teuthology.orchestra.run.smithi117.stdout:/proc 2024-03-19T03:30:51.237 INFO:teuthology.orchestra.run.smithi117.stdout:/sys 2024-03-19T03:30:51.237 INFO:teuthology.orchestra.run.smithi117.stdout:/dev 2024-03-19T03:30:51.237 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/kernel/security 2024-03-19T03:30:51.237 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/shm 2024-03-19T03:30:51.237 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/pts 2024-03-19T03:30:51.237 INFO:teuthology.orchestra.run.smithi117.stdout:/run 2024-03-19T03:30:51.237 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/cgroup 2024-03-19T03:30:51.237 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/pstore 2024-03-19T03:30:51.237 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/bpf 2024-03-19T03:30:51.237 INFO:teuthology.orchestra.run.smithi117.stdout:/ 2024-03-19T03:30:51.238 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/selinux 2024-03-19T03:30:51.238 INFO:teuthology.orchestra.run.smithi117.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T03:30:51.238 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/mqueue 2024-03-19T03:30:51.238 INFO:teuthology.orchestra.run.smithi117.stdout:/dev/hugepages 2024-03-19T03:30:51.238 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/kernel/debug 2024-03-19T03:30:51.238 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/kernel/tracing 2024-03-19T03:30:51.238 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/fs/fuse/connections 2024-03-19T03:30:51.238 INFO:teuthology.orchestra.run.smithi117.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-19T03:30:51.238 INFO:teuthology.orchestra.run.smithi117.stdout:/sys/kernel/config 2024-03-19T03:30:51.238 INFO:teuthology.orchestra.run.smithi117.stdout:/run/credentials/systemd-sysctl.service 2024-03-19T03:30:51.238 INFO:teuthology.orchestra.run.smithi117.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-19T03:30:51.238 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/nfs/rpc_pipefs 2024-03-19T03:30:51.238 INFO:teuthology.orchestra.run.smithi117.stdout:/run/user/1000 2024-03-19T03:30:51.238 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/ceph 2024-03-19T03:30:51.239 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay 2024-03-19T03:30:51.239 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/ffcb41e49d8a60d7438e8a70df807eb4d007e0b5849e6f3849397d53059129ae/merged 2024-03-19T03:30:51.239 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/46a4dc4127c2edbd670054f0315c040b89c4fe10a7dc1007d6d72c17f7b56dc2/merged 2024-03-19T03:30:51.239 INFO:teuthology.orchestra.run.smithi117.stdout:/run/user/0 2024-03-19T03:30:51.239 INFO:teuthology.orchestra.run.smithi117.stdout:/proc/sys/fs/binfmt_misc 2024-03-19T03:30:51.239 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/b7948c96b409834b2d64e8652bf66a3d40c48d875a9367792d6ffdcd2b345b2c/merged 2024-03-19T03:30:51.239 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/a5678fb3e75e2613880448e4f088d5e68abbc851d0312377caa2f00ba257195e/merged 2024-03-19T03:30:51.239 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/c1cd9cb5777c985125a799d940052cf0f7f1abf4a9a5faf71ade6b1fa0ee5c6d/merged 2024-03-19T03:30:51.239 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/a77a26bb231ab360a3837039fde2434861acbb9b112875e882101e02bfa7dc3b/merged 2024-03-19T03:30:51.239 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/0d18c46d2dc20ab64abe0cfa724931dd6b4b9459a0a02e9630fae6d4a0d48068/merged 2024-03-19T03:30:51.239 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/4b36ddb9a4bdf1b9e02f367ef94385c2257ea8a4ec1cf83f6f2e0b0622d0030f/merged 2024-03-19T03:30:51.239 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/4f23a87d11ff137837a44bb90099357250b84015f62a0232753c933468bc6b7a/merged 2024-03-19T03:30:51.239 INFO:teuthology.orchestra.run.smithi117.stdout:/var/lib/containers/storage/overlay/00807ca34357a6b73f371350940d94662cee6a0dbf6c17cfcfb32380f1a9b7fc/merged 2024-03-19T03:30:51.239 INFO:teuthology.orchestra.run.smithi117.stdout:/run/netns 2024-03-19T03:30:51.239 INFO:teuthology.orchestra.run.smithi117.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T03:30:51.240 INFO:teuthology.orchestra.run.smithi117.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T03:30:51.240 INFO:teuthology.orchestra.run.smithi117.stdout:/home/ubuntu/cephtest/mnt.admin 2024-03-19T03:30:51.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:30:51.241 DEBUG:teuthology.orchestra.run.smithi117:> ls /sys/fs/fuse/connections 2024-03-19T03:30:51.289 INFO:teuthology.orchestra.run.smithi117.stdout:118 2024-03-19T03:30:51.289 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [118] 2024-03-19T03:30:51.290 DEBUG:teuthology.orchestra.run.smithi117:> sudo stdin-killer -- python3 -c ' 2024-03-19T03:30:51.290 DEBUG:teuthology.orchestra.run.smithi117:> import glob 2024-03-19T03:30:51.290 DEBUG:teuthology.orchestra.run.smithi117:> import re 2024-03-19T03:30:51.290 DEBUG:teuthology.orchestra.run.smithi117:> import os 2024-03-19T03:30:51.290 DEBUG:teuthology.orchestra.run.smithi117:> import subprocess 2024-03-19T03:30:51.290 DEBUG:teuthology.orchestra.run.smithi117:> 2024-03-19T03:30:51.290 DEBUG:teuthology.orchestra.run.smithi117:> def _find_admin_socket(client_name): 2024-03-19T03:30:51.290 DEBUG:teuthology.orchestra.run.smithi117:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-19T03:30:51.290 DEBUG:teuthology.orchestra.run.smithi117:> files = glob.glob(asok_path) 2024-03-19T03:30:51.290 DEBUG:teuthology.orchestra.run.smithi117:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-19T03:30:51.290 DEBUG:teuthology.orchestra.run.smithi117:> 2024-03-19T03:30:51.290 DEBUG:teuthology.orchestra.run.smithi117:> # Given a non-glob path, it better be there 2024-03-19T03:30:51.291 DEBUG:teuthology.orchestra.run.smithi117:> if "*" not in asok_path: 2024-03-19T03:30:51.291 DEBUG:teuthology.orchestra.run.smithi117:> assert(len(files) == 1) 2024-03-19T03:30:51.291 DEBUG:teuthology.orchestra.run.smithi117:> return files[0] 2024-03-19T03:30:51.291 DEBUG:teuthology.orchestra.run.smithi117:> 2024-03-19T03:30:51.291 DEBUG:teuthology.orchestra.run.smithi117:> for f in files: 2024-03-19T03:30:51.291 DEBUG:teuthology.orchestra.run.smithi117:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-19T03:30:51.291 DEBUG:teuthology.orchestra.run.smithi117:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-19T03:30:51.291 DEBUG:teuthology.orchestra.run.smithi117:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-19T03:30:51.291 DEBUG:teuthology.orchestra.run.smithi117:> contents = proc_f.read() 2024-03-19T03:30:51.291 DEBUG:teuthology.orchestra.run.smithi117:> if mountpoint in contents: 2024-03-19T03:30:51.291 DEBUG:teuthology.orchestra.run.smithi117:> return f 2024-03-19T03:30:51.291 DEBUG:teuthology.orchestra.run.smithi117:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-19T03:30:51.291 DEBUG:teuthology.orchestra.run.smithi117:> 2024-03-19T03:30:51.292 DEBUG:teuthology.orchestra.run.smithi117:> print(_find_admin_socket("client.admin")) 2024-03-19T03:30:51.292 DEBUG:teuthology.orchestra.run.smithi117:> ' 2024-03-19T03:30:51.457 INFO:teuthology.orchestra.run.smithi117.stdout:/var/run/ceph/ceph-client.admin.76168.asok 2024-03-19T03:30:51.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:30:51.470 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:51 smithi117 ceph-mon[31838]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-19T03:30:51.473 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.76168.asok 2024-03-19T03:30:51.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:30:51.474 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.76168.asok status 2024-03-19T03:30:51.648 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:51 smithi178 ceph-mon[67578]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-19T03:30:51.715 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-19T03:30:51.715 INFO:teuthology.orchestra.run.smithi117.stdout: "metadata": { 2024-03-19T03:30:51.715 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_sha1": "b6ae24918b03b1b8a19c9263a50655bba80bb810", 2024-03-19T03:30:51.715 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_version": "ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev)", 2024-03-19T03:30:51.715 INFO:teuthology.orchestra.run.smithi117.stdout: "entity_id": "admin", 2024-03-19T03:30:51.715 INFO:teuthology.orchestra.run.smithi117.stdout: "hostname": "smithi117", 2024-03-19T03:30:51.715 INFO:teuthology.orchestra.run.smithi117.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-19T03:30:51.715 INFO:teuthology.orchestra.run.smithi117.stdout: "pid": "76168", 2024-03-19T03:30:51.715 INFO:teuthology.orchestra.run.smithi117.stdout: "root": "/" 2024-03-19T03:30:51.715 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:30:51.716 INFO:teuthology.orchestra.run.smithi117.stdout: "dentry_count": 0, 2024-03-19T03:30:51.716 INFO:teuthology.orchestra.run.smithi117.stdout: "dentry_pinned_count": 0, 2024-03-19T03:30:51.716 INFO:teuthology.orchestra.run.smithi117.stdout: "id": 24725, 2024-03-19T03:30:51.716 INFO:teuthology.orchestra.run.smithi117.stdout: "inst": { 2024-03-19T03:30:51.716 INFO:teuthology.orchestra.run.smithi117.stdout: "name": { 2024-03-19T03:30:51.716 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "client", 2024-03-19T03:30:51.716 INFO:teuthology.orchestra.run.smithi117.stdout: "num": 24725 2024-03-19T03:30:51.716 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:30:51.716 INFO:teuthology.orchestra.run.smithi117.stdout: "addr": { 2024-03-19T03:30:51.716 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "v1", 2024-03-19T03:30:51.716 INFO:teuthology.orchestra.run.smithi117.stdout: "addr": "172.21.15.117:0", 2024-03-19T03:30:51.716 INFO:teuthology.orchestra.run.smithi117.stdout: "nonce": 3729986476 2024-03-19T03:30:51.716 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:30:51.716 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:30:51.716 INFO:teuthology.orchestra.run.smithi117.stdout: "addr": { 2024-03-19T03:30:51.717 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "v1", 2024-03-19T03:30:51.717 INFO:teuthology.orchestra.run.smithi117.stdout: "addr": "172.21.15.117:0", 2024-03-19T03:30:51.717 INFO:teuthology.orchestra.run.smithi117.stdout: "nonce": 3729986476 2024-03-19T03:30:51.717 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:30:51.717 INFO:teuthology.orchestra.run.smithi117.stdout: "inst_str": "client.24725 172.21.15.117:0/3729986476", 2024-03-19T03:30:51.717 INFO:teuthology.orchestra.run.smithi117.stdout: "addr_str": "172.21.15.117:0/3729986476", 2024-03-19T03:30:51.717 INFO:teuthology.orchestra.run.smithi117.stdout: "inode_count": 1, 2024-03-19T03:30:51.717 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_epoch": 17, 2024-03-19T03:30:51.717 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_epoch": 89, 2024-03-19T03:30:51.717 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_epoch_barrier": 0, 2024-03-19T03:30:51.717 INFO:teuthology.orchestra.run.smithi117.stdout: "blocklisted": false, 2024-03-19T03:30:51.717 INFO:teuthology.orchestra.run.smithi117.stdout: "fs_name": "cephfs" 2024-03-19T03:30:51.717 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-19T03:30:51.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:30:51.730 DEBUG:teuthology.orchestra.run.smithi117:> stat --file-system '--printf=%T 2024-03-19T03:30:51.730 DEBUG:teuthology.orchestra.run.smithi117:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-19T03:30:51.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:51 smithi119 ceph-mon[66861]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-19T03:30:51.778 INFO:teuthology.orchestra.run.smithi117.stdout:fuseblk 2024-03-19T03:30:51.778 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-19T03:30:51.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:30:51.779 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-19T03:30:51.919 DEBUG:teuthology.orchestra.run.smithi117:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-19T03:30:52.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-19T03:30:52.065 INFO:teuthology.orchestra.run.smithi117.stdout:# file: . 2024-03-19T03:30:52.065 INFO:teuthology.orchestra.run.smithi117.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-19T03:30:52.065 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:30:52.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:30:52.076 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-19T03:30:52.076 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T03:30:52.127 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:30:52.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.331+0000 7fce50c00640 1 -- 172.21.15.117:0/105546050 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce4c077dc0 msgr2=0x7fce4c078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:52.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.331+0000 7fce50c00640 1 --2- 172.21.15.117:0/105546050 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce4c077dc0 0x7fce4c078660 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fce44009ca0 tx=0x7fce4402c100 comp rx=0 tx=0).stop 2024-03-19T03:30:52.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.331+0000 7fce50c00640 1 -- 172.21.15.117:0/105546050 shutdown_connections 2024-03-19T03:30:52.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.331+0000 7fce50c00640 1 --2- 172.21.15.117:0/105546050 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce4c077dc0 0x7fce4c078660 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:52.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.331+0000 7fce50c00640 1 --2- 172.21.15.117:0/105546050 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce4c077420 0x7fce4c077880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:52.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.331+0000 7fce50c00640 1 --2- 172.21.15.117:0/105546050 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fce4c0761e0 0x7fce4c076600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:52.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.331+0000 7fce50c00640 1 -- 172.21.15.117:0/105546050 >> 172.21.15.117:0/105546050 conn(0x7fce4c0fefd0 msgr2=0x7fce4c101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:52.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.331+0000 7fce50c00640 1 -- 172.21.15.117:0/105546050 shutdown_connections 2024-03-19T03:30:52.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.331+0000 7fce50c00640 1 -- 172.21.15.117:0/105546050 wait complete. 2024-03-19T03:30:52.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.332+0000 7fce50c00640 1 Processor -- start 2024-03-19T03:30:52.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.332+0000 7fce50c00640 1 -- start start 2024-03-19T03:30:52.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.332+0000 7fce50c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce4c0761e0 0x7fce4c19e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:52.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.332+0000 7fce50c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fce4c077420 0x7fce4c19e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:52.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.332+0000 7fce50c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce4c077dc0 0x7fce4c1a5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:52.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.332+0000 7fce50c00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fce4c113af0 con 0x7fce4c0761e0 2024-03-19T03:30:52.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.332+0000 7fce50c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fce4c113970 con 0x7fce4c077dc0 2024-03-19T03:30:52.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.332+0000 7fce50c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fce4c113c70 con 0x7fce4c077420 2024-03-19T03:30:52.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.333+0000 7fce49e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce4c0761e0 0x7fce4c19e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:52.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.333+0000 7fce4a800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce4c077dc0 0x7fce4c1a5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:52.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.333+0000 7fce41400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fce4c077420 0x7fce4c19e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:52.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.333+0000 7fce49e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce4c0761e0 0x7fce4c19e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:47784/0 (socket says 172.21.15.117:47784) 2024-03-19T03:30:52.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.333+0000 7fce49e00640 1 -- 172.21.15.117:0/386508155 learned_addr learned my addr 172.21.15.117:0/386508155 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:52.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.333+0000 7fce4a800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce4c077dc0 0x7fce4c1a5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:47548/0 (socket says 172.21.15.117:47548) 2024-03-19T03:30:52.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.333+0000 7fce4a800640 1 -- 172.21.15.117:0/386508155 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fce4c077420 msgr2=0x7fce4c19e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:52.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.333+0000 7fce4a800640 1 --2- 172.21.15.117:0/386508155 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fce4c077420 0x7fce4c19e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:52.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.333+0000 7fce4a800640 1 -- 172.21.15.117:0/386508155 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce4c0761e0 msgr2=0x7fce4c19e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:52.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.333+0000 7fce4a800640 1 --2- 172.21.15.117:0/386508155 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce4c0761e0 0x7fce4c19e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:52.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.333+0000 7fce4a800640 1 -- 172.21.15.117:0/386508155 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce4c1a5cb0 con 0x7fce4c077dc0 2024-03-19T03:30:52.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.333+0000 7fce4a800640 1 --2- 172.21.15.117:0/386508155 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce4c077dc0 0x7fce4c1a5640 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fce44037040 tx=0x7fce44035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:52.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.334+0000 7fce43400640 1 -- 172.21.15.117:0/386508155 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce44039040 con 0x7fce4c077dc0 2024-03-19T03:30:52.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.334+0000 7fce43400640 1 -- 172.21.15.117:0/386508155 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce44044070 con 0x7fce4c077dc0 2024-03-19T03:30:52.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.334+0000 7fce43400640 1 -- 172.21.15.117:0/386508155 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce44004370 con 0x7fce4c077dc0 2024-03-19T03:30:52.336 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.334+0000 7fce50c00640 1 -- 172.21.15.117:0/386508155 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce4c1a5f40 con 0x7fce4c077dc0 2024-03-19T03:30:52.336 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.334+0000 7fce50c00640 1 -- 172.21.15.117:0/386508155 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce4c1a6350 con 0x7fce4c077dc0 2024-03-19T03:30:52.336 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.335+0000 7fce50c00640 1 -- 172.21.15.117:0/386508155 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce4c1104e0 con 0x7fce4c077dc0 2024-03-19T03:30:52.336 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.336+0000 7fce43400640 1 -- 172.21.15.117:0/386508155 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fce44004510 con 0x7fce4c077dc0 2024-03-19T03:30:52.338 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.338+0000 7fce43400640 1 --2- 172.21.15.117:0/386508155 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fce300a2880 0x7fce300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:52.338 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.338+0000 7fce43400640 1 -- 172.21.15.117:0/386508155 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fce440f1250 con 0x7fce4c077dc0 2024-03-19T03:30:52.338 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.338+0000 7fce49e00640 1 --2- 172.21.15.117:0/386508155 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fce300a2880 0x7fce300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:52.338 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.339+0000 7fce49e00640 1 --2- 172.21.15.117:0/386508155 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fce300a2880 0x7fce300a4d40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fce3c009cd0 tx=0x7fce3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:52.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.342+0000 7fce43400640 1 -- 172.21.15.117:0/386508155 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fce440be050 con 0x7fce4c077dc0 2024-03-19T03:30:52.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.546+0000 7fce50c00640 1 -- 172.21.15.117:0/386508155 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fce4c062420 con 0x7fce4c077dc0 2024-03-19T03:30:52.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.548+0000 7fce43400640 1 -- 172.21.15.117:0/386508155 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24210 (secure 0 0 0) 0x7fce440098b0 con 0x7fce4c077dc0 2024-03-19T03:30:52.548 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:30:52.548 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":89,"fsid":"a3a82456-e59f-11ee-95c9-87774f69a715","created":"2024-03-19T03:21:30.427136+0000","modified":"2024-03-19T03:30:40.180745+0000","last_up_change":"2024-03-19T03:29:32.967505+0000","last_in_change":"2024-03-19T03:29:12.848740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:25:18.824270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T03:30:33.137170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T03:30:34.598838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T03:30:38.230023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b32eb0ee-9d4b-4748-87ec-eda02c907415","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6803","nonce":3014934900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6805","nonce":3014934900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6809","nonce":3014934900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6807","nonce":3014934900}]},"public_addr":"172.21.15.117:6803/3014934900","cluster_addr":"172.21.15.117:6805/3014934900","heartbeat_back_addr":"172.21.15.117:6809/3014934900","heartbeat_front_addr":"172.21.15.117:6807/3014934900","state":["exists","up"]},{"osd":1,"uuid":"e07585be-c016-4466-b07c-abbc22a58de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6801","nonce":2425927341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6803","nonce":2425927341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6807","nonce":2425927341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6805","nonce":2425927341}]},"public_addr":"172.21.15.119:6801/2425927341","cluster_addr":"172.21.15.119:6803/2425927341","heartbeat_back_addr":"172.21.15.119:6807/2425927341","heartbeat_front_addr":"172.21.15.119:6805/2425927341","state":["exists","up"]},{"osd":2,"uuid":"ede9edda-3b9f-41ca-ad04-5dcb738ab757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6801","nonce":3733812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6803","nonce":3733812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6807","nonce":3733812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6805","nonce":3733812989}]},"public_addr":"172.21.15.178:6801/3733812989","cluster_addr":"172.21.15.178:6803/3733812989","heartbeat_back_addr":"172.21.15.178:6807/3733812989","heartbeat_front_addr":"172.21.15.178:6805/3733812989","state":["exists","up"]},{"osd":3,"uuid":"f81c41e1-3a69-40db-a90f-58151286b081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6811","nonce":2178310135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6813","nonce":2178310135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6817","nonce":2178310135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6815","nonce":2178310135}]},"public_addr":"172.21.15.117:6811/2178310135","cluster_addr":"172.21.15.117:6813/2178310135","heartbeat_back_addr":"172.21.15.117:6817/2178310135","heartbeat_front_addr":"172.21.15.117:6815/2178310135","state":["exists","up"]},{"osd":4,"uuid":"caf7ce86-caf0-4eaf-bbab-bd2d93e63c33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6809","nonce":3455398086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6811","nonce":3455398086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6815","nonce":3455398086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6813","nonce":3455398086}]},"public_addr":"172.21.15.119:6809/3455398086","cluster_addr":"172.21.15.119:6811/3455398086","heartbeat_back_addr":"172.21.15.119:6815/3455398086","heartbeat_front_addr":"172.21.15.119:6813/3455398086","state":["exists","up"]},{"osd":5,"uuid":"20ba0b8a-2e0f-4ea7-b435-430e44766200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6809","nonce":1259631912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6811","nonce":1259631912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6815","nonce":1259631912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6813","nonce":1259631912}]},"public_addr":"172.21.15.178:6809/1259631912","cluster_addr":"172.21.15.178:6811/1259631912","heartbeat_back_addr":"172.21.15.178:6815/1259631912","heartbeat_front_addr":"172.21.15.178:6813/1259631912","state":["exists","up"]},{"osd":6,"uuid":"75c2ddd7-f837-42e0-ac47-8cfbcf97a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6819","nonce":864146757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6821","nonce":864146757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6825","nonce":864146757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6823","nonce":864146757}]},"public_addr":"172.21.15.117:6819/864146757","cluster_addr":"172.21.15.117:6821/864146757","heartbeat_back_addr":"172.21.15.117:6825/864146757","heartbeat_front_addr":"172.21.15.117:6823/864146757","state":["exists","up"]},{"osd":7,"uuid":"929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6817","nonce":1273365545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6819","nonce":1273365545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6823","nonce":1273365545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6821","nonce":1273365545}]},"public_addr":"172.21.15.119:6817/1273365545","cluster_addr":"172.21.15.119:6819/1273365545","heartbeat_back_addr":"172.21.15.119:6823/1273365545","heartbeat_front_addr":"172.21.15.119:6821/1273365545","state":["exists","up"]},{"osd":8,"uuid":"fd371461-a538-4781-92e1-5eead4ca3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6817","nonce":1059587501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6819","nonce":1059587501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6823","nonce":1059587501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6821","nonce":1059587501}]},"public_addr":"172.21.15.178:6817/1059587501","cluster_addr":"172.21.15.178:6819/1059587501","heartbeat_back_addr":"172.21.15.178:6823/1059587501","heartbeat_front_addr":"172.21.15.178:6821/1059587501","state":["exists","up"]},{"osd":9,"uuid":"0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6827","nonce":3653875600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6829","nonce":3653875600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6833","nonce":3653875600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6831","nonce":3653875600}]},"public_addr":"172.21.15.117:6827/3653875600","cluster_addr":"172.21.15.117:6829/3653875600","heartbeat_back_addr":"172.21.15.117:6833/3653875600","heartbeat_front_addr":"172.21.15.117:6831/3653875600","state":["exists","up"]},{"osd":10,"uuid":"de2049da-d19e-4386-afbf-fd6b5fddbcba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6825","nonce":1050976194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6827","nonce":1050976194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6831","nonce":1050976194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6829","nonce":1050976194}]},"public_addr":"172.21.15.119:6825/1050976194","cluster_addr":"172.21.15.119:6827/1050976194","heartbeat_back_addr":"172.21.15.119:6831/1050976194","heartbeat_front_addr":"172.21.15.119:6829/1050976194","state":["exists","up"]},{"osd":11,"uuid":"3cccc0f9-d273-4b6b-9b8e-c926371472dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6825","nonce":2915080368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6827","nonce":2915080368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6831","nonce":2915080368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6829","nonce":2915080368}]},"public_addr":"172.21.15.178:6825/2915080368","cluster_addr":"172.21.15.178:6827/2915080368","heartbeat_back_addr":"172.21.15.178:6831/2915080368","heartbeat_front_addr":"172.21.15.178:6829/2915080368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:27.876274+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:53.094706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:16.598813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:48.160419+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:14.048925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:38.814304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:12.542328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:38.942747+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:04.488963+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:38.196053+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:05.254728+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:30.874915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.117:0/185684447":"2024-03-20T03:22:38.134578+0000","172.21.15.117:0/361367101":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/2788067603":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3506200675":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6800/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3283471541":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3750920169":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/2780813298":"2024-03-20T03:22:03.905461+0000","172.21.15.117:6800/2788067603":"2024-03-20T03:22:38.134578+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:30:52.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.551+0000 7fce50c00640 1 -- 172.21.15.117:0/386508155 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fce300a2880 msgr2=0x7fce300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:52.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.551+0000 7fce50c00640 1 --2- 172.21.15.117:0/386508155 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fce300a2880 0x7fce300a4d40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fce3c009cd0 tx=0x7fce3c005f90 comp rx=0 tx=0).stop 2024-03-19T03:30:52.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.551+0000 7fce50c00640 1 -- 172.21.15.117:0/386508155 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce4c077dc0 msgr2=0x7fce4c1a5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:52.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.551+0000 7fce50c00640 1 --2- 172.21.15.117:0/386508155 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce4c077dc0 0x7fce4c1a5640 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fce44037040 tx=0x7fce44035d10 comp rx=0 tx=0).stop 2024-03-19T03:30:52.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.551+0000 7fce50c00640 1 -- 172.21.15.117:0/386508155 shutdown_connections 2024-03-19T03:30:52.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.551+0000 7fce50c00640 1 --2- 172.21.15.117:0/386508155 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fce300a2880 0x7fce300a4d40 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:52.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.551+0000 7fce50c00640 1 --2- 172.21.15.117:0/386508155 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce4c077dc0 0x7fce4c1a5640 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:52.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.551+0000 7fce50c00640 1 --2- 172.21.15.117:0/386508155 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fce4c077420 0x7fce4c19e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:52.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.551+0000 7fce50c00640 1 --2- 172.21.15.117:0/386508155 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce4c0761e0 0x7fce4c19e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:52.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.551+0000 7fce50c00640 1 -- 172.21.15.117:0/386508155 >> 172.21.15.117:0/386508155 conn(0x7fce4c0fefd0 msgr2=0x7fce4c100650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:52.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.551+0000 7fce50c00640 1 -- 172.21.15.117:0/386508155 shutdown_connections 2024-03-19T03:30:52.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.551+0000 7fce50c00640 1 -- 172.21.15.117:0/386508155 wait complete. 2024-03-19T03:30:52.572 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T03:25:18.824270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T03:30:33.137170+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T03:30:34.598838+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T03:30:38.230023+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T03:30:52.572 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T03:30:52.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.786+0000 7f353f800640 1 -- 172.21.15.117:0/2042986605 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3538102a20 msgr2=0x7f3538102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:52.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.786+0000 7f353f800640 1 --2- 172.21.15.117:0/2042986605 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3538102a20 0x7f3538102ea0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3524009900 tx=0x7f352402bc40 comp rx=0 tx=0).stop 2024-03-19T03:30:52.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.787+0000 7f353f800640 1 -- 172.21.15.117:0/2042986605 shutdown_connections 2024-03-19T03:30:52.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.787+0000 7f353f800640 1 --2- 172.21.15.117:0/2042986605 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35381033e0 0x7f3538109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:52.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.787+0000 7f353f800640 1 --2- 172.21.15.117:0/2042986605 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3538102a20 0x7f3538102ea0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:52.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.787+0000 7f353f800640 1 --2- 172.21.15.117:0/2042986605 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3538101830 0x7f3538101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:52.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.787+0000 7f353f800640 1 -- 172.21.15.117:0/2042986605 >> 172.21.15.117:0/2042986605 conn(0x7f35380fcda0 msgr2=0x7f35380ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:52.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.787+0000 7f353f800640 1 -- 172.21.15.117:0/2042986605 shutdown_connections 2024-03-19T03:30:52.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.787+0000 7f353f800640 1 -- 172.21.15.117:0/2042986605 wait complete. 2024-03-19T03:30:52.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.788+0000 7f353f800640 1 Processor -- start 2024-03-19T03:30:52.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.788+0000 7f353f800640 1 -- start start 2024-03-19T03:30:52.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.788+0000 7f353f800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3538101830 0x7f3538075580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:52.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.788+0000 7f353f800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3538102a20 0x7f3538073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:52.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.788+0000 7f353f800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35381033e0 0x7f3538074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:52.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.788+0000 7f353f800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f353810d4e0 con 0x7f35381033e0 2024-03-19T03:30:52.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.788+0000 7f353f800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f353810d360 con 0x7f3538101830 2024-03-19T03:30:52.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.788+0000 7f353f800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f353810d660 con 0x7f3538102a20 2024-03-19T03:30:52.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.788+0000 7f353ce00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3538101830 0x7f3538075580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:52.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.788+0000 7f353d800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35381033e0 0x7f3538074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:52.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.788+0000 7f352fe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3538102a20 0x7f3538073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:52.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.788+0000 7f353ce00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3538101830 0x7f3538075580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:47572/0 (socket says 172.21.15.117:47572) 2024-03-19T03:30:52.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.788+0000 7f353ce00640 1 -- 172.21.15.117:0/2958223475 learned_addr learned my addr 172.21.15.117:0/2958223475 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:52.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.789+0000 7f353ce00640 1 -- 172.21.15.117:0/2958223475 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3538102a20 msgr2=0x7f3538073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:52.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.789+0000 7f353ce00640 1 --2- 172.21.15.117:0/2958223475 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3538102a20 0x7f3538073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:52.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.789+0000 7f353ce00640 1 -- 172.21.15.117:0/2958223475 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35381033e0 msgr2=0x7f3538074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:52.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.789+0000 7f353ce00640 1 --2- 172.21.15.117:0/2958223475 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35381033e0 0x7f3538074150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:52.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.789+0000 7f353ce00640 1 -- 172.21.15.117:0/2958223475 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3538074a10 con 0x7f3538101830 2024-03-19T03:30:52.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.789+0000 7f353ce00640 1 --2- 172.21.15.117:0/2958223475 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3538101830 0x7f3538075580 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f352800e9c0 tx=0x7f352800edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:52.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.789+0000 7f3536000640 1 -- 172.21.15.117:0/2958223475 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352800cd80 con 0x7f3538101830 2024-03-19T03:30:52.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.789+0000 7f3536000640 1 -- 172.21.15.117:0/2958223475 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3528010910 con 0x7f3538101830 2024-03-19T03:30:52.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.789+0000 7f3536000640 1 -- 172.21.15.117:0/2958223475 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528018950 con 0x7f3538101830 2024-03-19T03:30:52.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.789+0000 7f353f800640 1 -- 172.21.15.117:0/2958223475 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35381a1cd0 con 0x7f3538101830 2024-03-19T03:30:52.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.789+0000 7f353f800640 1 -- 172.21.15.117:0/2958223475 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35381a2190 con 0x7f3538101830 2024-03-19T03:30:52.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.791+0000 7f353f800640 1 -- 172.21.15.117:0/2958223475 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3538108bf0 con 0x7f3538101830 2024-03-19T03:30:52.792 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.792+0000 7f3536000640 1 -- 172.21.15.117:0/2958223475 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f352801f020 con 0x7f3538101830 2024-03-19T03:30:52.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.793+0000 7f3536000640 1 --2- 172.21.15.117:0/2958223475 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f351c0a2880 0x7f351c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:52.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.793+0000 7f3536000640 1 -- 172.21.15.117:0/2958223475 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f35280ceca0 con 0x7f3538101830 2024-03-19T03:30:52.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.794+0000 7f352fe00640 1 --2- 172.21.15.117:0/2958223475 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f351c0a2880 0x7f351c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:52.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.794+0000 7f352fe00640 1 --2- 172.21.15.117:0/2958223475 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f351c0a2880 0x7f351c0a4d40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f352402c250 tx=0x7f352402cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:52.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.798+0000 7f3536000640 1 -- 172.21.15.117:0/2958223475 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f352809bb00 con 0x7f3538101830 2024-03-19T03:30:52.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.995+0000 7f353f800640 1 -- 172.21.15.117:0/2958223475 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3538062420 con 0x7f3538101830 2024-03-19T03:30:52.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.996+0000 7f3536000640 1 -- 172.21.15.117:0/2958223475 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f352809b400 con 0x7f3538101830 2024-03-19T03:30:52.996 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 1 2024-03-19T03:30:52.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.998+0000 7f353f800640 1 -- 172.21.15.117:0/2958223475 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f351c0a2880 msgr2=0x7f351c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:52.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.998+0000 7f353f800640 1 --2- 172.21.15.117:0/2958223475 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f351c0a2880 0x7f351c0a4d40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f352402c250 tx=0x7f352402cf90 comp rx=0 tx=0).stop 2024-03-19T03:30:52.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.999+0000 7f353f800640 1 -- 172.21.15.117:0/2958223475 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3538101830 msgr2=0x7f3538075580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:52.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.999+0000 7f353f800640 1 --2- 172.21.15.117:0/2958223475 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3538101830 0x7f3538075580 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f352800e9c0 tx=0x7f352800edc0 comp rx=0 tx=0).stop 2024-03-19T03:30:52.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.999+0000 7f353f800640 1 -- 172.21.15.117:0/2958223475 shutdown_connections 2024-03-19T03:30:52.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.999+0000 7f353f800640 1 --2- 172.21.15.117:0/2958223475 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35381033e0 0x7f3538074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:52.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.999+0000 7f353f800640 1 --2- 172.21.15.117:0/2958223475 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3538102a20 0x7f3538073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:52.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.999+0000 7f353f800640 1 --2- 172.21.15.117:0/2958223475 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f351c0a2880 0x7f351c0a4d40 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:52.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.999+0000 7f353f800640 1 --2- 172.21.15.117:0/2958223475 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3538101830 0x7f3538075580 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:52.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.999+0000 7f353f800640 1 -- 172.21.15.117:0/2958223475 >> 172.21.15.117:0/2958223475 conn(0x7f35380fcda0 msgr2=0x7f3538105dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:52.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.999+0000 7f353f800640 1 -- 172.21.15.117:0/2958223475 shutdown_connections 2024-03-19T03:30:52.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:52.999+0000 7f353f800640 1 -- 172.21.15.117:0/2958223475 wait complete. 2024-03-19T03:30:53.018 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T03:30:53.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.237+0000 7fd281a00640 1 -- 172.21.15.117:0/2448314188 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd27c077420 msgr2=0x7fd27c077880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:53.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.237+0000 7fd281a00640 1 --2- 172.21.15.117:0/2448314188 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd27c077420 0x7fd27c077880 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd264009900 tx=0x7fd26402bc30 comp rx=0 tx=0).stop 2024-03-19T03:30:53.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.238+0000 7fd281a00640 1 -- 172.21.15.117:0/2448314188 shutdown_connections 2024-03-19T03:30:53.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.238+0000 7fd281a00640 1 --2- 172.21.15.117:0/2448314188 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd27c077dc0 0x7fd27c078660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:53.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.238+0000 7fd281a00640 1 --2- 172.21.15.117:0/2448314188 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd27c077420 0x7fd27c077880 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:53.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.238+0000 7fd281a00640 1 --2- 172.21.15.117:0/2448314188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd27c0761e0 0x7fd27c076600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:53.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.238+0000 7fd281a00640 1 -- 172.21.15.117:0/2448314188 >> 172.21.15.117:0/2448314188 conn(0x7fd27c0fefd0 msgr2=0x7fd27c101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:53.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.238+0000 7fd281a00640 1 -- 172.21.15.117:0/2448314188 shutdown_connections 2024-03-19T03:30:53.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.238+0000 7fd281a00640 1 -- 172.21.15.117:0/2448314188 wait complete. 2024-03-19T03:30:53.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.239+0000 7fd281a00640 1 Processor -- start 2024-03-19T03:30:53.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.239+0000 7fd281a00640 1 -- start start 2024-03-19T03:30:53.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.239+0000 7fd281a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd27c0761e0 0x7fd27c19e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:53.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.239+0000 7fd281a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd27c077420 0x7fd27c19e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:53.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.240+0000 7fd281a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd27c077dc0 0x7fd27c1a5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:53.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.240+0000 7fd281a00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd27c113af0 con 0x7fd27c0761e0 2024-03-19T03:30:53.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.240+0000 7fd281a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd27c113970 con 0x7fd27c077dc0 2024-03-19T03:30:53.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.240+0000 7fd281a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fd27c113c70 con 0x7fd27c077420 2024-03-19T03:30:53.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.240+0000 7fd27aa00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd27c0761e0 0x7fd27c19e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:53.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.240+0000 7fd27aa00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd27c0761e0 0x7fd27c19e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:47826/0 (socket says 172.21.15.117:47826) 2024-03-19T03:30:53.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.240+0000 7fd27aa00640 1 -- 172.21.15.117:0/3563698861 learned_addr learned my addr 172.21.15.117:0/3563698861 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:53.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.240+0000 7fd27b400640 1 --2- 172.21.15.117:0/3563698861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd27c077dc0 0x7fd27c1a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:53.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.240+0000 7fd272000640 1 --2- 172.21.15.117:0/3563698861 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd27c077420 0x7fd27c19e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:53.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.240+0000 7fd27b400640 1 -- 172.21.15.117:0/3563698861 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd27c077420 msgr2=0x7fd27c19e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:53.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.240+0000 7fd27b400640 1 --2- 172.21.15.117:0/3563698861 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd27c077420 0x7fd27c19e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:53.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.240+0000 7fd27b400640 1 -- 172.21.15.117:0/3563698861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd27c0761e0 msgr2=0x7fd27c19e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:53.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.240+0000 7fd27b400640 1 --2- 172.21.15.117:0/3563698861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd27c0761e0 0x7fd27c19e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:53.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.240+0000 7fd27b400640 1 -- 172.21.15.117:0/3563698861 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd27c1a5dc0 con 0x7fd27c077dc0 2024-03-19T03:30:53.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.240+0000 7fd27aa00640 1 --2- 172.21.15.117:0/3563698861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd27c0761e0 0x7fd27c19e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:30:53.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.240+0000 7fd27b400640 1 --2- 172.21.15.117:0/3563698861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd27c077dc0 0x7fd27c1a5750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd274009fd0 tx=0x7fd27400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:53.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.241+0000 7fd273e00640 1 -- 172.21.15.117:0/3563698861 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27400ce20 con 0x7fd27c077dc0 2024-03-19T03:30:53.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.241+0000 7fd273e00640 1 -- 172.21.15.117:0/3563698861 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd274005020 con 0x7fd27c077dc0 2024-03-19T03:30:53.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.241+0000 7fd281a00640 1 -- 172.21.15.117:0/3563698861 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd27c1a60b0 con 0x7fd27c077dc0 2024-03-19T03:30:53.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.241+0000 7fd273e00640 1 -- 172.21.15.117:0/3563698861 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd274010720 con 0x7fd27c077dc0 2024-03-19T03:30:53.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.241+0000 7fd281a00640 1 -- 172.21.15.117:0/3563698861 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd27c1a65d0 con 0x7fd27c077dc0 2024-03-19T03:30:53.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.242+0000 7fd281a00640 1 -- 172.21.15.117:0/3563698861 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd27c1103d0 con 0x7fd27c077dc0 2024-03-19T03:30:53.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.244+0000 7fd273e00640 1 -- 172.21.15.117:0/3563698861 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fd274005540 con 0x7fd27c077dc0 2024-03-19T03:30:53.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.245+0000 7fd273e00640 1 --2- 172.21.15.117:0/3563698861 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd2600a2880 0x7fd2600a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:53.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.245+0000 7fd273e00640 1 -- 172.21.15.117:0/3563698861 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd2740cf390 con 0x7fd27c077dc0 2024-03-19T03:30:53.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.245+0000 7fd27aa00640 1 --2- 172.21.15.117:0/3563698861 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd2600a2880 0x7fd2600a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:53.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.246+0000 7fd27aa00640 1 --2- 172.21.15.117:0/3563698861 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd2600a2880 0x7fd2600a4d40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd26c011500 tx=0x7fd26c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:53.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.249+0000 7fd273e00640 1 -- 172.21.15.117:0/3563698861 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd27409c190 con 0x7fd27c077dc0 2024-03-19T03:30:53.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.452+0000 7fd281a00640 1 -- 172.21.15.117:0/3563698861 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd27c062420 con 0x7fd27c077dc0 2024-03-19T03:30:53.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.452+0000 7fd273e00640 1 -- 172.21.15.117:0/3563698861 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7fd27409ba90 con 0x7fd27c077dc0 2024-03-19T03:30:53.453 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 32 2024-03-19T03:30:53.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.455+0000 7fd281a00640 1 -- 172.21.15.117:0/3563698861 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd2600a2880 msgr2=0x7fd2600a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:53.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.455+0000 7fd281a00640 1 --2- 172.21.15.117:0/3563698861 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd2600a2880 0x7fd2600a4d40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd26c011500 tx=0x7fd26c005f90 comp rx=0 tx=0).stop 2024-03-19T03:30:53.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.455+0000 7fd281a00640 1 -- 172.21.15.117:0/3563698861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd27c077dc0 msgr2=0x7fd27c1a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:53.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.455+0000 7fd281a00640 1 --2- 172.21.15.117:0/3563698861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd27c077dc0 0x7fd27c1a5750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd274009fd0 tx=0x7fd27400ef10 comp rx=0 tx=0).stop 2024-03-19T03:30:53.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.455+0000 7fd281a00640 1 -- 172.21.15.117:0/3563698861 shutdown_connections 2024-03-19T03:30:53.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.455+0000 7fd281a00640 1 --2- 172.21.15.117:0/3563698861 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd2600a2880 0x7fd2600a4d40 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:53.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.455+0000 7fd281a00640 1 --2- 172.21.15.117:0/3563698861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd27c077dc0 0x7fd27c1a5750 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:53.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.455+0000 7fd281a00640 1 --2- 172.21.15.117:0/3563698861 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd27c077420 0x7fd27c19e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:53.456 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.455+0000 7fd281a00640 1 --2- 172.21.15.117:0/3563698861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd27c0761e0 0x7fd27c19e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:53.456 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.455+0000 7fd281a00640 1 -- 172.21.15.117:0/3563698861 >> 172.21.15.117:0/3563698861 conn(0x7fd27c0fefd0 msgr2=0x7fd27c100ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:53.456 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.455+0000 7fd281a00640 1 -- 172.21.15.117:0/3563698861 shutdown_connections 2024-03-19T03:30:53.456 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.455+0000 7fd281a00640 1 -- 172.21.15.117:0/3563698861 wait complete. 2024-03-19T03:30:53.474 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T03:30:53.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.690+0000 7f1cbe600640 1 -- 172.21.15.117:0/2482647208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cb8104c50 msgr2=0x7f1cb81050d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:53.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.690+0000 7f1cbe600640 1 --2- 172.21.15.117:0/2482647208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cb8104c50 0x7f1cb81050d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1ca0009ca0 tx=0x7f1ca002c100 comp rx=0 tx=0).stop 2024-03-19T03:30:53.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.690+0000 7f1cbe600640 1 -- 172.21.15.117:0/2482647208 shutdown_connections 2024-03-19T03:30:53.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.690+0000 7f1cbe600640 1 --2- 172.21.15.117:0/2482647208 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1cb8105610 0x7f1cb810bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:53.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.690+0000 7f1cbe600640 1 --2- 172.21.15.117:0/2482647208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cb8104c50 0x7f1cb81050d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:53.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.690+0000 7f1cbe600640 1 --2- 172.21.15.117:0/2482647208 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1cb8103a60 0x7f1cb8103e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:53.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.690+0000 7f1cbe600640 1 -- 172.21.15.117:0/2482647208 >> 172.21.15.117:0/2482647208 conn(0x7f1cb80fefd0 msgr2=0x7f1cb8101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:53.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.690+0000 7f1cbe600640 1 -- 172.21.15.117:0/2482647208 shutdown_connections 2024-03-19T03:30:53.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.690+0000 7f1cbe600640 1 -- 172.21.15.117:0/2482647208 wait complete. 2024-03-19T03:30:53.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.691+0000 7f1cbe600640 1 Processor -- start 2024-03-19T03:30:53.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.691+0000 7f1cbe600640 1 -- start start 2024-03-19T03:30:53.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.691+0000 7f1cbe600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1cb8103a60 0x7f1cb8199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:53.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.691+0000 7f1cbe600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cb8104c50 0x7f1cb819a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:53.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.691+0000 7f1cbe600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1cb8105610 0x7f1cb81a1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:53.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.691+0000 7f1cbe600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb810f760 con 0x7f1cb8103a60 2024-03-19T03:30:53.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.691+0000 7f1cbe600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb810f5e0 con 0x7f1cb8104c50 2024-03-19T03:30:53.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.691+0000 7f1cbe600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb810f8e0 con 0x7f1cb8105610 2024-03-19T03:30:53.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.692+0000 7f1cb6a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cb8104c50 0x7f1cb819a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:53.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.692+0000 7f1cb6a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cb8104c50 0x7f1cb819a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:47600/0 (socket says 172.21.15.117:47600) 2024-03-19T03:30:53.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.692+0000 7f1cb6a00640 1 -- 172.21.15.117:0/495260649 learned_addr learned my addr 172.21.15.117:0/495260649 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:53.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.692+0000 7f1cb7400640 1 --2- 172.21.15.117:0/495260649 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1cb8103a60 0x7f1cb8199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:53.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.692+0000 7f1cb7e00640 1 --2- 172.21.15.117:0/495260649 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1cb8105610 0x7f1cb81a1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:53.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.692+0000 7f1cb6a00640 1 -- 172.21.15.117:0/495260649 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1cb8105610 msgr2=0x7f1cb81a1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:53.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.692+0000 7f1cb6a00640 1 --2- 172.21.15.117:0/495260649 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1cb8105610 0x7f1cb81a1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:53.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.692+0000 7f1cb6a00640 1 -- 172.21.15.117:0/495260649 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1cb8103a60 msgr2=0x7f1cb8199dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:53.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.692+0000 7f1cb6a00640 1 --2- 172.21.15.117:0/495260649 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1cb8103a60 0x7f1cb8199dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:53.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.692+0000 7f1cb6a00640 1 -- 172.21.15.117:0/495260649 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cb81a19f0 con 0x7f1cb8104c50 2024-03-19T03:30:53.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.692+0000 7f1cb7e00640 1 --2- 172.21.15.117:0/495260649 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1cb8105610 0x7f1cb81a1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:30:53.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.692+0000 7f1cb6a00640 1 --2- 172.21.15.117:0/495260649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cb8104c50 0x7f1cb819a310 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1ca00325f0 tx=0x7f1ca0035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:53.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.693+0000 7f1cafe00640 1 -- 172.21.15.117:0/495260649 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca0039040 con 0x7f1cb8104c50 2024-03-19T03:30:53.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.693+0000 7f1cafe00640 1 -- 172.21.15.117:0/495260649 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ca0044070 con 0x7f1cb8104c50 2024-03-19T03:30:53.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.693+0000 7f1cbe600640 1 -- 172.21.15.117:0/495260649 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cb81a1c80 con 0x7f1cb8104c50 2024-03-19T03:30:53.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.693+0000 7f1cafe00640 1 -- 172.21.15.117:0/495260649 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca0004610 con 0x7f1cb8104c50 2024-03-19T03:30:53.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.693+0000 7f1cbe600640 1 -- 172.21.15.117:0/495260649 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cb81a2190 con 0x7f1cb8104c50 2024-03-19T03:30:53.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.694+0000 7f1cbe600640 1 -- 172.21.15.117:0/495260649 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cb810ae00 con 0x7f1cb8104c50 2024-03-19T03:30:53.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.696+0000 7f1cafe00640 1 -- 172.21.15.117:0/495260649 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f1ca0003970 con 0x7f1cb8104c50 2024-03-19T03:30:53.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.697+0000 7f1cafe00640 1 --2- 172.21.15.117:0/495260649 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1c9c0a2880 0x7f1c9c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:53.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.697+0000 7f1cafe00640 1 -- 172.21.15.117:0/495260649 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f1ca00f1400 con 0x7f1cb8104c50 2024-03-19T03:30:53.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.697+0000 7f1cb7400640 1 --2- 172.21.15.117:0/495260649 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1c9c0a2880 0x7f1c9c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:53.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.698+0000 7f1cb7400640 1 --2- 172.21.15.117:0/495260649 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1c9c0a2880 0x7f1c9c0a4d40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0005fd0 tx=0x7f1cb000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:53.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.701+0000 7f1cafe00640 1 -- 172.21.15.117:0/495260649 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1ca00be200 con 0x7f1cb8104c50 2024-03-19T03:30:53.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:53 smithi119 ceph-mon[66861]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-19T03:30:53.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:53 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/386508155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:30:53.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:53 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2958223475' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:30:53.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:53 smithi117 ceph-mon[31838]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-19T03:30:53.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:53 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/386508155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:30:53.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:53 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2958223475' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:30:53.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:53 smithi178 ceph-mon[67578]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-19T03:30:53.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:53 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/386508155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:30:53.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:53 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2958223475' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:30:53.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.898+0000 7f1cbe600640 1 -- 172.21.15.117:0/495260649 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1cb8062420 con 0x7f1cb8104c50 2024-03-19T03:30:53.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.899+0000 7f1cafe00640 1 -- 172.21.15.117:0/495260649 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f1ca00bdb00 con 0x7f1cb8104c50 2024-03-19T03:30:53.899 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 64 2024-03-19T03:30:53.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.902+0000 7f1cbe600640 1 -- 172.21.15.117:0/495260649 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1c9c0a2880 msgr2=0x7f1c9c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:53.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.902+0000 7f1cbe600640 1 --2- 172.21.15.117:0/495260649 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1c9c0a2880 0x7f1c9c0a4d40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0005fd0 tx=0x7f1cb000b390 comp rx=0 tx=0).stop 2024-03-19T03:30:53.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.902+0000 7f1cbe600640 1 -- 172.21.15.117:0/495260649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cb8104c50 msgr2=0x7f1cb819a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:53.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.902+0000 7f1cbe600640 1 --2- 172.21.15.117:0/495260649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cb8104c50 0x7f1cb819a310 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1ca00325f0 tx=0x7f1ca0035d10 comp rx=0 tx=0).stop 2024-03-19T03:30:53.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.902+0000 7f1cbe600640 1 -- 172.21.15.117:0/495260649 shutdown_connections 2024-03-19T03:30:53.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.902+0000 7f1cbe600640 1 --2- 172.21.15.117:0/495260649 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1cb8105610 0x7f1cb81a1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:53.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.902+0000 7f1cbe600640 1 --2- 172.21.15.117:0/495260649 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1c9c0a2880 0x7f1c9c0a4d40 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:53.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.902+0000 7f1cbe600640 1 --2- 172.21.15.117:0/495260649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cb8104c50 0x7f1cb819a310 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:53.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.902+0000 7f1cbe600640 1 --2- 172.21.15.117:0/495260649 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1cb8103a60 0x7f1cb8199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:53.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.902+0000 7f1cbe600640 1 -- 172.21.15.117:0/495260649 >> 172.21.15.117:0/495260649 conn(0x7f1cb80fefd0 msgr2=0x7f1cb8107fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:53.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.902+0000 7f1cbe600640 1 -- 172.21.15.117:0/495260649 shutdown_connections 2024-03-19T03:30:53.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:53.902+0000 7f1cbe600640 1 -- 172.21.15.117:0/495260649 wait complete. 2024-03-19T03:30:53.921 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T03:30:54.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.130+0000 7ff9a4800640 1 -- 172.21.15.117:0/2285598705 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff99c102a20 msgr2=0x7ff99c102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:54.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.130+0000 7ff9a4800640 1 --2- 172.21.15.117:0/2285598705 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff99c102a20 0x7ff99c102ea0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff98c009900 tx=0x7ff98c02bc30 comp rx=0 tx=0).stop 2024-03-19T03:30:54.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.131+0000 7ff9a4800640 1 -- 172.21.15.117:0/2285598705 shutdown_connections 2024-03-19T03:30:54.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.131+0000 7ff9a4800640 1 --2- 172.21.15.117:0/2285598705 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff99c1033e0 0x7ff99c109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:54.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.131+0000 7ff9a4800640 1 --2- 172.21.15.117:0/2285598705 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff99c102a20 0x7ff99c102ea0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:54.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.131+0000 7ff9a4800640 1 --2- 172.21.15.117:0/2285598705 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff99c101830 0x7ff99c101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:54.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.131+0000 7ff9a4800640 1 -- 172.21.15.117:0/2285598705 >> 172.21.15.117:0/2285598705 conn(0x7ff99c0fcda0 msgr2=0x7ff99c0ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:54.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.131+0000 7ff9a4800640 1 -- 172.21.15.117:0/2285598705 shutdown_connections 2024-03-19T03:30:54.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.131+0000 7ff9a4800640 1 -- 172.21.15.117:0/2285598705 wait complete. 2024-03-19T03:30:54.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.132+0000 7ff9a4800640 1 Processor -- start 2024-03-19T03:30:54.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.132+0000 7ff9a4800640 1 -- start start 2024-03-19T03:30:54.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.132+0000 7ff9a4800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff99c101830 0x7ff99c197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:54.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.132+0000 7ff9a4800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff99c102a20 0x7ff99c1980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:54.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.132+0000 7ff9a4800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff99c1033e0 0x7ff99c19f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:54.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.132+0000 7ff9a4800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff99c10d4b0 con 0x7ff99c102a20 2024-03-19T03:30:54.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.132+0000 7ff9a4800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ff99c10d330 con 0x7ff99c101830 2024-03-19T03:30:54.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.132+0000 7ff9a4800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7ff99c10d630 con 0x7ff99c1033e0 2024-03-19T03:30:54.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.133+0000 7ff9a1e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff99c101830 0x7ff99c197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:54.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.133+0000 7ff9a1e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff99c101830 0x7ff99c197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:47624/0 (socket says 172.21.15.117:47624) 2024-03-19T03:30:54.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.133+0000 7ff9a1e00640 1 -- 172.21.15.117:0/835081362 learned_addr learned my addr 172.21.15.117:0/835081362 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:54.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.133+0000 7ff9a2800640 1 --2- 172.21.15.117:0/835081362 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff99c1033e0 0x7ff99c19f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:54.135 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.133+0000 7ff9a1400640 1 --2- 172.21.15.117:0/835081362 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff99c102a20 0x7ff99c1980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:54.135 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.133+0000 7ff9a1e00640 1 -- 172.21.15.117:0/835081362 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff99c1033e0 msgr2=0x7ff99c19f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:54.135 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.133+0000 7ff9a1e00640 1 --2- 172.21.15.117:0/835081362 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff99c1033e0 0x7ff99c19f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:54.135 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.133+0000 7ff9a1e00640 1 -- 172.21.15.117:0/835081362 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff99c102a20 msgr2=0x7ff99c1980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:54.135 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.133+0000 7ff9a1e00640 1 --2- 172.21.15.117:0/835081362 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff99c102a20 0x7ff99c1980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:54.135 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.133+0000 7ff9a1e00640 1 -- 172.21.15.117:0/835081362 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff99c19f7b0 con 0x7ff99c101830 2024-03-19T03:30:54.135 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.133+0000 7ff9a1400640 1 --2- 172.21.15.117:0/835081362 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff99c102a20 0x7ff99c1980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T03:30:54.135 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.133+0000 7ff9a2800640 1 --2- 172.21.15.117:0/835081362 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff99c1033e0 0x7ff99c19f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:30:54.135 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.133+0000 7ff9a1e00640 1 --2- 172.21.15.117:0/835081362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff99c101830 0x7ff99c197b90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff99800eaa0 tx=0x7ff99800ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:54.135 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.133+0000 7ff992a00640 1 -- 172.21.15.117:0/835081362 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff998016070 con 0x7ff99c101830 2024-03-19T03:30:54.135 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.134+0000 7ff9a4800640 1 -- 172.21.15.117:0/835081362 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff99c19faa0 con 0x7ff99c101830 2024-03-19T03:30:54.135 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.134+0000 7ff992a00640 1 -- 172.21.15.117:0/835081362 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff998003e50 con 0x7ff99c101830 2024-03-19T03:30:54.136 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.134+0000 7ff992a00640 1 -- 172.21.15.117:0/835081362 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9980116d0 con 0x7ff99c101830 2024-03-19T03:30:54.136 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.134+0000 7ff9a4800640 1 -- 172.21.15.117:0/835081362 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff99c19ffa0 con 0x7ff99c101830 2024-03-19T03:30:54.136 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.135+0000 7ff9a4800640 1 -- 172.21.15.117:0/835081362 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff99c108bd0 con 0x7ff99c101830 2024-03-19T03:30:54.136 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.136+0000 7ff992a00640 1 -- 172.21.15.117:0/835081362 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7ff998019070 con 0x7ff99c101830 2024-03-19T03:30:54.138 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.138+0000 7ff992a00640 1 --2- 172.21.15.117:0/835081362 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7ff9880a2800 0x7ff9880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:54.138 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.138+0000 7ff992a00640 1 -- 172.21.15.117:0/835081362 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff9980cee90 con 0x7ff99c101830 2024-03-19T03:30:54.138 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.138+0000 7ff9a1400640 1 --2- 172.21.15.117:0/835081362 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7ff9880a2800 0x7ff9880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:54.138 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.139+0000 7ff9a1400640 1 --2- 172.21.15.117:0/835081362 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7ff9880a2800 0x7ff9880a4cc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff98c036040 tx=0x7ff98c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:54.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.142+0000 7ff992a00640 1 -- 172.21.15.117:0/835081362 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff99809bc90 con 0x7ff99c101830 2024-03-19T03:30:54.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.341+0000 7ff9a4800640 1 -- 172.21.15.117:0/835081362 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ff99c062420 con 0x7ff99c101830 2024-03-19T03:30:54.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.341+0000 7ff992a00640 1 -- 172.21.15.117:0/835081362 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7ff99809b590 con 0x7ff99c101830 2024-03-19T03:30:54.341 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 32 2024-03-19T03:30:54.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.344+0000 7ff9a4800640 1 -- 172.21.15.117:0/835081362 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7ff9880a2800 msgr2=0x7ff9880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:54.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.344+0000 7ff9a4800640 1 --2- 172.21.15.117:0/835081362 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7ff9880a2800 0x7ff9880a4cc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff98c036040 tx=0x7ff98c000bc0 comp rx=0 tx=0).stop 2024-03-19T03:30:54.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.344+0000 7ff9a4800640 1 -- 172.21.15.117:0/835081362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff99c101830 msgr2=0x7ff99c197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:54.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.344+0000 7ff9a4800640 1 --2- 172.21.15.117:0/835081362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff99c101830 0x7ff99c197b90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff99800eaa0 tx=0x7ff99800ef60 comp rx=0 tx=0).stop 2024-03-19T03:30:54.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.344+0000 7ff9a4800640 1 -- 172.21.15.117:0/835081362 shutdown_connections 2024-03-19T03:30:54.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.344+0000 7ff9a4800640 1 --2- 172.21.15.117:0/835081362 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff99c1033e0 0x7ff99c19f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:54.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.344+0000 7ff9a4800640 1 --2- 172.21.15.117:0/835081362 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff99c102a20 0x7ff99c1980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:54.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.344+0000 7ff9a4800640 1 --2- 172.21.15.117:0/835081362 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7ff9880a2800 0x7ff9880a4cc0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:54.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.344+0000 7ff9a4800640 1 --2- 172.21.15.117:0/835081362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff99c101830 0x7ff99c197b90 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:54.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.344+0000 7ff9a4800640 1 -- 172.21.15.117:0/835081362 >> 172.21.15.117:0/835081362 conn(0x7ff99c0fcda0 msgr2=0x7ff99c105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:54.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.344+0000 7ff9a4800640 1 -- 172.21.15.117:0/835081362 shutdown_connections 2024-03-19T03:30:54.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.344+0000 7ff9a4800640 1 -- 172.21.15.117:0/835081362 wait complete. 2024-03-19T03:30:54.364 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-19T03:30:54.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.569+0000 7f9614800640 1 -- 172.21.15.117:0/1834290332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f960c102a30 msgr2=0x7f960c102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:54.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.569+0000 7f9614800640 1 --2- 172.21.15.117:0/1834290332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f960c102a30 0x7f960c102eb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f95fc009860 tx=0x7f95fc02bb70 comp rx=0 tx=0).stop 2024-03-19T03:30:54.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.569+0000 7f9614800640 1 -- 172.21.15.117:0/1834290332 shutdown_connections 2024-03-19T03:30:54.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.569+0000 7f9614800640 1 --2- 172.21.15.117:0/1834290332 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f960c1033f0 0x7f960c109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:54.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.569+0000 7f9614800640 1 --2- 172.21.15.117:0/1834290332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f960c102a30 0x7f960c102eb0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:54.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.569+0000 7f9614800640 1 --2- 172.21.15.117:0/1834290332 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f960c101840 0x7f960c101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:54.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.569+0000 7f9614800640 1 -- 172.21.15.117:0/1834290332 >> 172.21.15.117:0/1834290332 conn(0x7f960c0fcdd0 msgr2=0x7f960c0ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:54.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.569+0000 7f9614800640 1 -- 172.21.15.117:0/1834290332 shutdown_connections 2024-03-19T03:30:54.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.569+0000 7f9614800640 1 -- 172.21.15.117:0/1834290332 wait complete. 2024-03-19T03:30:54.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.570+0000 7f9614800640 1 Processor -- start 2024-03-19T03:30:54.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.570+0000 7f9614800640 1 -- start start 2024-03-19T03:30:54.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.570+0000 7f9614800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f960c101840 0x7f960c197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:54.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.570+0000 7f9614800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f960c102a30 0x7f960c1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:54.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.570+0000 7f9614800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f960c1033f0 0x7f960c19f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:54.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.570+0000 7f9614800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f960c10d4c0 con 0x7f960c1033f0 2024-03-19T03:30:54.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.570+0000 7f9614800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f960c10d340 con 0x7f960c102a30 2024-03-19T03:30:54.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.570+0000 7f9614800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f960c10d640 con 0x7f960c101840 2024-03-19T03:30:54.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.570+0000 7f9611400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f960c102a30 0x7f960c1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:54.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.571+0000 7f9611400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f960c102a30 0x7f960c1980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:47630/0 (socket says 172.21.15.117:47630) 2024-03-19T03:30:54.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.571+0000 7f9611400640 1 -- 172.21.15.117:0/1715589956 learned_addr learned my addr 172.21.15.117:0/1715589956 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:54.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.571+0000 7f9612800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f960c1033f0 0x7f960c19f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:54.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.571+0000 7f9611e00640 1 --2- 172.21.15.117:0/1715589956 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f960c101840 0x7f960c197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:54.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.571+0000 7f9611400640 1 -- 172.21.15.117:0/1715589956 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f960c101840 msgr2=0x7f960c197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:54.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.571+0000 7f9611400640 1 --2- 172.21.15.117:0/1715589956 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f960c101840 0x7f960c197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:54.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.571+0000 7f9611400640 1 -- 172.21.15.117:0/1715589956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f960c1033f0 msgr2=0x7f960c19f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:54.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.571+0000 7f9611400640 1 --2- 172.21.15.117:0/1715589956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f960c1033f0 0x7f960c19f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:54.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.571+0000 7f9611400640 1 -- 172.21.15.117:0/1715589956 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f960c19f790 con 0x7f960c102a30 2024-03-19T03:30:54.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.571+0000 7f9611400640 1 --2- 172.21.15.117:0/1715589956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f960c102a30 0x7f960c1980b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f95fc0349e0 tx=0x7f95fc032450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:54.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.571+0000 7f9602a00640 1 -- 172.21.15.117:0/1715589956 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95fc041030 con 0x7f960c102a30 2024-03-19T03:30:54.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.571+0000 7f9602a00640 1 -- 172.21.15.117:0/1715589956 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95fc02c510 con 0x7f960c102a30 2024-03-19T03:30:54.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.571+0000 7f9602a00640 1 -- 172.21.15.117:0/1715589956 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95fc039040 con 0x7f960c102a30 2024-03-19T03:30:54.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.571+0000 7f9614800640 1 -- 172.21.15.117:0/1715589956 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f960c19fa20 con 0x7f960c102a30 2024-03-19T03:30:54.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.572+0000 7f9614800640 1 -- 172.21.15.117:0/1715589956 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f960c19ff60 con 0x7f960c102a30 2024-03-19T03:30:54.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.573+0000 7f9614800640 1 -- 172.21.15.117:0/1715589956 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f960c108b60 con 0x7f960c102a30 2024-03-19T03:30:54.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.574+0000 7f9602a00640 1 -- 172.21.15.117:0/1715589956 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f95fc044070 con 0x7f960c102a30 2024-03-19T03:30:54.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.575+0000 7f9602a00640 1 --2- 172.21.15.117:0/1715589956 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f95f80a2880 0x7f95f80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:54.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.576+0000 7f9602a00640 1 -- 172.21.15.117:0/1715589956 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f95fc0f0260 con 0x7f960c102a30 2024-03-19T03:30:54.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.576+0000 7f9611e00640 1 --2- 172.21.15.117:0/1715589956 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f95f80a2880 0x7f95f80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:54.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.576+0000 7f9611e00640 1 --2- 172.21.15.117:0/1715589956 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f95f80a2880 0x7f95f80a4d40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f960c102890 tx=0x7f96040095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:54.579 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.580+0000 7f9602a00640 1 -- 172.21.15.117:0/1715589956 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f95fc0bd060 con 0x7f960c102a30 2024-03-19T03:30:54.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:54 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3563698861' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:30:54.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:54 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/495260649' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:30:54.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:54 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/835081362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:30:54.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:54 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3563698861' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:30:54.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:54 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/495260649' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:30:54.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:54 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/835081362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:30:54.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.793+0000 7f9614800640 1 -- 172.21.15.117:0/1715589956 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f960c02d1f0 con 0x7f960c102a30 2024-03-19T03:30:54.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.794+0000 7f9602a00640 1 -- 172.21.15.117:0/1715589956 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+600 (secure 0 0 0) 0x7f95fc0bc960 con 0x7f960c102a30 2024-03-19T03:30:54.794 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:0/185684447 2024-03-20T03:22:38.134578+0000 2024-03-19T03:30:54.794 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:0/361367101 2024-03-20T03:22:38.134578+0000 2024-03-19T03:30:54.794 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:6801/2788067603 2024-03-20T03:22:38.134578+0000 2024-03-19T03:30:54.794 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:6801/1570899077 2024-03-20T03:22:03.905461+0000 2024-03-19T03:30:54.794 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:0/3506200675 2024-03-20T03:22:38.134578+0000 2024-03-19T03:30:54.794 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:6800/1570899077 2024-03-20T03:22:03.905461+0000 2024-03-19T03:30:54.794 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:0/3283471541 2024-03-20T03:22:03.905461+0000 2024-03-19T03:30:54.794 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:0/3750920169 2024-03-20T03:22:03.905461+0000 2024-03-19T03:30:54.794 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:0/2780813298 2024-03-20T03:22:03.905461+0000 2024-03-19T03:30:54.794 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:6800/2788067603 2024-03-20T03:22:38.134578+0000 2024-03-19T03:30:54.795 INFO:teuthology.orchestra.run.smithi117.stderr:listed 10 entries 2024-03-19T03:30:54.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.797+0000 7f9614800640 1 -- 172.21.15.117:0/1715589956 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f95f80a2880 msgr2=0x7f95f80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:54.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.797+0000 7f9614800640 1 --2- 172.21.15.117:0/1715589956 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f95f80a2880 0x7f95f80a4d40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f960c102890 tx=0x7f96040095c0 comp rx=0 tx=0).stop 2024-03-19T03:30:54.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.797+0000 7f9614800640 1 -- 172.21.15.117:0/1715589956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f960c102a30 msgr2=0x7f960c1980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:54.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.797+0000 7f9614800640 1 --2- 172.21.15.117:0/1715589956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f960c102a30 0x7f960c1980b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f95fc0349e0 tx=0x7f95fc032450 comp rx=0 tx=0).stop 2024-03-19T03:30:54.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.797+0000 7f9614800640 1 -- 172.21.15.117:0/1715589956 shutdown_connections 2024-03-19T03:30:54.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.797+0000 7f9614800640 1 --2- 172.21.15.117:0/1715589956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f960c1033f0 0x7f960c19f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:54.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.797+0000 7f9614800640 1 --2- 172.21.15.117:0/1715589956 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f95f80a2880 0x7f95f80a4d40 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:54.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.797+0000 7f9614800640 1 --2- 172.21.15.117:0/1715589956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f960c102a30 0x7f960c1980b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:54.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.797+0000 7f9614800640 1 --2- 172.21.15.117:0/1715589956 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f960c101840 0x7f960c197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:54.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.797+0000 7f9614800640 1 -- 172.21.15.117:0/1715589956 >> 172.21.15.117:0/1715589956 conn(0x7f960c0fcdd0 msgr2=0x7f960c105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:54.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.797+0000 7f9614800640 1 -- 172.21.15.117:0/1715589956 shutdown_connections 2024-03-19T03:30:54.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:54.797+0000 7f9614800640 1 -- 172.21.15.117:0/1715589956 wait complete. 2024-03-19T03:30:54.816 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-19T03:30:54.816 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T03:30:54.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:54 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3563698861' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:30:54.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:54 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/495260649' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:30:54.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:54 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/835081362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:30:54.866 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:30:55.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.078+0000 7fc872e00640 1 -- 172.21.15.117:0/958706048 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc86c103a60 msgr2=0x7fc86c103e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:55.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.078+0000 7fc872e00640 1 --2- 172.21.15.117:0/958706048 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc86c103a60 0x7fc86c103e80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc85c009900 tx=0x7fc85c02bc30 comp rx=0 tx=0).stop 2024-03-19T03:30:55.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.078+0000 7fc872e00640 1 -- 172.21.15.117:0/958706048 shutdown_connections 2024-03-19T03:30:55.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.078+0000 7fc872e00640 1 --2- 172.21.15.117:0/958706048 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc86c105610 0x7fc86c10bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:55.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.078+0000 7fc872e00640 1 --2- 172.21.15.117:0/958706048 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc86c104c50 0x7fc86c1050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:55.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.078+0000 7fc872e00640 1 --2- 172.21.15.117:0/958706048 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc86c103a60 0x7fc86c103e80 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:55.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.078+0000 7fc872e00640 1 -- 172.21.15.117:0/958706048 >> 172.21.15.117:0/958706048 conn(0x7fc86c0fefd0 msgr2=0x7fc86c101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:55.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.078+0000 7fc872e00640 1 -- 172.21.15.117:0/958706048 shutdown_connections 2024-03-19T03:30:55.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.078+0000 7fc872e00640 1 -- 172.21.15.117:0/958706048 wait complete. 2024-03-19T03:30:55.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.079+0000 7fc872e00640 1 Processor -- start 2024-03-19T03:30:55.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.079+0000 7fc872e00640 1 -- start start 2024-03-19T03:30:55.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.080+0000 7fc872e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc86c103a60 0x7fc86c199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:55.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.080+0000 7fc872e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc86c104c50 0x7fc86c19a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:55.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.080+0000 7fc872e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc86c105610 0x7fc86c1a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:55.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.080+0000 7fc872e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc86c10f700 con 0x7fc86c105610 2024-03-19T03:30:55.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.080+0000 7fc872e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc86c10f580 con 0x7fc86c103a60 2024-03-19T03:30:55.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.080+0000 7fc872e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fc86c10f880 con 0x7fc86c104c50 2024-03-19T03:30:55.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.080+0000 7fc86be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc86c103a60 0x7fc86c199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:55.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.080+0000 7fc86be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc86c103a60 0x7fc86c199d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:47656/0 (socket says 172.21.15.117:47656) 2024-03-19T03:30:55.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.080+0000 7fc870e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc86c105610 0x7fc86c1a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:55.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.080+0000 7fc86be00640 1 -- 172.21.15.117:0/1574779329 learned_addr learned my addr 172.21.15.117:0/1574779329 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:55.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.080+0000 7fc863400640 1 --2- 172.21.15.117:0/1574779329 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc86c104c50 0x7fc86c19a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:55.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.080+0000 7fc86be00640 1 -- 172.21.15.117:0/1574779329 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc86c104c50 msgr2=0x7fc86c19a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:55.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.080+0000 7fc86be00640 1 --2- 172.21.15.117:0/1574779329 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc86c104c50 0x7fc86c19a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:55.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.080+0000 7fc86be00640 1 -- 172.21.15.117:0/1574779329 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc86c105610 msgr2=0x7fc86c1a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:55.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.080+0000 7fc86be00640 1 --2- 172.21.15.117:0/1574779329 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc86c105610 0x7fc86c1a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:55.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.080+0000 7fc86be00640 1 -- 172.21.15.117:0/1574779329 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc86c1a1990 con 0x7fc86c103a60 2024-03-19T03:30:55.082 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.080+0000 7fc863400640 1 --2- 172.21.15.117:0/1574779329 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc86c104c50 0x7fc86c19a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:30:55.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.081+0000 7fc86be00640 1 --2- 172.21.15.117:0/1574779329 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc86c103a60 0x7fc86c199d70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc85c000c00 tx=0x7fc85c02c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:55.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.081+0000 7fc869600640 1 -- 172.21.15.117:0/1574779329 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c043020 con 0x7fc86c103a60 2024-03-19T03:30:55.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.081+0000 7fc869600640 1 -- 172.21.15.117:0/1574779329 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc85c0054e0 con 0x7fc86c103a60 2024-03-19T03:30:55.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.081+0000 7fc872e00640 1 -- 172.21.15.117:0/1574779329 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc86c1a1c20 con 0x7fc86c103a60 2024-03-19T03:30:55.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.081+0000 7fc869600640 1 -- 172.21.15.117:0/1574779329 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c003ce0 con 0x7fc86c103a60 2024-03-19T03:30:55.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.081+0000 7fc872e00640 1 -- 172.21.15.117:0/1574779329 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc86c1a2190 con 0x7fc86c103a60 2024-03-19T03:30:55.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.082+0000 7fc872e00640 1 -- 172.21.15.117:0/1574779329 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc86c10ae00 con 0x7fc86c103a60 2024-03-19T03:30:55.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.084+0000 7fc869600640 1 -- 172.21.15.117:0/1574779329 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fc85c046020 con 0x7fc86c103a60 2024-03-19T03:30:55.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.085+0000 7fc869600640 1 --2- 172.21.15.117:0/1574779329 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc8500a2880 0x7fc8500a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:55.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.085+0000 7fc869600640 1 -- 172.21.15.117:0/1574779329 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc85c0f0e20 con 0x7fc86c103a60 2024-03-19T03:30:55.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.085+0000 7fc863400640 1 --2- 172.21.15.117:0/1574779329 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc8500a2880 0x7fc8500a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:55.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.086+0000 7fc863400640 1 --2- 172.21.15.117:0/1574779329 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc8500a2880 0x7fc8500a4d40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc86c19b280 tx=0x7fc854005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:55.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.089+0000 7fc869600640 1 -- 172.21.15.117:0/1574779329 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc85c04d020 con 0x7fc86c103a60 2024-03-19T03:30:55.288 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.288+0000 7fc872e00640 1 -- 172.21.15.117:0/1574779329 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc86c062420 con 0x7fc86c103a60 2024-03-19T03:30:55.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.290+0000 7fc869600640 1 -- 172.21.15.117:0/1574779329 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24210 (secure 0 0 0) 0x7fc85c0bdc20 con 0x7fc86c103a60 2024-03-19T03:30:55.291 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:30:55.291 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":89,"fsid":"a3a82456-e59f-11ee-95c9-87774f69a715","created":"2024-03-19T03:21:30.427136+0000","modified":"2024-03-19T03:30:40.180745+0000","last_up_change":"2024-03-19T03:29:32.967505+0000","last_in_change":"2024-03-19T03:29:12.848740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:25:18.824270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T03:30:33.137170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T03:30:34.598838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T03:30:38.230023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b32eb0ee-9d4b-4748-87ec-eda02c907415","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6803","nonce":3014934900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6805","nonce":3014934900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6809","nonce":3014934900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6807","nonce":3014934900}]},"public_addr":"172.21.15.117:6803/3014934900","cluster_addr":"172.21.15.117:6805/3014934900","heartbeat_back_addr":"172.21.15.117:6809/3014934900","heartbeat_front_addr":"172.21.15.117:6807/3014934900","state":["exists","up"]},{"osd":1,"uuid":"e07585be-c016-4466-b07c-abbc22a58de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6801","nonce":2425927341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6803","nonce":2425927341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6807","nonce":2425927341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6805","nonce":2425927341}]},"public_addr":"172.21.15.119:6801/2425927341","cluster_addr":"172.21.15.119:6803/2425927341","heartbeat_back_addr":"172.21.15.119:6807/2425927341","heartbeat_front_addr":"172.21.15.119:6805/2425927341","state":["exists","up"]},{"osd":2,"uuid":"ede9edda-3b9f-41ca-ad04-5dcb738ab757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6801","nonce":3733812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6803","nonce":3733812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6807","nonce":3733812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6805","nonce":3733812989}]},"public_addr":"172.21.15.178:6801/3733812989","cluster_addr":"172.21.15.178:6803/3733812989","heartbeat_back_addr":"172.21.15.178:6807/3733812989","heartbeat_front_addr":"172.21.15.178:6805/3733812989","state":["exists","up"]},{"osd":3,"uuid":"f81c41e1-3a69-40db-a90f-58151286b081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6811","nonce":2178310135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6813","nonce":2178310135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6817","nonce":2178310135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6815","nonce":2178310135}]},"public_addr":"172.21.15.117:6811/2178310135","cluster_addr":"172.21.15.117:6813/2178310135","heartbeat_back_addr":"172.21.15.117:6817/2178310135","heartbeat_front_addr":"172.21.15.117:6815/2178310135","state":["exists","up"]},{"osd":4,"uuid":"caf7ce86-caf0-4eaf-bbab-bd2d93e63c33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6809","nonce":3455398086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6811","nonce":3455398086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6815","nonce":3455398086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6813","nonce":3455398086}]},"public_addr":"172.21.15.119:6809/3455398086","cluster_addr":"172.21.15.119:6811/3455398086","heartbeat_back_addr":"172.21.15.119:6815/3455398086","heartbeat_front_addr":"172.21.15.119:6813/3455398086","state":["exists","up"]},{"osd":5,"uuid":"20ba0b8a-2e0f-4ea7-b435-430e44766200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6809","nonce":1259631912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6811","nonce":1259631912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6815","nonce":1259631912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6813","nonce":1259631912}]},"public_addr":"172.21.15.178:6809/1259631912","cluster_addr":"172.21.15.178:6811/1259631912","heartbeat_back_addr":"172.21.15.178:6815/1259631912","heartbeat_front_addr":"172.21.15.178:6813/1259631912","state":["exists","up"]},{"osd":6,"uuid":"75c2ddd7-f837-42e0-ac47-8cfbcf97a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6819","nonce":864146757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6821","nonce":864146757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6825","nonce":864146757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6823","nonce":864146757}]},"public_addr":"172.21.15.117:6819/864146757","cluster_addr":"172.21.15.117:6821/864146757","heartbeat_back_addr":"172.21.15.117:6825/864146757","heartbeat_front_addr":"172.21.15.117:6823/864146757","state":["exists","up"]},{"osd":7,"uuid":"929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6817","nonce":1273365545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6819","nonce":1273365545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6823","nonce":1273365545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6821","nonce":1273365545}]},"public_addr":"172.21.15.119:6817/1273365545","cluster_addr":"172.21.15.119:6819/1273365545","heartbeat_back_addr":"172.21.15.119:6823/1273365545","heartbeat_front_addr":"172.21.15.119:6821/1273365545","state":["exists","up"]},{"osd":8,"uuid":"fd371461-a538-4781-92e1-5eead4ca3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6817","nonce":1059587501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6819","nonce":1059587501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6823","nonce":1059587501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6821","nonce":1059587501}]},"public_addr":"172.21.15.178:6817/1059587501","cluster_addr":"172.21.15.178:6819/1059587501","heartbeat_back_addr":"172.21.15.178:6823/1059587501","heartbeat_front_addr":"172.21.15.178:6821/1059587501","state":["exists","up"]},{"osd":9,"uuid":"0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6827","nonce":3653875600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6829","nonce":3653875600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6833","nonce":3653875600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6831","nonce":3653875600}]},"public_addr":"172.21.15.117:6827/3653875600","cluster_addr":"172.21.15.117:6829/3653875600","heartbeat_back_addr":"172.21.15.117:6833/3653875600","heartbeat_front_addr":"172.21.15.117:6831/3653875600","state":["exists","up"]},{"osd":10,"uuid":"de2049da-d19e-4386-afbf-fd6b5fddbcba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6825","nonce":1050976194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6827","nonce":1050976194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6831","nonce":1050976194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6829","nonce":1050976194}]},"public_addr":"172.21.15.119:6825/1050976194","cluster_addr":"172.21.15.119:6827/1050976194","heartbeat_back_addr":"172.21.15.119:6831/1050976194","heartbeat_front_addr":"172.21.15.119:6829/1050976194","state":["exists","up"]},{"osd":11,"uuid":"3cccc0f9-d273-4b6b-9b8e-c926371472dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6825","nonce":2915080368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6827","nonce":2915080368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6831","nonce":2915080368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6829","nonce":2915080368}]},"public_addr":"172.21.15.178:6825/2915080368","cluster_addr":"172.21.15.178:6827/2915080368","heartbeat_back_addr":"172.21.15.178:6831/2915080368","heartbeat_front_addr":"172.21.15.178:6829/2915080368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:27.876274+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:53.094706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:16.598813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:48.160419+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:14.048925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:38.814304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:12.542328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:38.942747+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:04.488963+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:38.196053+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:05.254728+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:30.874915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.117:0/185684447":"2024-03-20T03:22:38.134578+0000","172.21.15.117:0/361367101":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/2788067603":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3506200675":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6800/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3283471541":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3750920169":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/2780813298":"2024-03-20T03:22:03.905461+0000","172.21.15.117:6800/2788067603":"2024-03-20T03:22:38.134578+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:30:55.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.293+0000 7fc872e00640 1 -- 172.21.15.117:0/1574779329 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc8500a2880 msgr2=0x7fc8500a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:55.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.293+0000 7fc872e00640 1 --2- 172.21.15.117:0/1574779329 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc8500a2880 0x7fc8500a4d40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc86c19b280 tx=0x7fc854005f90 comp rx=0 tx=0).stop 2024-03-19T03:30:55.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.293+0000 7fc872e00640 1 -- 172.21.15.117:0/1574779329 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc86c103a60 msgr2=0x7fc86c199d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:55.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.293+0000 7fc872e00640 1 --2- 172.21.15.117:0/1574779329 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc86c103a60 0x7fc86c199d70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc85c000c00 tx=0x7fc85c02c6c0 comp rx=0 tx=0).stop 2024-03-19T03:30:55.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.293+0000 7fc872e00640 1 -- 172.21.15.117:0/1574779329 shutdown_connections 2024-03-19T03:30:55.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.293+0000 7fc872e00640 1 --2- 172.21.15.117:0/1574779329 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc86c105610 0x7fc86c1a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:55.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.293+0000 7fc872e00640 1 --2- 172.21.15.117:0/1574779329 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc86c104c50 0x7fc86c19a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:55.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.293+0000 7fc872e00640 1 --2- 172.21.15.117:0/1574779329 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc8500a2880 0x7fc8500a4d40 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:55.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.293+0000 7fc872e00640 1 --2- 172.21.15.117:0/1574779329 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc86c103a60 0x7fc86c199d70 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:55.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.293+0000 7fc872e00640 1 -- 172.21.15.117:0/1574779329 >> 172.21.15.117:0/1574779329 conn(0x7fc86c0fefd0 msgr2=0x7fc86c107fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:55.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.293+0000 7fc872e00640 1 -- 172.21.15.117:0/1574779329 shutdown_connections 2024-03-19T03:30:55.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.293+0000 7fc872e00640 1 -- 172.21.15.117:0/1574779329 wait complete. 2024-03-19T03:30:55.311 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T03:25:18.824270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T03:30:33.137170+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T03:30:34.598838+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T03:30:38.230023+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T03:30:55.312 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T03:30:55.529 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.525+0000 7f1dd4600640 1 -- 172.21.15.117:0/334971589 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dcc077dc0 msgr2=0x7f1dcc078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:55.530 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.525+0000 7f1dd4600640 1 --2- 172.21.15.117:0/334971589 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dcc077dc0 0x7f1dcc078660 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1dc4009ca0 tx=0x7f1dc402c100 comp rx=0 tx=0).stop 2024-03-19T03:30:55.530 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.525+0000 7f1dd4600640 1 -- 172.21.15.117:0/334971589 shutdown_connections 2024-03-19T03:30:55.530 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.525+0000 7f1dd4600640 1 --2- 172.21.15.117:0/334971589 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dcc077dc0 0x7f1dcc078660 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:55.530 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.525+0000 7f1dd4600640 1 --2- 172.21.15.117:0/334971589 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1dcc077420 0x7f1dcc077880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:55.530 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.525+0000 7f1dd4600640 1 --2- 172.21.15.117:0/334971589 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1dcc0761e0 0x7f1dcc076600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:55.530 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.525+0000 7f1dd4600640 1 -- 172.21.15.117:0/334971589 >> 172.21.15.117:0/334971589 conn(0x7f1dcc0fefd0 msgr2=0x7f1dcc101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:55.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.525+0000 7f1dd4600640 1 -- 172.21.15.117:0/334971589 shutdown_connections 2024-03-19T03:30:55.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.525+0000 7f1dd4600640 1 -- 172.21.15.117:0/334971589 wait complete. 2024-03-19T03:30:55.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.526+0000 7f1dd4600640 1 Processor -- start 2024-03-19T03:30:55.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.527+0000 7f1dd4600640 1 -- start start 2024-03-19T03:30:55.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.527+0000 7f1dd4600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dcc0761e0 0x7f1dcc19e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:55.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.527+0000 7f1dd4600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1dcc077420 0x7f1dcc19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:55.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.527+0000 7f1dd4600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1dcc077dc0 0x7f1dcc1a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:55.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.527+0000 7f1dd4600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1dcc113af0 con 0x7f1dcc077dc0 2024-03-19T03:30:55.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.527+0000 7f1dd4600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1dcc113970 con 0x7f1dcc0761e0 2024-03-19T03:30:55.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.527+0000 7f1dd4600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f1dcc113c70 con 0x7f1dcc077420 2024-03-19T03:30:55.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.527+0000 7f1dd1c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dcc0761e0 0x7f1dcc19e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:55.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.527+0000 7f1dd1c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dcc0761e0 0x7f1dcc19e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:47672/0 (socket says 172.21.15.117:47672) 2024-03-19T03:30:55.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.527+0000 7f1dd1c00640 1 -- 172.21.15.117:0/3057716151 learned_addr learned my addr 172.21.15.117:0/3057716151 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:55.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.527+0000 7f1dd1200640 1 --2- 172.21.15.117:0/3057716151 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1dcc077420 0x7f1dcc19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:55.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.527+0000 7f1dd2600640 1 --2- 172.21.15.117:0/3057716151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1dcc077dc0 0x7f1dcc1a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:55.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.528+0000 7f1dd1c00640 1 -- 172.21.15.117:0/3057716151 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1dcc077420 msgr2=0x7f1dcc19e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:55.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.528+0000 7f1dd1c00640 1 --2- 172.21.15.117:0/3057716151 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1dcc077420 0x7f1dcc19e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:55.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.528+0000 7f1dd1c00640 1 -- 172.21.15.117:0/3057716151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1dcc077dc0 msgr2=0x7f1dcc1a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:55.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.528+0000 7f1dd1c00640 1 --2- 172.21.15.117:0/3057716151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1dcc077dc0 0x7f1dcc1a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:55.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.528+0000 7f1dd1c00640 1 -- 172.21.15.117:0/3057716151 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dcc1a5d30 con 0x7f1dcc0761e0 2024-03-19T03:30:55.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.528+0000 7f1dd2600640 1 --2- 172.21.15.117:0/3057716151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1dcc077dc0 0x7f1dcc1a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:30:55.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.528+0000 7f1dd1200640 1 --2- 172.21.15.117:0/3057716151 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1dcc077420 0x7f1dcc19e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:30:55.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.528+0000 7f1dd1c00640 1 --2- 172.21.15.117:0/3057716151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dcc0761e0 0x7f1dcc19e110 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1dc000e9f0 tx=0x7f1dc000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:55.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.528+0000 7f1dca000640 1 -- 172.21.15.117:0/3057716151 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc000cd80 con 0x7f1dcc0761e0 2024-03-19T03:30:55.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.528+0000 7f1dca000640 1 -- 172.21.15.117:0/3057716151 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dc0010910 con 0x7f1dcc0761e0 2024-03-19T03:30:55.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.528+0000 7f1dd4600640 1 -- 172.21.15.117:0/3057716151 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dcc1a5fc0 con 0x7f1dcc0761e0 2024-03-19T03:30:55.532 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.528+0000 7f1dca000640 1 -- 172.21.15.117:0/3057716151 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc0018990 con 0x7f1dcc0761e0 2024-03-19T03:30:55.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.528+0000 7f1dd4600640 1 -- 172.21.15.117:0/3057716151 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dcc1a64d0 con 0x7f1dcc0761e0 2024-03-19T03:30:55.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.529+0000 7f1dd4600640 1 -- 172.21.15.117:0/3057716151 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dcc1103d0 con 0x7f1dcc0761e0 2024-03-19T03:30:55.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.531+0000 7f1dca000640 1 -- 172.21.15.117:0/3057716151 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f1dc0010ac0 con 0x7f1dcc0761e0 2024-03-19T03:30:55.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.532+0000 7f1dca000640 1 --2- 172.21.15.117:0/3057716151 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1db40a2880 0x7f1db40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:55.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.532+0000 7f1dca000640 1 -- 172.21.15.117:0/3057716151 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f1dc00cf0f0 con 0x7f1dcc0761e0 2024-03-19T03:30:55.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.533+0000 7f1dd1200640 1 --2- 172.21.15.117:0/3057716151 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1db40a2880 0x7f1db40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:55.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.533+0000 7f1dd1200640 1 --2- 172.21.15.117:0/3057716151 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1db40a2880 0x7f1db40a4d40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1db80058c0 tx=0x7f1db8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:55.536 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.536+0000 7f1dca000640 1 -- 172.21.15.117:0/3057716151 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1dc009bef0 con 0x7f1dcc0761e0 2024-03-19T03:30:55.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:55 smithi119 ceph-mon[66861]: pgmap v327: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-19T03:30:55.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:55 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1715589956' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T03:30:55.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:55 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1574779329' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:30:55.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.736+0000 7f1dd4600640 1 -- 172.21.15.117:0/3057716151 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1dcc0624c0 con 0x7f1dcc0761e0 2024-03-19T03:30:55.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.737+0000 7f1dca000640 1 -- 172.21.15.117:0/3057716151 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f1dc009b7f0 con 0x7f1dcc0761e0 2024-03-19T03:30:55.737 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 1 2024-03-19T03:30:55.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.739+0000 7f1dd4600640 1 -- 172.21.15.117:0/3057716151 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1db40a2880 msgr2=0x7f1db40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:55.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.740+0000 7f1dd4600640 1 --2- 172.21.15.117:0/3057716151 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1db40a2880 0x7f1db40a4d40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1db80058c0 tx=0x7f1db8005850 comp rx=0 tx=0).stop 2024-03-19T03:30:55.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.740+0000 7f1dd4600640 1 -- 172.21.15.117:0/3057716151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dcc0761e0 msgr2=0x7f1dcc19e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:55.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.740+0000 7f1dd4600640 1 --2- 172.21.15.117:0/3057716151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dcc0761e0 0x7f1dcc19e110 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1dc000e9f0 tx=0x7f1dc000eeb0 comp rx=0 tx=0).stop 2024-03-19T03:30:55.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.740+0000 7f1dd4600640 1 -- 172.21.15.117:0/3057716151 shutdown_connections 2024-03-19T03:30:55.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.740+0000 7f1dd4600640 1 --2- 172.21.15.117:0/3057716151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1dcc077dc0 0x7f1dcc1a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:55.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.740+0000 7f1dd4600640 1 --2- 172.21.15.117:0/3057716151 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1dcc077420 0x7f1dcc19e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:55.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.740+0000 7f1dd4600640 1 --2- 172.21.15.117:0/3057716151 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1db40a2880 0x7f1db40a4d40 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:55.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.740+0000 7f1dd4600640 1 --2- 172.21.15.117:0/3057716151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dcc0761e0 0x7f1dcc19e110 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:55.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.740+0000 7f1dd4600640 1 -- 172.21.15.117:0/3057716151 >> 172.21.15.117:0/3057716151 conn(0x7f1dcc0fefd0 msgr2=0x7f1dcc100ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:55.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.740+0000 7f1dd4600640 1 -- 172.21.15.117:0/3057716151 shutdown_connections 2024-03-19T03:30:55.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.740+0000 7f1dd4600640 1 -- 172.21.15.117:0/3057716151 wait complete. 2024-03-19T03:30:55.758 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:55 smithi117 ceph-mon[31838]: pgmap v327: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-19T03:30:55.758 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:55 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1715589956' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T03:30:55.758 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:55 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1574779329' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:30:55.763 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T03:30:55.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:55 smithi178 ceph-mon[67578]: pgmap v327: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-19T03:30:55.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:55 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1715589956' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T03:30:55.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:55 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1574779329' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:30:55.966 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.966+0000 7fd0e2400640 1 -- 172.21.15.117:0/4209133525 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0dc077dc0 msgr2=0x7fd0dc078660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:55.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.966+0000 7fd0e2400640 1 --2- 172.21.15.117:0/4209133525 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0dc077dc0 0x7fd0dc078660 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4009ca0 tx=0x7fd0d402c100 comp rx=0 tx=0).stop 2024-03-19T03:30:55.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.966+0000 7fd0e2400640 1 -- 172.21.15.117:0/4209133525 shutdown_connections 2024-03-19T03:30:55.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.966+0000 7fd0e2400640 1 --2- 172.21.15.117:0/4209133525 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0dc077dc0 0x7fd0dc078660 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:55.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.966+0000 7fd0e2400640 1 --2- 172.21.15.117:0/4209133525 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0dc077420 0x7fd0dc077880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:55.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.966+0000 7fd0e2400640 1 --2- 172.21.15.117:0/4209133525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0dc0761e0 0x7fd0dc076600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:55.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.966+0000 7fd0e2400640 1 -- 172.21.15.117:0/4209133525 >> 172.21.15.117:0/4209133525 conn(0x7fd0dc0fefd0 msgr2=0x7fd0dc101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:55.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.966+0000 7fd0e2400640 1 -- 172.21.15.117:0/4209133525 shutdown_connections 2024-03-19T03:30:55.967 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.966+0000 7fd0e2400640 1 -- 172.21.15.117:0/4209133525 wait complete. 2024-03-19T03:30:55.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.967+0000 7fd0e2400640 1 Processor -- start 2024-03-19T03:30:55.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.967+0000 7fd0e2400640 1 -- start start 2024-03-19T03:30:55.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.968+0000 7fd0e2400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0dc0761e0 0x7fd0dc19e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:55.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.968+0000 7fd0e2400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0dc077420 0x7fd0dc19e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:55.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.968+0000 7fd0e2400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0dc077dc0 0x7fd0dc1a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:55.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.968+0000 7fd0e2400640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd0dc113af0 con 0x7fd0dc077420 2024-03-19T03:30:55.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.968+0000 7fd0e2400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd0dc113970 con 0x7fd0dc077dc0 2024-03-19T03:30:55.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.968+0000 7fd0e2400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fd0dc113c70 con 0x7fd0dc0761e0 2024-03-19T03:30:55.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.968+0000 7fd0dbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0dc077dc0 0x7fd0dc1a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:55.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.968+0000 7fd0daa00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0dc077420 0x7fd0dc19e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:55.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.968+0000 7fd0db400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0dc0761e0 0x7fd0dc19e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:55.970 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.968+0000 7fd0dbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0dc077dc0 0x7fd0dc1a5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:53348/0 (socket says 172.21.15.117:53348) 2024-03-19T03:30:55.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.968+0000 7fd0daa00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0dc077420 0x7fd0dc19e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:53928/0 (socket says 172.21.15.117:53928) 2024-03-19T03:30:55.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.968+0000 7fd0dbe00640 1 -- 172.21.15.117:0/4263745278 learned_addr learned my addr 172.21.15.117:0/4263745278 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:55.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.968+0000 7fd0dbe00640 1 -- 172.21.15.117:0/4263745278 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0dc0761e0 msgr2=0x7fd0dc19e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:55.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.968+0000 7fd0dbe00640 1 --2- 172.21.15.117:0/4263745278 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0dc0761e0 0x7fd0dc19e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:55.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.968+0000 7fd0dbe00640 1 -- 172.21.15.117:0/4263745278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0dc077420 msgr2=0x7fd0dc19e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:55.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.968+0000 7fd0dbe00640 1 --2- 172.21.15.117:0/4263745278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0dc077420 0x7fd0dc19e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:55.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.969+0000 7fd0dbe00640 1 -- 172.21.15.117:0/4263745278 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0dc1a5d00 con 0x7fd0dc077dc0 2024-03-19T03:30:55.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.969+0000 7fd0dbe00640 1 --2- 172.21.15.117:0/4263745278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0dc077dc0 0x7fd0dc1a5690 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd0d402c640 tx=0x7fd0d4035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:55.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.969+0000 7fd0d3e00640 1 -- 172.21.15.117:0/4263745278 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d4039040 con 0x7fd0dc077dc0 2024-03-19T03:30:55.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.969+0000 7fd0d3e00640 1 -- 172.21.15.117:0/4263745278 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0d4044070 con 0x7fd0dc077dc0 2024-03-19T03:30:55.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.969+0000 7fd0d3e00640 1 -- 172.21.15.117:0/4263745278 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d40043c0 con 0x7fd0dc077dc0 2024-03-19T03:30:55.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.969+0000 7fd0e2400640 1 -- 172.21.15.117:0/4263745278 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0dc1a5f90 con 0x7fd0dc077dc0 2024-03-19T03:30:55.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.969+0000 7fd0e2400640 1 -- 172.21.15.117:0/4263745278 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0dc1a63a0 con 0x7fd0dc077dc0 2024-03-19T03:30:55.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.970+0000 7fd0e2400640 1 -- 172.21.15.117:0/4263745278 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0dc110510 con 0x7fd0dc077dc0 2024-03-19T03:30:55.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.972+0000 7fd0d3e00640 1 -- 172.21.15.117:0/4263745278 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fd0d4004560 con 0x7fd0dc077dc0 2024-03-19T03:30:55.973 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.974+0000 7fd0d3e00640 1 --2- 172.21.15.117:0/4263745278 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd0c00a2880 0x7fd0c00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:55.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.974+0000 7fd0d3e00640 1 -- 172.21.15.117:0/4263745278 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd0d40f0ff0 con 0x7fd0dc077dc0 2024-03-19T03:30:55.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.974+0000 7fd0db400640 1 --2- 172.21.15.117:0/4263745278 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd0c00a2880 0x7fd0c00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:55.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.975+0000 7fd0db400640 1 --2- 172.21.15.117:0/4263745278 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd0c00a2880 0x7fd0c00a4d40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc009bd0 tx=0x7fd0cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:55.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:55.978+0000 7fd0d3e00640 1 -- 172.21.15.117:0/4263745278 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd0d404e020 con 0x7fd0dc077dc0 2024-03-19T03:30:56.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.176+0000 7fd0e2400640 1 -- 172.21.15.117:0/4263745278 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd0dc0624c0 con 0x7fd0dc077dc0 2024-03-19T03:30:56.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.177+0000 7fd0d3e00640 1 -- 172.21.15.117:0/4263745278 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7fd0d40bddf0 con 0x7fd0dc077dc0 2024-03-19T03:30:56.177 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 32 2024-03-19T03:30:56.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.179+0000 7fd0e2400640 1 -- 172.21.15.117:0/4263745278 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd0c00a2880 msgr2=0x7fd0c00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:56.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.179+0000 7fd0e2400640 1 --2- 172.21.15.117:0/4263745278 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd0c00a2880 0x7fd0c00a4d40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc009bd0 tx=0x7fd0cc005f90 comp rx=0 tx=0).stop 2024-03-19T03:30:56.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.179+0000 7fd0e2400640 1 -- 172.21.15.117:0/4263745278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0dc077dc0 msgr2=0x7fd0dc1a5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:56.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.179+0000 7fd0e2400640 1 --2- 172.21.15.117:0/4263745278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0dc077dc0 0x7fd0dc1a5690 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd0d402c640 tx=0x7fd0d4035d10 comp rx=0 tx=0).stop 2024-03-19T03:30:56.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.180+0000 7fd0e2400640 1 -- 172.21.15.117:0/4263745278 shutdown_connections 2024-03-19T03:30:56.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.180+0000 7fd0e2400640 1 --2- 172.21.15.117:0/4263745278 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd0c00a2880 0x7fd0c00a4d40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:56.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.180+0000 7fd0e2400640 1 --2- 172.21.15.117:0/4263745278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0dc077dc0 0x7fd0dc1a5690 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:56.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.180+0000 7fd0e2400640 1 --2- 172.21.15.117:0/4263745278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0dc077420 0x7fd0dc19e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:56.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.180+0000 7fd0e2400640 1 --2- 172.21.15.117:0/4263745278 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0dc0761e0 0x7fd0dc19e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:56.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.180+0000 7fd0e2400640 1 -- 172.21.15.117:0/4263745278 >> 172.21.15.117:0/4263745278 conn(0x7fd0dc0fefd0 msgr2=0x7fd0dc1006a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:56.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.180+0000 7fd0e2400640 1 -- 172.21.15.117:0/4263745278 shutdown_connections 2024-03-19T03:30:56.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.180+0000 7fd0e2400640 1 -- 172.21.15.117:0/4263745278 wait complete. 2024-03-19T03:30:56.197 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T03:30:56.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.415+0000 7f6800000640 1 -- 172.21.15.117:0/788052636 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67f8100720 msgr2=0x7f67f8100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:56.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.415+0000 7f6800000640 1 --2- 172.21.15.117:0/788052636 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67f8100720 0x7f67f8100b40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f67ec009900 tx=0x7f67ec02bc30 comp rx=0 tx=0).stop 2024-03-19T03:30:56.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.415+0000 7f6800000640 1 -- 172.21.15.117:0/788052636 shutdown_connections 2024-03-19T03:30:56.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.415+0000 7f6800000640 1 --2- 172.21.15.117:0/788052636 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f67f81022d0 0x7f67f8108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:56.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.415+0000 7f6800000640 1 --2- 172.21.15.117:0/788052636 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67f8101910 0x7f67f8101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:56.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.415+0000 7f6800000640 1 --2- 172.21.15.117:0/788052636 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67f8100720 0x7f67f8100b40 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:56.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.415+0000 7f6800000640 1 -- 172.21.15.117:0/788052636 >> 172.21.15.117:0/788052636 conn(0x7f67f80fbcb0 msgr2=0x7f67f80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:56.416 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.415+0000 7f6800000640 1 -- 172.21.15.117:0/788052636 shutdown_connections 2024-03-19T03:30:56.416 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.415+0000 7f6800000640 1 -- 172.21.15.117:0/788052636 wait complete. 2024-03-19T03:30:56.416 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.416+0000 7f6800000640 1 Processor -- start 2024-03-19T03:30:56.416 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.416+0000 7f6800000640 1 -- start start 2024-03-19T03:30:56.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.417+0000 7f6800000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67f8100720 0x7f67f8199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:56.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.417+0000 7f6800000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f67f8101910 0x7f67f819a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:56.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.417+0000 7f6800000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67f81022d0 0x7f67f81a13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:56.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.417+0000 7f6800000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f67f810c3c0 con 0x7f67f81022d0 2024-03-19T03:30:56.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.417+0000 7f6800000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f67f810c240 con 0x7f67f8100720 2024-03-19T03:30:56.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.417+0000 7f6800000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f67f810c540 con 0x7f67f8101910 2024-03-19T03:30:56.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.417+0000 7f67fd600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67f8100720 0x7f67f8199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:56.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.417+0000 7f67fd600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67f8100720 0x7f67f8199df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:53366/0 (socket says 172.21.15.117:53366) 2024-03-19T03:30:56.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.417+0000 7f67fd600640 1 -- 172.21.15.117:0/3001738285 learned_addr learned my addr 172.21.15.117:0/3001738285 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:56.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.417+0000 7f67fe000640 1 --2- 172.21.15.117:0/3001738285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67f81022d0 0x7f67f81a13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:56.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.417+0000 7f67fcc00640 1 --2- 172.21.15.117:0/3001738285 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f67f8101910 0x7f67f819a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:56.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.417+0000 7f67fd600640 1 -- 172.21.15.117:0/3001738285 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f67f8101910 msgr2=0x7f67f819a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:56.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.417+0000 7f67fd600640 1 --2- 172.21.15.117:0/3001738285 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f67f8101910 0x7f67f819a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:56.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.417+0000 7f67fd600640 1 -- 172.21.15.117:0/3001738285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67f81022d0 msgr2=0x7f67f81a13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:56.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.418+0000 7f67fd600640 1 --2- 172.21.15.117:0/3001738285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67f81022d0 0x7f67f81a13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:56.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.418+0000 7f67fd600640 1 -- 172.21.15.117:0/3001738285 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f81a1a10 con 0x7f67f8100720 2024-03-19T03:30:56.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.418+0000 7f67fe000640 1 --2- 172.21.15.117:0/3001738285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67f81022d0 0x7f67f81a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:30:56.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.418+0000 7f67fd600640 1 --2- 172.21.15.117:0/3001738285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67f8100720 0x7f67f8199df0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f67ec000c00 tx=0x7f67ec02c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:56.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.418+0000 7f67f6000640 1 -- 172.21.15.117:0/3001738285 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec043020 con 0x7f67f8100720 2024-03-19T03:30:56.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.418+0000 7f67f6000640 1 -- 172.21.15.117:0/3001738285 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67ec0054e0 con 0x7f67f8100720 2024-03-19T03:30:56.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.418+0000 7f67f6000640 1 -- 172.21.15.117:0/3001738285 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec003ce0 con 0x7f67f8100720 2024-03-19T03:30:56.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.418+0000 7f6800000640 1 -- 172.21.15.117:0/3001738285 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67f81a1ca0 con 0x7f67f8100720 2024-03-19T03:30:56.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.418+0000 7f6800000640 1 -- 172.21.15.117:0/3001738285 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67f81a2170 con 0x7f67f8100720 2024-03-19T03:30:56.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.420+0000 7f6800000640 1 -- 172.21.15.117:0/3001738285 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67f81a2340 con 0x7f67f8100720 2024-03-19T03:30:56.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.421+0000 7f67f6000640 1 -- 172.21.15.117:0/3001738285 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f67ec046020 con 0x7f67f8100720 2024-03-19T03:30:56.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.422+0000 7f67f6000640 1 --2- 172.21.15.117:0/3001738285 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f67e00a2880 0x7f67e00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:56.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.422+0000 7f67f6000640 1 -- 172.21.15.117:0/3001738285 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f67ec0f0e80 con 0x7f67f8100720 2024-03-19T03:30:56.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.422+0000 7f67fcc00640 1 --2- 172.21.15.117:0/3001738285 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f67e00a2880 0x7f67e00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:56.423 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.423+0000 7f67fcc00640 1 --2- 172.21.15.117:0/3001738285 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f67e00a2880 0x7f67e00a4d40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f67f819b300 tx=0x7f67e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:56.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.426+0000 7f67f6000640 1 -- 172.21.15.117:0/3001738285 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f67ec04d020 con 0x7f67f8100720 2024-03-19T03:30:56.489 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:56 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3057716151' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:30:56.489 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:56 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/4263745278' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:30:56.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.641+0000 7f6800000640 1 -- 172.21.15.117:0/3001738285 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f67f81a2510 con 0x7f67f8100720 2024-03-19T03:30:56.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.641+0000 7f67f6000640 1 -- 172.21.15.117:0/3001738285 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f67ec03c350 con 0x7f67f8100720 2024-03-19T03:30:56.641 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 64 2024-03-19T03:30:56.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.644+0000 7f6800000640 1 -- 172.21.15.117:0/3001738285 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f67e00a2880 msgr2=0x7f67e00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:56.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.644+0000 7f6800000640 1 --2- 172.21.15.117:0/3001738285 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f67e00a2880 0x7f67e00a4d40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f67f819b300 tx=0x7f67e4005f90 comp rx=0 tx=0).stop 2024-03-19T03:30:56.644 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.644+0000 7f6800000640 1 -- 172.21.15.117:0/3001738285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67f8100720 msgr2=0x7f67f8199df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:56.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.644+0000 7f6800000640 1 --2- 172.21.15.117:0/3001738285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67f8100720 0x7f67f8199df0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f67ec000c00 tx=0x7f67ec02c6c0 comp rx=0 tx=0).stop 2024-03-19T03:30:56.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.644+0000 7f6800000640 1 -- 172.21.15.117:0/3001738285 shutdown_connections 2024-03-19T03:30:56.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.644+0000 7f6800000640 1 --2- 172.21.15.117:0/3001738285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67f81022d0 0x7f67f81a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:56.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.644+0000 7f6800000640 1 --2- 172.21.15.117:0/3001738285 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f67f8101910 0x7f67f819a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:56.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.644+0000 7f6800000640 1 --2- 172.21.15.117:0/3001738285 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f67e00a2880 0x7f67e00a4d40 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:56.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.644+0000 7f6800000640 1 --2- 172.21.15.117:0/3001738285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67f8100720 0x7f67f8199df0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:56.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.644+0000 7f6800000640 1 -- 172.21.15.117:0/3001738285 >> 172.21.15.117:0/3001738285 conn(0x7f67f80fbcb0 msgr2=0x7f67f80fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:56.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.644+0000 7f6800000640 1 -- 172.21.15.117:0/3001738285 shutdown_connections 2024-03-19T03:30:56.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.644+0000 7f6800000640 1 -- 172.21.15.117:0/3001738285 wait complete. 2024-03-19T03:30:56.663 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T03:30:56.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:56 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3057716151' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:30:56.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:56 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/4263745278' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:30:56.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:56 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3057716151' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:30:56.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:56 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/4263745278' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:30:56.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.873+0000 7f8725600640 1 -- 172.21.15.117:0/2175391352 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8720077420 msgr2=0x7f8720077880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:56.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.873+0000 7f8725600640 1 --2- 172.21.15.117:0/2175391352 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8720077420 0x7f8720077880 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8708009900 tx=0x7f870802bc30 comp rx=0 tx=0).stop 2024-03-19T03:30:56.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.873+0000 7f8725600640 1 -- 172.21.15.117:0/2175391352 shutdown_connections 2024-03-19T03:30:56.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.873+0000 7f8725600640 1 --2- 172.21.15.117:0/2175391352 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8720077dc0 0x7f8720078660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:56.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.873+0000 7f8725600640 1 --2- 172.21.15.117:0/2175391352 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8720077420 0x7f8720077880 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:56.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.873+0000 7f8725600640 1 --2- 172.21.15.117:0/2175391352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f87200761e0 0x7f8720076600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:56.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.873+0000 7f8725600640 1 -- 172.21.15.117:0/2175391352 >> 172.21.15.117:0/2175391352 conn(0x7f87200fefd0 msgr2=0x7f8720101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:56.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.873+0000 7f8725600640 1 -- 172.21.15.117:0/2175391352 shutdown_connections 2024-03-19T03:30:56.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.873+0000 7f8725600640 1 -- 172.21.15.117:0/2175391352 wait complete. 2024-03-19T03:30:56.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.874+0000 7f8725600640 1 Processor -- start 2024-03-19T03:30:56.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.875+0000 7f8725600640 1 -- start start 2024-03-19T03:30:56.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.875+0000 7f8725600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87200761e0 0x7f872019e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:56.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.875+0000 7f8725600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8720077420 0x7f872019e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:56.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.875+0000 7f8725600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8720077dc0 0x7f87201a5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:56.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.875+0000 7f8725600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8720113ab0 con 0x7f8720077420 2024-03-19T03:30:56.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.875+0000 7f8725600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8720113930 con 0x7f8720077dc0 2024-03-19T03:30:56.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.875+0000 7f8725600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f8720113c30 con 0x7f87200761e0 2024-03-19T03:30:56.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.875+0000 7f871ea00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87200761e0 0x7f872019e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:56.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.875+0000 7f8717e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8720077420 0x7f872019e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:56.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.875+0000 7f871f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8720077dc0 0x7f87201a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:56.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.875+0000 7f871ea00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87200761e0 0x7f872019e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:60294/0 (socket says 172.21.15.117:60294) 2024-03-19T03:30:56.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.875+0000 7f871ea00640 1 -- 172.21.15.117:0/1921175394 learned_addr learned my addr 172.21.15.117:0/1921175394 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:56.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.876+0000 7f871f400640 1 -- 172.21.15.117:0/1921175394 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87200761e0 msgr2=0x7f872019e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:56.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.876+0000 7f871f400640 1 --2- 172.21.15.117:0/1921175394 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87200761e0 0x7f872019e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:56.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.876+0000 7f871f400640 1 -- 172.21.15.117:0/1921175394 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8720077420 msgr2=0x7f872019e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:56.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.876+0000 7f871f400640 1 --2- 172.21.15.117:0/1921175394 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8720077420 0x7f872019e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:56.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.876+0000 7f871f400640 1 -- 172.21.15.117:0/1921175394 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87201a5dc0 con 0x7f8720077dc0 2024-03-19T03:30:56.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.876+0000 7f871f400640 1 --2- 172.21.15.117:0/1921175394 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8720077dc0 0x7f87201a5750 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f871800ca30 tx=0x7f871800cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:56.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.876+0000 7f8717400640 1 -- 172.21.15.117:0/1921175394 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8718007d90 con 0x7f8720077dc0 2024-03-19T03:30:56.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.876+0000 7f8717400640 1 -- 172.21.15.117:0/1921175394 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8718005020 con 0x7f8720077dc0 2024-03-19T03:30:56.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.876+0000 7f8725600640 1 -- 172.21.15.117:0/1921175394 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87201a60b0 con 0x7f8720077dc0 2024-03-19T03:30:56.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.876+0000 7f8717400640 1 -- 172.21.15.117:0/1921175394 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871800f780 con 0x7f8720077dc0 2024-03-19T03:30:56.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.876+0000 7f8725600640 1 -- 172.21.15.117:0/1921175394 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87201a65d0 con 0x7f8720077dc0 2024-03-19T03:30:56.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.878+0000 7f8725600640 1 -- 172.21.15.117:0/1921175394 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87201103d0 con 0x7f8720077dc0 2024-03-19T03:30:56.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.879+0000 7f8717400640 1 -- 172.21.15.117:0/1921175394 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f87180051d0 con 0x7f8720077dc0 2024-03-19T03:30:56.880 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.880+0000 7f8717400640 1 --2- 172.21.15.117:0/1921175394 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f87000a2880 0x7f87000a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:56.880 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.880+0000 7f8717400640 1 -- 172.21.15.117:0/1921175394 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f87180cedd0 con 0x7f8720077dc0 2024-03-19T03:30:56.880 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.880+0000 7f871ea00640 1 --2- 172.21.15.117:0/1921175394 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f87000a2880 0x7f87000a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:56.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.881+0000 7f871ea00640 1 --2- 172.21.15.117:0/1921175394 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f87000a2880 0x7f87000a4d40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8710005fd0 tx=0x7f871000b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:56.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:56.884+0000 7f8717400640 1 -- 172.21.15.117:0/1921175394 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8718014070 con 0x7f8720077dc0 2024-03-19T03:30:57.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.080+0000 7f8725600640 1 -- 172.21.15.117:0/1921175394 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8720062420 con 0x7f8720077dc0 2024-03-19T03:30:57.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.080+0000 7f8717400640 1 -- 172.21.15.117:0/1921175394 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f871809b580 con 0x7f8720077dc0 2024-03-19T03:30:57.080 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 32 2024-03-19T03:30:57.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.083+0000 7f8725600640 1 -- 172.21.15.117:0/1921175394 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f87000a2880 msgr2=0x7f87000a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:57.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.083+0000 7f8725600640 1 --2- 172.21.15.117:0/1921175394 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f87000a2880 0x7f87000a4d40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8710005fd0 tx=0x7f871000b4a0 comp rx=0 tx=0).stop 2024-03-19T03:30:57.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.083+0000 7f8725600640 1 -- 172.21.15.117:0/1921175394 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8720077dc0 msgr2=0x7f87201a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:57.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.083+0000 7f8725600640 1 --2- 172.21.15.117:0/1921175394 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8720077dc0 0x7f87201a5750 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f871800ca30 tx=0x7f871800cef0 comp rx=0 tx=0).stop 2024-03-19T03:30:57.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.083+0000 7f8725600640 1 -- 172.21.15.117:0/1921175394 shutdown_connections 2024-03-19T03:30:57.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.083+0000 7f8725600640 1 --2- 172.21.15.117:0/1921175394 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f87000a2880 0x7f87000a4d40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:57.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.083+0000 7f8725600640 1 --2- 172.21.15.117:0/1921175394 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8720077dc0 0x7f87201a5750 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:57.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.083+0000 7f8725600640 1 --2- 172.21.15.117:0/1921175394 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8720077420 0x7f872019e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:57.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.083+0000 7f8725600640 1 --2- 172.21.15.117:0/1921175394 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87200761e0 0x7f872019e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:57.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.083+0000 7f8725600640 1 -- 172.21.15.117:0/1921175394 >> 172.21.15.117:0/1921175394 conn(0x7f87200fefd0 msgr2=0x7f8720100ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:57.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.083+0000 7f8725600640 1 -- 172.21.15.117:0/1921175394 shutdown_connections 2024-03-19T03:30:57.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.083+0000 7f8725600640 1 -- 172.21.15.117:0/1921175394 wait complete. 2024-03-19T03:30:57.102 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-19T03:30:57.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.299+0000 7f7283e00640 1 -- 172.21.15.117:0/2375715912 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f727c104c50 msgr2=0x7f727c1050d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:57.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.299+0000 7f7283e00640 1 --2- 172.21.15.117:0/2375715912 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f727c104c50 0x7f727c1050d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7270009900 tx=0x7f727002bc40 comp rx=0 tx=0).stop 2024-03-19T03:30:57.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.299+0000 7f7283e00640 1 -- 172.21.15.117:0/2375715912 shutdown_connections 2024-03-19T03:30:57.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.299+0000 7f7283e00640 1 --2- 172.21.15.117:0/2375715912 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f727c105610 0x7f727c10bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:57.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.299+0000 7f7283e00640 1 --2- 172.21.15.117:0/2375715912 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f727c104c50 0x7f727c1050d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:57.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.299+0000 7f7283e00640 1 --2- 172.21.15.117:0/2375715912 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f727c103a60 0x7f727c103e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:57.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.299+0000 7f7283e00640 1 -- 172.21.15.117:0/2375715912 >> 172.21.15.117:0/2375715912 conn(0x7f727c0fefd0 msgr2=0x7f727c101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:57.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.299+0000 7f7283e00640 1 -- 172.21.15.117:0/2375715912 shutdown_connections 2024-03-19T03:30:57.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.299+0000 7f7283e00640 1 -- 172.21.15.117:0/2375715912 wait complete. 2024-03-19T03:30:57.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.300+0000 7f7283e00640 1 Processor -- start 2024-03-19T03:30:57.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.301+0000 7f7283e00640 1 -- start start 2024-03-19T03:30:57.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.301+0000 7f7283e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f727c103a60 0x7f727c199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:57.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.301+0000 7f7283e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f727c104c50 0x7f727c19a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:57.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.301+0000 7f7283e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f727c105610 0x7f727c1a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:57.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.301+0000 7f7283e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f727c10f6c0 con 0x7f727c103a60 2024-03-19T03:30:57.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.301+0000 7f7283e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f727c10f540 con 0x7f727c104c50 2024-03-19T03:30:57.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.301+0000 7f7283e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f727c10f840 con 0x7f727c105610 2024-03-19T03:30:57.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.301+0000 7f7281400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f727c103a60 0x7f727c199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:57.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.301+0000 7f7280a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f727c104c50 0x7f727c19a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:57.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.301+0000 7f7280a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f727c104c50 0x7f727c19a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:53414/0 (socket says 172.21.15.117:53414) 2024-03-19T03:30:57.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.301+0000 7f7280a00640 1 -- 172.21.15.117:0/3381371883 learned_addr learned my addr 172.21.15.117:0/3381371883 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:30:57.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.301+0000 7f7281e00640 1 --2- 172.21.15.117:0/3381371883 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f727c105610 0x7f727c1a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:57.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.302+0000 7f7280a00640 1 -- 172.21.15.117:0/3381371883 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f727c105610 msgr2=0x7f727c1a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:57.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.302+0000 7f7280a00640 1 --2- 172.21.15.117:0/3381371883 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f727c105610 0x7f727c1a12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:57.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.302+0000 7f7280a00640 1 -- 172.21.15.117:0/3381371883 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f727c103a60 msgr2=0x7f727c199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:57.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.302+0000 7f7280a00640 1 --2- 172.21.15.117:0/3381371883 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f727c103a60 0x7f727c199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:57.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.302+0000 7f7280a00640 1 -- 172.21.15.117:0/3381371883 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f727c1a1960 con 0x7f727c104c50 2024-03-19T03:30:57.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.302+0000 7f7280a00640 1 --2- 172.21.15.117:0/3381371883 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f727c104c50 0x7f727c19a280 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7270009230 tx=0x7f7270005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:57.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.302+0000 7f727a000640 1 -- 172.21.15.117:0/3381371883 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7270041070 con 0x7f727c104c50 2024-03-19T03:30:57.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.302+0000 7f727a000640 1 -- 172.21.15.117:0/3381371883 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72700054e0 con 0x7f727c104c50 2024-03-19T03:30:57.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.302+0000 7f7283e00640 1 -- 172.21.15.117:0/3381371883 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f727c1a1bf0 con 0x7f727c104c50 2024-03-19T03:30:57.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.302+0000 7f727a000640 1 -- 172.21.15.117:0/3381371883 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7270034440 con 0x7f727c104c50 2024-03-19T03:30:57.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.302+0000 7f7283e00640 1 -- 172.21.15.117:0/3381371883 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f727c1a2130 con 0x7f727c104c50 2024-03-19T03:30:57.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.303+0000 7f7283e00640 1 -- 172.21.15.117:0/3381371883 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f727c10ae00 con 0x7f727c104c50 2024-03-19T03:30:57.304 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.305+0000 7f727a000640 1 -- 172.21.15.117:0/3381371883 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f7270044070 con 0x7f727c104c50 2024-03-19T03:30:57.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.306+0000 7f727a000640 1 --2- 172.21.15.117:0/3381371883 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f72640a2880 0x7f72640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:30:57.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.306+0000 7f727a000640 1 -- 172.21.15.117:0/3381371883 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f72700f08b0 con 0x7f727c104c50 2024-03-19T03:30:57.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.306+0000 7f7281400640 1 --2- 172.21.15.117:0/3381371883 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f72640a2880 0x7f72640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:30:57.307 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.307+0000 7f7281400640 1 --2- 172.21.15.117:0/3381371883 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f72640a2880 0x7f72640a4d40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f727c104ab0 tx=0x7f7274009720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:30:57.310 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.310+0000 7f727a000640 1 -- 172.21.15.117:0/3381371883 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f72700bd0c0 con 0x7f727c104c50 2024-03-19T03:30:57.523 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.523+0000 7f7283e00640 1 -- 172.21.15.117:0/3381371883 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f727c062420 con 0x7f727c104c50 2024-03-19T03:30:57.524 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.524+0000 7f727a000640 1 -- 172.21.15.117:0/3381371883 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+600 (secure 0 0 0) 0x7f72700c1160 con 0x7f727c104c50 2024-03-19T03:30:57.524 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:0/185684447 2024-03-20T03:22:38.134578+0000 2024-03-19T03:30:57.524 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:0/361367101 2024-03-20T03:22:38.134578+0000 2024-03-19T03:30:57.524 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:6801/2788067603 2024-03-20T03:22:38.134578+0000 2024-03-19T03:30:57.524 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:6801/1570899077 2024-03-20T03:22:03.905461+0000 2024-03-19T03:30:57.524 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:0/3506200675 2024-03-20T03:22:38.134578+0000 2024-03-19T03:30:57.524 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:6800/1570899077 2024-03-20T03:22:03.905461+0000 2024-03-19T03:30:57.525 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:0/3283471541 2024-03-20T03:22:03.905461+0000 2024-03-19T03:30:57.525 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:0/3750920169 2024-03-20T03:22:03.905461+0000 2024-03-19T03:30:57.525 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:0/2780813298 2024-03-20T03:22:03.905461+0000 2024-03-19T03:30:57.525 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:6800/2788067603 2024-03-20T03:22:38.134578+0000 2024-03-19T03:30:57.525 INFO:teuthology.orchestra.run.smithi117.stderr:listed 10 entries 2024-03-19T03:30:57.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.526+0000 7f7283e00640 1 -- 172.21.15.117:0/3381371883 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f72640a2880 msgr2=0x7f72640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:57.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.526+0000 7f7283e00640 1 --2- 172.21.15.117:0/3381371883 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f72640a2880 0x7f72640a4d40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f727c104ab0 tx=0x7f7274009720 comp rx=0 tx=0).stop 2024-03-19T03:30:57.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.526+0000 7f7283e00640 1 -- 172.21.15.117:0/3381371883 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f727c104c50 msgr2=0x7f727c19a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:30:57.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.527+0000 7f7283e00640 1 --2- 172.21.15.117:0/3381371883 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f727c104c50 0x7f727c19a280 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7270009230 tx=0x7f7270005dd0 comp rx=0 tx=0).stop 2024-03-19T03:30:57.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.527+0000 7f7283e00640 1 -- 172.21.15.117:0/3381371883 shutdown_connections 2024-03-19T03:30:57.527 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.527+0000 7f7283e00640 1 --2- 172.21.15.117:0/3381371883 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f727c105610 0x7f727c1a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:57.528 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.527+0000 7f7283e00640 1 --2- 172.21.15.117:0/3381371883 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f72640a2880 0x7f72640a4d40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:57.528 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.527+0000 7f7283e00640 1 --2- 172.21.15.117:0/3381371883 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f727c104c50 0x7f727c19a280 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:57.528 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.527+0000 7f7283e00640 1 --2- 172.21.15.117:0/3381371883 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f727c103a60 0x7f727c199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:30:57.528 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.527+0000 7f7283e00640 1 -- 172.21.15.117:0/3381371883 >> 172.21.15.117:0/3381371883 conn(0x7f727c0fefd0 msgr2=0x7f727c107fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:30:57.528 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.527+0000 7f7283e00640 1 -- 172.21.15.117:0/3381371883 shutdown_connections 2024-03-19T03:30:57.528 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:30:57.527+0000 7f7283e00640 1 -- 172.21.15.117:0/3381371883 wait complete. 2024-03-19T03:30:57.546 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi117.front.sepia.ceph.com... 2024-03-19T03:30:57.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:30:57.547 DEBUG:teuthology.orchestra.run.smithi117:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-19T03:30:57.639 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi117.stderr:ceph-fuse[76168]: fuse finished with error 0 and tester_r 0 2024-03-19T03:30:57.640 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:57 smithi117 ceph-mon[31838]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-19T03:30:57.640 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:57 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3001738285' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:30:57.640 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:57 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1921175394' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:30:57.647 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T03:30:57.647 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:57 smithi119 ceph-mon[66861]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-19T03:30:57.648 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:57 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3001738285' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:30:57.648 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:57 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1921175394' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:30:57.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:57 smithi178 ceph-mon[67578]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-19T03:30:57.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:57 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3001738285' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:30:57.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:57 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1921175394' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:30:58.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:58 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3381371883' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T03:30:58.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:58 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3381371883' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T03:30:58.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:58 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3381371883' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T03:30:59.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:30:59 smithi119 ceph-mon[66861]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-19T03:30:59.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:30:59 smithi117 ceph-mon[31838]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-19T03:30:59.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:30:59 smithi178 ceph-mon[67578]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-19T03:31:01.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:01 smithi119 ceph-mon[66861]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-19T03:31:01.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:01 smithi117 ceph-mon[31838]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-19T03:31:01.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:01 smithi178 ceph-mon[67578]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-19T03:31:03.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:03 smithi119 ceph-mon[66861]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-19T03:31:03.748 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi117.front.sepia.ceph.com 2024-03-19T03:31:03.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:31:03.748 DEBUG:teuthology.orchestra.run.smithi117:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-19T03:31:03.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:03 smithi117 ceph-mon[31838]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-19T03:31:03.813 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:31:03.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:03 smithi178 ceph-mon[67578]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-19T03:31:04.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.074+0000 7f0503600640 1 -- 172.21.15.117:0/1809039054 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04fc104c50 msgr2=0x7f04fc1050d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:04.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.074+0000 7f0503600640 1 --2- 172.21.15.117:0/1809039054 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04fc104c50 0x7f04fc1050d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f04e4009860 tx=0x7f04e402bb70 comp rx=0 tx=0).stop 2024-03-19T03:31:04.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.075+0000 7f0503600640 1 -- 172.21.15.117:0/1809039054 shutdown_connections 2024-03-19T03:31:04.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.075+0000 7f0503600640 1 --2- 172.21.15.117:0/1809039054 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f04fc105610 0x7f04fc10bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:04.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.075+0000 7f0503600640 1 --2- 172.21.15.117:0/1809039054 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04fc104c50 0x7f04fc1050d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:04.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.075+0000 7f0503600640 1 --2- 172.21.15.117:0/1809039054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04fc103a60 0x7f04fc103e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:04.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.075+0000 7f0503600640 1 -- 172.21.15.117:0/1809039054 >> 172.21.15.117:0/1809039054 conn(0x7f04fc0fefd0 msgr2=0x7f04fc101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:04.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.075+0000 7f0503600640 1 -- 172.21.15.117:0/1809039054 shutdown_connections 2024-03-19T03:31:04.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.075+0000 7f0503600640 1 -- 172.21.15.117:0/1809039054 wait complete. 2024-03-19T03:31:04.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.076+0000 7f0503600640 1 Processor -- start 2024-03-19T03:31:04.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.076+0000 7f0503600640 1 -- start start 2024-03-19T03:31:04.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.076+0000 7f0503600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04fc103a60 0x7f04fc199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:04.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.076+0000 7f0503600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f04fc104c50 0x7f04fc19a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:04.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.076+0000 7f0503600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04fc105610 0x7f04fc1a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:04.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.076+0000 7f0503600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f04fc10f6c0 con 0x7f04fc105610 2024-03-19T03:31:04.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.076+0000 7f0503600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f04fc10f540 con 0x7f04fc103a60 2024-03-19T03:31:04.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.076+0000 7f0503600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f04fc10f840 con 0x7f04fc104c50 2024-03-19T03:31:04.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.076+0000 7f0501600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04fc105610 0x7f04fc1a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:04.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.076+0000 7f0500c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04fc103a60 0x7f04fc199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:04.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.076+0000 7f0501600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04fc105610 0x7f04fc1a1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:53996/0 (socket says 172.21.15.117:53996) 2024-03-19T03:31:04.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.076+0000 7f0501600640 1 -- 172.21.15.117:0/2086990876 learned_addr learned my addr 172.21.15.117:0/2086990876 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:04.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.077+0000 7f04fbe00640 1 --2- 172.21.15.117:0/2086990876 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f04fc104c50 0x7f04fc19a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:04.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.077+0000 7f0500c00640 1 -- 172.21.15.117:0/2086990876 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f04fc104c50 msgr2=0x7f04fc19a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:04.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.077+0000 7f0500c00640 1 --2- 172.21.15.117:0/2086990876 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f04fc104c50 0x7f04fc19a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:04.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.077+0000 7f0500c00640 1 -- 172.21.15.117:0/2086990876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04fc105610 msgr2=0x7f04fc1a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:04.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.077+0000 7f0500c00640 1 --2- 172.21.15.117:0/2086990876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04fc105610 0x7f04fc1a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:04.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.077+0000 7f0500c00640 1 -- 172.21.15.117:0/2086990876 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04fc1a1980 con 0x7f04fc103a60 2024-03-19T03:31:04.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.077+0000 7f0501600640 1 --2- 172.21.15.117:0/2086990876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04fc105610 0x7f04fc1a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:31:04.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.077+0000 7f0500c00640 1 --2- 172.21.15.117:0/2086990876 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04fc103a60 0x7f04fc199d60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f04f4005fd0 tx=0x7f04f400c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:04.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.077+0000 7f04f9600640 1 -- 172.21.15.117:0/2086990876 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f4003970 con 0x7f04fc103a60 2024-03-19T03:31:04.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.078+0000 7f04f9600640 1 -- 172.21.15.117:0/2086990876 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04f4004020 con 0x7f04fc103a60 2024-03-19T03:31:04.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.078+0000 7f0503600640 1 -- 172.21.15.117:0/2086990876 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04fc1a1c70 con 0x7f04fc103a60 2024-03-19T03:31:04.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.078+0000 7f04f9600640 1 -- 172.21.15.117:0/2086990876 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f400b990 con 0x7f04fc103a60 2024-03-19T03:31:04.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.078+0000 7f0503600640 1 -- 172.21.15.117:0/2086990876 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04fc1a2190 con 0x7f04fc103a60 2024-03-19T03:31:04.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.079+0000 7f0503600640 1 -- 172.21.15.117:0/2086990876 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04fc10ad80 con 0x7f04fc103a60 2024-03-19T03:31:04.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.081+0000 7f04f9600640 1 -- 172.21.15.117:0/2086990876 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f04f401d020 con 0x7f04fc103a60 2024-03-19T03:31:04.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.082+0000 7f04f9600640 1 --2- 172.21.15.117:0/2086990876 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f04e00a2880 0x7f04e00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:04.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.082+0000 7f04f9600640 1 -- 172.21.15.117:0/2086990876 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f04f40cec10 con 0x7f04fc103a60 2024-03-19T03:31:04.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.082+0000 7f04fbe00640 1 --2- 172.21.15.117:0/2086990876 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f04e00a2880 0x7f04e00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:04.084 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.083+0000 7f04fbe00640 1 --2- 172.21.15.117:0/2086990876 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f04e00a2880 0x7f04e00a4d40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f04e4009830 tx=0x7f04e4034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:04.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.087+0000 7f04f9600640 1 -- 172.21.15.117:0/2086990876 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f04f409ba70 con 0x7f04fc103a60 2024-03-19T03:31:04.288 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.287+0000 7f0503600640 1 -- 172.21.15.117:0/2086990876 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f04fc0624c0 con 0x7f04fc103a60 2024-03-19T03:31:04.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.289+0000 7f04f9600640 1 -- 172.21.15.117:0/2086990876 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24210 (secure 0 0 0) 0x7f04f409b370 con 0x7f04fc103a60 2024-03-19T03:31:04.290 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:31:04.290 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":89,"fsid":"a3a82456-e59f-11ee-95c9-87774f69a715","created":"2024-03-19T03:21:30.427136+0000","modified":"2024-03-19T03:30:40.180745+0000","last_up_change":"2024-03-19T03:29:32.967505+0000","last_in_change":"2024-03-19T03:29:12.848740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:25:18.824270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T03:30:33.137170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T03:30:34.598838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T03:30:38.230023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b32eb0ee-9d4b-4748-87ec-eda02c907415","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6803","nonce":3014934900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6805","nonce":3014934900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6809","nonce":3014934900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6807","nonce":3014934900}]},"public_addr":"172.21.15.117:6803/3014934900","cluster_addr":"172.21.15.117:6805/3014934900","heartbeat_back_addr":"172.21.15.117:6809/3014934900","heartbeat_front_addr":"172.21.15.117:6807/3014934900","state":["exists","up"]},{"osd":1,"uuid":"e07585be-c016-4466-b07c-abbc22a58de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6801","nonce":2425927341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6803","nonce":2425927341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6807","nonce":2425927341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6805","nonce":2425927341}]},"public_addr":"172.21.15.119:6801/2425927341","cluster_addr":"172.21.15.119:6803/2425927341","heartbeat_back_addr":"172.21.15.119:6807/2425927341","heartbeat_front_addr":"172.21.15.119:6805/2425927341","state":["exists","up"]},{"osd":2,"uuid":"ede9edda-3b9f-41ca-ad04-5dcb738ab757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6801","nonce":3733812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6803","nonce":3733812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6807","nonce":3733812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6805","nonce":3733812989}]},"public_addr":"172.21.15.178:6801/3733812989","cluster_addr":"172.21.15.178:6803/3733812989","heartbeat_back_addr":"172.21.15.178:6807/3733812989","heartbeat_front_addr":"172.21.15.178:6805/3733812989","state":["exists","up"]},{"osd":3,"uuid":"f81c41e1-3a69-40db-a90f-58151286b081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6811","nonce":2178310135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6813","nonce":2178310135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6817","nonce":2178310135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6815","nonce":2178310135}]},"public_addr":"172.21.15.117:6811/2178310135","cluster_addr":"172.21.15.117:6813/2178310135","heartbeat_back_addr":"172.21.15.117:6817/2178310135","heartbeat_front_addr":"172.21.15.117:6815/2178310135","state":["exists","up"]},{"osd":4,"uuid":"caf7ce86-caf0-4eaf-bbab-bd2d93e63c33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6809","nonce":3455398086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6811","nonce":3455398086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6815","nonce":3455398086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6813","nonce":3455398086}]},"public_addr":"172.21.15.119:6809/3455398086","cluster_addr":"172.21.15.119:6811/3455398086","heartbeat_back_addr":"172.21.15.119:6815/3455398086","heartbeat_front_addr":"172.21.15.119:6813/3455398086","state":["exists","up"]},{"osd":5,"uuid":"20ba0b8a-2e0f-4ea7-b435-430e44766200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6809","nonce":1259631912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6811","nonce":1259631912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6815","nonce":1259631912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6813","nonce":1259631912}]},"public_addr":"172.21.15.178:6809/1259631912","cluster_addr":"172.21.15.178:6811/1259631912","heartbeat_back_addr":"172.21.15.178:6815/1259631912","heartbeat_front_addr":"172.21.15.178:6813/1259631912","state":["exists","up"]},{"osd":6,"uuid":"75c2ddd7-f837-42e0-ac47-8cfbcf97a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6819","nonce":864146757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6821","nonce":864146757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6825","nonce":864146757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6823","nonce":864146757}]},"public_addr":"172.21.15.117:6819/864146757","cluster_addr":"172.21.15.117:6821/864146757","heartbeat_back_addr":"172.21.15.117:6825/864146757","heartbeat_front_addr":"172.21.15.117:6823/864146757","state":["exists","up"]},{"osd":7,"uuid":"929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6817","nonce":1273365545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6819","nonce":1273365545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6823","nonce":1273365545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6821","nonce":1273365545}]},"public_addr":"172.21.15.119:6817/1273365545","cluster_addr":"172.21.15.119:6819/1273365545","heartbeat_back_addr":"172.21.15.119:6823/1273365545","heartbeat_front_addr":"172.21.15.119:6821/1273365545","state":["exists","up"]},{"osd":8,"uuid":"fd371461-a538-4781-92e1-5eead4ca3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6817","nonce":1059587501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6819","nonce":1059587501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6823","nonce":1059587501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6821","nonce":1059587501}]},"public_addr":"172.21.15.178:6817/1059587501","cluster_addr":"172.21.15.178:6819/1059587501","heartbeat_back_addr":"172.21.15.178:6823/1059587501","heartbeat_front_addr":"172.21.15.178:6821/1059587501","state":["exists","up"]},{"osd":9,"uuid":"0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6827","nonce":3653875600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6829","nonce":3653875600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6833","nonce":3653875600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6831","nonce":3653875600}]},"public_addr":"172.21.15.117:6827/3653875600","cluster_addr":"172.21.15.117:6829/3653875600","heartbeat_back_addr":"172.21.15.117:6833/3653875600","heartbeat_front_addr":"172.21.15.117:6831/3653875600","state":["exists","up"]},{"osd":10,"uuid":"de2049da-d19e-4386-afbf-fd6b5fddbcba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6825","nonce":1050976194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6827","nonce":1050976194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6831","nonce":1050976194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6829","nonce":1050976194}]},"public_addr":"172.21.15.119:6825/1050976194","cluster_addr":"172.21.15.119:6827/1050976194","heartbeat_back_addr":"172.21.15.119:6831/1050976194","heartbeat_front_addr":"172.21.15.119:6829/1050976194","state":["exists","up"]},{"osd":11,"uuid":"3cccc0f9-d273-4b6b-9b8e-c926371472dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6825","nonce":2915080368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6827","nonce":2915080368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6831","nonce":2915080368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6829","nonce":2915080368}]},"public_addr":"172.21.15.178:6825/2915080368","cluster_addr":"172.21.15.178:6827/2915080368","heartbeat_back_addr":"172.21.15.178:6831/2915080368","heartbeat_front_addr":"172.21.15.178:6829/2915080368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:27.876274+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:53.094706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:16.598813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:48.160419+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:14.048925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:38.814304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:12.542328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:38.942747+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:04.488963+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:38.196053+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:05.254728+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:30.874915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.117:0/185684447":"2024-03-20T03:22:38.134578+0000","172.21.15.117:0/361367101":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/2788067603":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3506200675":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6800/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3283471541":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3750920169":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/2780813298":"2024-03-20T03:22:03.905461+0000","172.21.15.117:6800/2788067603":"2024-03-20T03:22:38.134578+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:31:04.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.292+0000 7f0503600640 1 -- 172.21.15.117:0/2086990876 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f04e00a2880 msgr2=0x7f04e00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:04.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.292+0000 7f0503600640 1 --2- 172.21.15.117:0/2086990876 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f04e00a2880 0x7f04e00a4d40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f04e4009830 tx=0x7f04e4034ab0 comp rx=0 tx=0).stop 2024-03-19T03:31:04.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.292+0000 7f0503600640 1 -- 172.21.15.117:0/2086990876 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04fc103a60 msgr2=0x7f04fc199d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:04.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.292+0000 7f0503600640 1 --2- 172.21.15.117:0/2086990876 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04fc103a60 0x7f04fc199d60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f04f4005fd0 tx=0x7f04f400c800 comp rx=0 tx=0).stop 2024-03-19T03:31:04.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.292+0000 7f0503600640 1 -- 172.21.15.117:0/2086990876 shutdown_connections 2024-03-19T03:31:04.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.292+0000 7f0503600640 1 --2- 172.21.15.117:0/2086990876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04fc105610 0x7f04fc1a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:04.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.292+0000 7f0503600640 1 --2- 172.21.15.117:0/2086990876 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f04fc104c50 0x7f04fc19a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:04.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.292+0000 7f0503600640 1 --2- 172.21.15.117:0/2086990876 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f04e00a2880 0x7f04e00a4d40 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:04.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.292+0000 7f0503600640 1 --2- 172.21.15.117:0/2086990876 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04fc103a60 0x7f04fc199d60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:04.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.292+0000 7f0503600640 1 -- 172.21.15.117:0/2086990876 >> 172.21.15.117:0/2086990876 conn(0x7f04fc0fefd0 msgr2=0x7f04fc107f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:04.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.292+0000 7f0503600640 1 -- 172.21.15.117:0/2086990876 shutdown_connections 2024-03-19T03:31:04.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.292+0000 7f0503600640 1 -- 172.21.15.117:0/2086990876 wait complete. 2024-03-19T03:31:04.312 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:31:04.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.515+0000 7f44ae200640 1 -- 172.21.15.117:0/3148525452 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44a8108fb0 msgr2=0x7f44a8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:04.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.515+0000 7f44ae200640 1 --2- 172.21.15.117:0/3148525452 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44a8108fb0 0x7f44a8109430 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4490009900 tx=0x7f449002bc30 comp rx=0 tx=0).stop 2024-03-19T03:31:04.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.515+0000 7f44ae200640 1 -- 172.21.15.117:0/3148525452 shutdown_connections 2024-03-19T03:31:04.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.515+0000 7f44ae200640 1 --2- 172.21.15.117:0/3148525452 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44a8109970 0x7f44a8110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:04.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.515+0000 7f44ae200640 1 --2- 172.21.15.117:0/3148525452 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44a8108fb0 0x7f44a8109430 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:04.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.515+0000 7f44ae200640 1 --2- 172.21.15.117:0/3148525452 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44a8107e60 0x7f44a8108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:04.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.515+0000 7f44ae200640 1 -- 172.21.15.117:0/3148525452 >> 172.21.15.117:0/3148525452 conn(0x7f44a8076e30 msgr2=0x7f44a8079250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:04.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.515+0000 7f44ae200640 1 -- 172.21.15.117:0/3148525452 shutdown_connections 2024-03-19T03:31:04.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.515+0000 7f44ae200640 1 -- 172.21.15.117:0/3148525452 wait complete. 2024-03-19T03:31:04.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.516+0000 7f44ae200640 1 Processor -- start 2024-03-19T03:31:04.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.516+0000 7f44ae200640 1 -- start start 2024-03-19T03:31:04.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.517+0000 7f44ae200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44a8107e60 0x7f44a819e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:04.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.517+0000 7f44ae200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44a8108fb0 0x7f44a819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:04.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.517+0000 7f44ae200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44a8109970 0x7f44a81a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:04.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.517+0000 7f44ae200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f44a8113b00 con 0x7f44a8107e60 2024-03-19T03:31:04.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.517+0000 7f44ae200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f44a8113980 con 0x7f44a8108fb0 2024-03-19T03:31:04.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.517+0000 7f44ae200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f44a8113c80 con 0x7f44a8109970 2024-03-19T03:31:04.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.517+0000 7f449ea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44a8108fb0 0x7f44a819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:04.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.517+0000 7f449ea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44a8108fb0 0x7f44a819e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:53454/0 (socket says 172.21.15.117:53454) 2024-03-19T03:31:04.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.517+0000 7f449ea00640 1 -- 172.21.15.117:0/2761801590 learned_addr learned my addr 172.21.15.117:0/2761801590 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:04.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.517+0000 7f44a7400640 1 --2- 172.21.15.117:0/2761801590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44a8107e60 0x7f44a819e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:04.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.517+0000 7f44a7e00640 1 --2- 172.21.15.117:0/2761801590 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44a8109970 0x7f44a81a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:04.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.517+0000 7f449ea00640 1 -- 172.21.15.117:0/2761801590 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44a8109970 msgr2=0x7f44a81a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:04.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.517+0000 7f449ea00640 1 --2- 172.21.15.117:0/2761801590 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44a8109970 0x7f44a81a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:04.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.517+0000 7f449ea00640 1 -- 172.21.15.117:0/2761801590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44a8107e60 msgr2=0x7f44a819e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:04.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.517+0000 7f449ea00640 1 --2- 172.21.15.117:0/2761801590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44a8107e60 0x7f44a819e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:04.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.517+0000 7f449ea00640 1 -- 172.21.15.117:0/2761801590 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44a81a5d30 con 0x7f44a8108fb0 2024-03-19T03:31:04.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.518+0000 7f449ea00640 1 --2- 172.21.15.117:0/2761801590 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44a8108fb0 0x7f44a819e650 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4490000c00 tx=0x7f449002c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:04.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.518+0000 7f44a4c00640 1 -- 172.21.15.117:0/2761801590 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4490041070 con 0x7f44a8108fb0 2024-03-19T03:31:04.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.518+0000 7f44a4c00640 1 -- 172.21.15.117:0/2761801590 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4490004450 con 0x7f44a8108fb0 2024-03-19T03:31:04.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.518+0000 7f44ae200640 1 -- 172.21.15.117:0/2761801590 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44a81a5fc0 con 0x7f44a8108fb0 2024-03-19T03:31:04.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.518+0000 7f44a4c00640 1 -- 172.21.15.117:0/2761801590 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4490034440 con 0x7f44a8108fb0 2024-03-19T03:31:04.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.518+0000 7f44ae200640 1 -- 172.21.15.117:0/2761801590 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44a81a6420 con 0x7f44a8108fb0 2024-03-19T03:31:04.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.519+0000 7f44ae200640 1 -- 172.21.15.117:0/2761801590 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44a810f170 con 0x7f44a8108fb0 2024-03-19T03:31:04.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.521+0000 7f44a4c00640 1 -- 172.21.15.117:0/2761801590 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f4490044070 con 0x7f44a8108fb0 2024-03-19T03:31:04.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.522+0000 7f44a4c00640 1 --2- 172.21.15.117:0/2761801590 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f448c0a2880 0x7f448c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:04.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.522+0000 7f44a4c00640 1 -- 172.21.15.117:0/2761801590 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f44900f0730 con 0x7f44a8108fb0 2024-03-19T03:31:04.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.522+0000 7f44a7400640 1 --2- 172.21.15.117:0/2761801590 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f448c0a2880 0x7f448c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:04.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.523+0000 7f44a7400640 1 --2- 172.21.15.117:0/2761801590 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f448c0a2880 0x7f448c0a4d40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4498005fd0 tx=0x7f4498009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:04.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.530+0000 7f44a4c00640 1 -- 172.21.15.117:0/2761801590 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f44900bcf70 con 0x7f44a8108fb0 2024-03-19T03:31:04.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.730+0000 7f44ae200640 1 -- 172.21.15.117:0/2761801590 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f44a80624c0 con 0x7f44a8108fb0 2024-03-19T03:31:04.733 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.732+0000 7f44a4c00640 1 -- 172.21.15.117:0/2761801590 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24210 (secure 0 0 0) 0x7f44900c1010 con 0x7f44a8108fb0 2024-03-19T03:31:04.734 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:31:04.734 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":89,"fsid":"a3a82456-e59f-11ee-95c9-87774f69a715","created":"2024-03-19T03:21:30.427136+0000","modified":"2024-03-19T03:30:40.180745+0000","last_up_change":"2024-03-19T03:29:32.967505+0000","last_in_change":"2024-03-19T03:29:12.848740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:25:18.824270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T03:30:33.137170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T03:30:34.598838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T03:30:38.230023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b32eb0ee-9d4b-4748-87ec-eda02c907415","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6803","nonce":3014934900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6805","nonce":3014934900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6809","nonce":3014934900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6807","nonce":3014934900}]},"public_addr":"172.21.15.117:6803/3014934900","cluster_addr":"172.21.15.117:6805/3014934900","heartbeat_back_addr":"172.21.15.117:6809/3014934900","heartbeat_front_addr":"172.21.15.117:6807/3014934900","state":["exists","up"]},{"osd":1,"uuid":"e07585be-c016-4466-b07c-abbc22a58de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6801","nonce":2425927341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6803","nonce":2425927341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6807","nonce":2425927341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6805","nonce":2425927341}]},"public_addr":"172.21.15.119:6801/2425927341","cluster_addr":"172.21.15.119:6803/2425927341","heartbeat_back_addr":"172.21.15.119:6807/2425927341","heartbeat_front_addr":"172.21.15.119:6805/2425927341","state":["exists","up"]},{"osd":2,"uuid":"ede9edda-3b9f-41ca-ad04-5dcb738ab757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6801","nonce":3733812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6803","nonce":3733812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6807","nonce":3733812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6805","nonce":3733812989}]},"public_addr":"172.21.15.178:6801/3733812989","cluster_addr":"172.21.15.178:6803/3733812989","heartbeat_back_addr":"172.21.15.178:6807/3733812989","heartbeat_front_addr":"172.21.15.178:6805/3733812989","state":["exists","up"]},{"osd":3,"uuid":"f81c41e1-3a69-40db-a90f-58151286b081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6811","nonce":2178310135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6813","nonce":2178310135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6817","nonce":2178310135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6815","nonce":2178310135}]},"public_addr":"172.21.15.117:6811/2178310135","cluster_addr":"172.21.15.117:6813/2178310135","heartbeat_back_addr":"172.21.15.117:6817/2178310135","heartbeat_front_addr":"172.21.15.117:6815/2178310135","state":["exists","up"]},{"osd":4,"uuid":"caf7ce86-caf0-4eaf-bbab-bd2d93e63c33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6809","nonce":3455398086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6811","nonce":3455398086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6815","nonce":3455398086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6813","nonce":3455398086}]},"public_addr":"172.21.15.119:6809/3455398086","cluster_addr":"172.21.15.119:6811/3455398086","heartbeat_back_addr":"172.21.15.119:6815/3455398086","heartbeat_front_addr":"172.21.15.119:6813/3455398086","state":["exists","up"]},{"osd":5,"uuid":"20ba0b8a-2e0f-4ea7-b435-430e44766200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6809","nonce":1259631912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6811","nonce":1259631912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6815","nonce":1259631912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6813","nonce":1259631912}]},"public_addr":"172.21.15.178:6809/1259631912","cluster_addr":"172.21.15.178:6811/1259631912","heartbeat_back_addr":"172.21.15.178:6815/1259631912","heartbeat_front_addr":"172.21.15.178:6813/1259631912","state":["exists","up"]},{"osd":6,"uuid":"75c2ddd7-f837-42e0-ac47-8cfbcf97a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6819","nonce":864146757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6821","nonce":864146757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6825","nonce":864146757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6823","nonce":864146757}]},"public_addr":"172.21.15.117:6819/864146757","cluster_addr":"172.21.15.117:6821/864146757","heartbeat_back_addr":"172.21.15.117:6825/864146757","heartbeat_front_addr":"172.21.15.117:6823/864146757","state":["exists","up"]},{"osd":7,"uuid":"929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6817","nonce":1273365545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6819","nonce":1273365545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6823","nonce":1273365545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6821","nonce":1273365545}]},"public_addr":"172.21.15.119:6817/1273365545","cluster_addr":"172.21.15.119:6819/1273365545","heartbeat_back_addr":"172.21.15.119:6823/1273365545","heartbeat_front_addr":"172.21.15.119:6821/1273365545","state":["exists","up"]},{"osd":8,"uuid":"fd371461-a538-4781-92e1-5eead4ca3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6817","nonce":1059587501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6819","nonce":1059587501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6823","nonce":1059587501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6821","nonce":1059587501}]},"public_addr":"172.21.15.178:6817/1059587501","cluster_addr":"172.21.15.178:6819/1059587501","heartbeat_back_addr":"172.21.15.178:6823/1059587501","heartbeat_front_addr":"172.21.15.178:6821/1059587501","state":["exists","up"]},{"osd":9,"uuid":"0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6827","nonce":3653875600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6829","nonce":3653875600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6833","nonce":3653875600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6831","nonce":3653875600}]},"public_addr":"172.21.15.117:6827/3653875600","cluster_addr":"172.21.15.117:6829/3653875600","heartbeat_back_addr":"172.21.15.117:6833/3653875600","heartbeat_front_addr":"172.21.15.117:6831/3653875600","state":["exists","up"]},{"osd":10,"uuid":"de2049da-d19e-4386-afbf-fd6b5fddbcba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6825","nonce":1050976194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6827","nonce":1050976194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6831","nonce":1050976194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6829","nonce":1050976194}]},"public_addr":"172.21.15.119:6825/1050976194","cluster_addr":"172.21.15.119:6827/1050976194","heartbeat_back_addr":"172.21.15.119:6831/1050976194","heartbeat_front_addr":"172.21.15.119:6829/1050976194","state":["exists","up"]},{"osd":11,"uuid":"3cccc0f9-d273-4b6b-9b8e-c926371472dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6825","nonce":2915080368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6827","nonce":2915080368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6831","nonce":2915080368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6829","nonce":2915080368}]},"public_addr":"172.21.15.178:6825/2915080368","cluster_addr":"172.21.15.178:6827/2915080368","heartbeat_back_addr":"172.21.15.178:6831/2915080368","heartbeat_front_addr":"172.21.15.178:6829/2915080368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:27.876274+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:53.094706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:16.598813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:48.160419+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:14.048925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:38.814304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:12.542328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:38.942747+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:04.488963+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:38.196053+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:05.254728+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:30.874915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.117:0/185684447":"2024-03-20T03:22:38.134578+0000","172.21.15.117:0/361367101":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/2788067603":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3506200675":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6800/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3283471541":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3750920169":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/2780813298":"2024-03-20T03:22:03.905461+0000","172.21.15.117:6800/2788067603":"2024-03-20T03:22:38.134578+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:31:04.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:04 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2086990876' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:31:04.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.735+0000 7f44ae200640 1 -- 172.21.15.117:0/2761801590 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f448c0a2880 msgr2=0x7f448c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:04.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.735+0000 7f44ae200640 1 --2- 172.21.15.117:0/2761801590 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f448c0a2880 0x7f448c0a4d40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4498005fd0 tx=0x7f4498009500 comp rx=0 tx=0).stop 2024-03-19T03:31:04.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.735+0000 7f44ae200640 1 -- 172.21.15.117:0/2761801590 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44a8108fb0 msgr2=0x7f44a819e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:04.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.735+0000 7f44ae200640 1 --2- 172.21.15.117:0/2761801590 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44a8108fb0 0x7f44a819e650 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4490000c00 tx=0x7f449002c640 comp rx=0 tx=0).stop 2024-03-19T03:31:04.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.735+0000 7f44ae200640 1 -- 172.21.15.117:0/2761801590 shutdown_connections 2024-03-19T03:31:04.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.735+0000 7f44ae200640 1 --2- 172.21.15.117:0/2761801590 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44a8109970 0x7f44a81a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:04.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.735+0000 7f44ae200640 1 --2- 172.21.15.117:0/2761801590 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f448c0a2880 0x7f448c0a4d40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:04.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.735+0000 7f44ae200640 1 --2- 172.21.15.117:0/2761801590 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f44a8108fb0 0x7f44a819e650 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:04.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.735+0000 7f44ae200640 1 --2- 172.21.15.117:0/2761801590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44a8107e60 0x7f44a819e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:04.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.735+0000 7f44ae200640 1 -- 172.21.15.117:0/2761801590 >> 172.21.15.117:0/2761801590 conn(0x7f44a8076e30 msgr2=0x7f44a810c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:04.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.735+0000 7f44ae200640 1 -- 172.21.15.117:0/2761801590 shutdown_connections 2024-03-19T03:31:04.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.735+0000 7f44ae200640 1 -- 172.21.15.117:0/2761801590 wait complete. 2024-03-19T03:31:04.752 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:04 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2086990876' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:31:04.756 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-19T03:31:04.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:04 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2086990876' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:31:04.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.969+0000 7f4555400640 1 -- 172.21.15.117:0/2469047469 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45501033e0 msgr2=0x7f4550109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:04.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.969+0000 7f4555400640 1 --2- 172.21.15.117:0/2469047469 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45501033e0 0x7f4550109c70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4540009900 tx=0x7f454002bc50 comp rx=0 tx=0).stop 2024-03-19T03:31:04.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.970+0000 7f4555400640 1 -- 172.21.15.117:0/2469047469 shutdown_connections 2024-03-19T03:31:04.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.970+0000 7f4555400640 1 --2- 172.21.15.117:0/2469047469 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45501033e0 0x7f4550109c70 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:04.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.970+0000 7f4555400640 1 --2- 172.21.15.117:0/2469047469 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4550102a20 0x7f4550102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:04.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.970+0000 7f4555400640 1 --2- 172.21.15.117:0/2469047469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4550101830 0x7f4550101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:04.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.970+0000 7f4555400640 1 -- 172.21.15.117:0/2469047469 >> 172.21.15.117:0/2469047469 conn(0x7f45500fcda0 msgr2=0x7f45500ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:04.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.970+0000 7f4555400640 1 -- 172.21.15.117:0/2469047469 shutdown_connections 2024-03-19T03:31:04.972 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.970+0000 7f4555400640 1 -- 172.21.15.117:0/2469047469 wait complete. 2024-03-19T03:31:04.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.971+0000 7f4555400640 1 Processor -- start 2024-03-19T03:31:04.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.971+0000 7f4555400640 1 -- start start 2024-03-19T03:31:04.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.971+0000 7f4555400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4550101830 0x7f45500755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:04.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.971+0000 7f4555400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4550102a20 0x7f4550073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:04.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.971+0000 7f4555400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45501033e0 0x7f4550074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:04.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.971+0000 7f4555400640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f455010d500 con 0x7f4550102a20 2024-03-19T03:31:04.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.971+0000 7f4555400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f455010d380 con 0x7f4550101830 2024-03-19T03:31:04.974 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.971+0000 7f4555400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f455010d680 con 0x7f45501033e0 2024-03-19T03:31:04.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.971+0000 7f454e800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4550101830 0x7f45500755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:04.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.971+0000 7f454e800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4550101830 0x7f45500755d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:53472/0 (socket says 172.21.15.117:53472) 2024-03-19T03:31:04.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.971+0000 7f454e800640 1 -- 172.21.15.117:0/2231963634 learned_addr learned my addr 172.21.15.117:0/2231963634 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:04.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.971+0000 7f454de00640 1 --2- 172.21.15.117:0/2231963634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4550102a20 0x7f4550073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:04.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.971+0000 7f454f200640 1 --2- 172.21.15.117:0/2231963634 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45501033e0 0x7f4550074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:04.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.972+0000 7f454e800640 1 -- 172.21.15.117:0/2231963634 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45501033e0 msgr2=0x7f4550074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:04.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.972+0000 7f454e800640 1 --2- 172.21.15.117:0/2231963634 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45501033e0 0x7f4550074150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:04.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.972+0000 7f454e800640 1 -- 172.21.15.117:0/2231963634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4550102a20 msgr2=0x7f4550073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:04.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.972+0000 7f454e800640 1 --2- 172.21.15.117:0/2231963634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4550102a20 0x7f4550073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:04.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.972+0000 7f454e800640 1 -- 172.21.15.117:0/2231963634 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45500749e0 con 0x7f4550101830 2024-03-19T03:31:04.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.972+0000 7f454e800640 1 --2- 172.21.15.117:0/2231963634 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4550101830 0x7f45500755d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f454800ebc0 tx=0x7f454800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:04.975 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.972+0000 7f4547400640 1 -- 172.21.15.117:0/2231963634 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4548003b70 con 0x7f4550101830 2024-03-19T03:31:04.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.972+0000 7f4547400640 1 -- 172.21.15.117:0/2231963634 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4548003d10 con 0x7f4550101830 2024-03-19T03:31:04.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.972+0000 7f4547400640 1 -- 172.21.15.117:0/2231963634 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4548019980 con 0x7f4550101830 2024-03-19T03:31:04.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.972+0000 7f4555400640 1 -- 172.21.15.117:0/2231963634 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45501a1c30 con 0x7f4550101830 2024-03-19T03:31:04.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.972+0000 7f4555400640 1 -- 172.21.15.117:0/2231963634 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45501a2180 con 0x7f4550101830 2024-03-19T03:31:04.976 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.974+0000 7f4555400640 1 -- 172.21.15.117:0/2231963634 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4550108b50 con 0x7f4550101830 2024-03-19T03:31:04.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.975+0000 7f4547400640 1 -- 172.21.15.117:0/2231963634 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f4548019b20 con 0x7f4550101830 2024-03-19T03:31:04.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.976+0000 7f4547400640 1 --2- 172.21.15.117:0/2231963634 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f45340a2800 0x7f45340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:04.977 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.977+0000 7f4547400640 1 -- 172.21.15.117:0/2231963634 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f45480cf530 con 0x7f4550101830 2024-03-19T03:31:04.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.977+0000 7f454de00640 1 --2- 172.21.15.117:0/2231963634 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f45340a2800 0x7f45340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:04.978 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.977+0000 7f454de00640 1 --2- 172.21.15.117:0/2231963634 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f45340a2800 0x7f45340a4cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4550075340 tx=0x7f4538005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:04.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:04.985+0000 7f4547400640 1 -- 172.21.15.117:0/2231963634 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f454809c330 con 0x7f4550101830 2024-03-19T03:31:05.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.223+0000 7f4555400640 1 -- 172.21.15.117:0/2231963634 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f4550062420 con 0x7f4550101830 2024-03-19T03:31:05.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.421+0000 7f4547400640 1 -- 172.21.15.117:0/2231963634 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f454809bc30 con 0x7f4550101830 2024-03-19T03:31:05.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.425+0000 7f4555400640 1 -- 172.21.15.117:0/2231963634 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f45340a2800 msgr2=0x7f45340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:05.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.425+0000 7f4555400640 1 --2- 172.21.15.117:0/2231963634 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f45340a2800 0x7f45340a4cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4550075340 tx=0x7f4538005f90 comp rx=0 tx=0).stop 2024-03-19T03:31:05.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.425+0000 7f4555400640 1 -- 172.21.15.117:0/2231963634 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4550101830 msgr2=0x7f45500755d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:05.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.425+0000 7f4555400640 1 --2- 172.21.15.117:0/2231963634 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4550101830 0x7f45500755d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f454800ebc0 tx=0x7f454800c6a0 comp rx=0 tx=0).stop 2024-03-19T03:31:05.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.425+0000 7f4555400640 1 -- 172.21.15.117:0/2231963634 shutdown_connections 2024-03-19T03:31:05.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.425+0000 7f4555400640 1 --2- 172.21.15.117:0/2231963634 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45501033e0 0x7f4550074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:05.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.425+0000 7f4555400640 1 --2- 172.21.15.117:0/2231963634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4550102a20 0x7f4550073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:05.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.425+0000 7f4555400640 1 --2- 172.21.15.117:0/2231963634 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f45340a2800 0x7f45340a4cc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:05.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.425+0000 7f4555400640 1 --2- 172.21.15.117:0/2231963634 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4550101830 0x7f45500755d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:05.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.425+0000 7f4555400640 1 -- 172.21.15.117:0/2231963634 >> 172.21.15.117:0/2231963634 conn(0x7f45500fcda0 msgr2=0x7f4550105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:05.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.425+0000 7f4555400640 1 -- 172.21.15.117:0/2231963634 shutdown_connections 2024-03-19T03:31:05.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.426+0000 7f4555400640 1 -- 172.21.15.117:0/2231963634 wait complete. 2024-03-19T03:31:05.451 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-19T03:31:05.452 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-19T03:31:05.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.669+0000 7fe412600640 1 -- 172.21.15.117:0/4224293653 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe40c108f30 msgr2=0x7fe40c1093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:05.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.669+0000 7fe412600640 1 --2- 172.21.15.117:0/4224293653 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe40c108f30 0x7fe40c1093b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8009900 tx=0x7fe3f802bc30 comp rx=0 tx=0).stop 2024-03-19T03:31:05.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.669+0000 7fe412600640 1 -- 172.21.15.117:0/4224293653 shutdown_connections 2024-03-19T03:31:05.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.669+0000 7fe412600640 1 --2- 172.21.15.117:0/4224293653 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe40c1098f0 0x7fe40c110180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:05.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.669+0000 7fe412600640 1 --2- 172.21.15.117:0/4224293653 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe40c108f30 0x7fe40c1093b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:05.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.669+0000 7fe412600640 1 --2- 172.21.15.117:0/4224293653 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe40c107d40 0x7fe40c108160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:05.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.669+0000 7fe412600640 1 -- 172.21.15.117:0/4224293653 >> 172.21.15.117:0/4224293653 conn(0x7fe40c076cf0 msgr2=0x7fe40c079110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:05.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.669+0000 7fe412600640 1 -- 172.21.15.117:0/4224293653 shutdown_connections 2024-03-19T03:31:05.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.669+0000 7fe412600640 1 -- 172.21.15.117:0/4224293653 wait complete. 2024-03-19T03:31:05.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.670+0000 7fe412600640 1 Processor -- start 2024-03-19T03:31:05.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.670+0000 7fe412600640 1 -- start start 2024-03-19T03:31:05.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.671+0000 7fe412600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe40c107d40 0x7fe40c19e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:05.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.671+0000 7fe412600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe40c108f30 0x7fe40c19e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:05.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.671+0000 7fe412600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe40c1098f0 0x7fe40c1a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:05.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.671+0000 7fe412600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe40c113a10 con 0x7fe40c107d40 2024-03-19T03:31:05.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.671+0000 7fe412600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe40c113890 con 0x7fe40c108f30 2024-03-19T03:31:05.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.671+0000 7fe412600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fe40c113b90 con 0x7fe40c1098f0 2024-03-19T03:31:05.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.671+0000 7fe411200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe40c107d40 0x7fe40c19e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:05.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.671+0000 7fe40be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe40c108f30 0x7fe40c19e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:05.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.671+0000 7fe40be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe40c108f30 0x7fe40c19e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:53484/0 (socket says 172.21.15.117:53484) 2024-03-19T03:31:05.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.671+0000 7fe411200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe40c107d40 0x7fe40c19e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:54044/0 (socket says 172.21.15.117:54044) 2024-03-19T03:31:05.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.671+0000 7fe40be00640 1 -- 172.21.15.117:0/4144559206 learned_addr learned my addr 172.21.15.117:0/4144559206 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:05.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.671+0000 7fe411c00640 1 --2- 172.21.15.117:0/4144559206 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe40c1098f0 0x7fe40c1a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:05.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.671+0000 7fe40be00640 1 -- 172.21.15.117:0/4144559206 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe40c1098f0 msgr2=0x7fe40c1a55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:05.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.671+0000 7fe40be00640 1 --2- 172.21.15.117:0/4144559206 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe40c1098f0 0x7fe40c1a55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:05.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.671+0000 7fe40be00640 1 -- 172.21.15.117:0/4144559206 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe40c107d40 msgr2=0x7fe40c19e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:05.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.671+0000 7fe40be00640 1 --2- 172.21.15.117:0/4144559206 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe40c107d40 0x7fe40c19e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:05.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.671+0000 7fe40be00640 1 -- 172.21.15.117:0/4144559206 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe40c1a5c50 con 0x7fe40c108f30 2024-03-19T03:31:05.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.671+0000 7fe411200640 1 --2- 172.21.15.117:0/4144559206 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe40c107d40 0x7fe40c19e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:31:05.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.672+0000 7fe40be00640 1 --2- 172.21.15.117:0/4144559206 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe40c108f30 0x7fe40c19e570 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8005ab0 tx=0x7fe3f802c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:05.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.672+0000 7fe409600640 1 -- 172.21.15.117:0/4144559206 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8043020 con 0x7fe40c108f30 2024-03-19T03:31:05.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.672+0000 7fe409600640 1 -- 172.21.15.117:0/4144559206 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3f80054e0 con 0x7fe40c108f30 2024-03-19T03:31:05.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.672+0000 7fe412600640 1 -- 172.21.15.117:0/4144559206 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe40c1a5ee0 con 0x7fe40c108f30 2024-03-19T03:31:05.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.672+0000 7fe409600640 1 -- 172.21.15.117:0/4144559206 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8003ce0 con 0x7fe40c108f30 2024-03-19T03:31:05.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.672+0000 7fe412600640 1 -- 172.21.15.117:0/4144559206 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe40c1a63a0 con 0x7fe40c108f30 2024-03-19T03:31:05.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.673+0000 7fe412600640 1 -- 172.21.15.117:0/4144559206 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe40c10f060 con 0x7fe40c108f30 2024-03-19T03:31:05.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.675+0000 7fe409600640 1 -- 172.21.15.117:0/4144559206 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fe3f8046020 con 0x7fe40c108f30 2024-03-19T03:31:05.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.676+0000 7fe409600640 1 --2- 172.21.15.117:0/4144559206 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe3f40a2800 0x7fe3f40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:05.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.676+0000 7fe409600640 1 -- 172.21.15.117:0/4144559206 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe3f80f0e40 con 0x7fe40c108f30 2024-03-19T03:31:05.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.676+0000 7fe411200640 1 --2- 172.21.15.117:0/4144559206 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe3f40a2800 0x7fe3f40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:05.678 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.677+0000 7fe411200640 1 --2- 172.21.15.117:0/4144559206 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe3f40a2800 0x7fe3f40a4cc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe400005fd0 tx=0x7fe40000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:05.685 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.685+0000 7fe409600640 1 -- 172.21.15.117:0/4144559206 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe3f804d020 con 0x7fe40c108f30 2024-03-19T03:31:05.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:05 smithi119 ceph-mon[66861]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-19T03:31:05.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:05 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2761801590' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:31:05.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:05 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2231963634' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-19T03:31:05.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:05 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-19T03:31:05.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:05 smithi117 ceph-mon[31838]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-19T03:31:05.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:05 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2761801590' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:31:05.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:05 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2231963634' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-19T03:31:05.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:05 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-19T03:31:05.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:05 smithi178 ceph-mon[67578]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-19T03:31:05.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:05 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2761801590' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:31:05.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:05 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2231963634' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-19T03:31:05.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:05 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-19T03:31:05.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:05.924+0000 7fe412600640 1 -- 172.21.15.117:0/4144559206 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7fe40c062420 con 0x7fe40c108f30 2024-03-19T03:31:06.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.443+0000 7fe409600640 1 -- 172.21.15.117:0/4144559206 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7fe3f80bdca0 con 0x7fe40c108f30 2024-03-19T03:31:06.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.446+0000 7fe412600640 1 -- 172.21.15.117:0/4144559206 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe3f40a2800 msgr2=0x7fe3f40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:06.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.446+0000 7fe412600640 1 --2- 172.21.15.117:0/4144559206 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe3f40a2800 0x7fe3f40a4cc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe400005fd0 tx=0x7fe40000b390 comp rx=0 tx=0).stop 2024-03-19T03:31:06.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.446+0000 7fe412600640 1 -- 172.21.15.117:0/4144559206 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe40c108f30 msgr2=0x7fe40c19e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:06.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.446+0000 7fe412600640 1 --2- 172.21.15.117:0/4144559206 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe40c108f30 0x7fe40c19e570 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8005ab0 tx=0x7fe3f802c6c0 comp rx=0 tx=0).stop 2024-03-19T03:31:06.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.447+0000 7fe412600640 1 -- 172.21.15.117:0/4144559206 shutdown_connections 2024-03-19T03:31:06.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.447+0000 7fe412600640 1 --2- 172.21.15.117:0/4144559206 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe40c1098f0 0x7fe40c1a55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:06.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.447+0000 7fe412600640 1 --2- 172.21.15.117:0/4144559206 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe3f40a2800 0x7fe3f40a4cc0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:06.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.447+0000 7fe412600640 1 --2- 172.21.15.117:0/4144559206 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe40c108f30 0x7fe40c19e570 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:06.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.447+0000 7fe412600640 1 --2- 172.21.15.117:0/4144559206 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe40c107d40 0x7fe40c19e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:06.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.447+0000 7fe412600640 1 -- 172.21.15.117:0/4144559206 >> 172.21.15.117:0/4144559206 conn(0x7fe40c076cf0 msgr2=0x7fe40c10c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:06.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.447+0000 7fe412600640 1 -- 172.21.15.117:0/4144559206 shutdown_connections 2024-03-19T03:31:06.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.447+0000 7fe412600640 1 -- 172.21.15.117:0/4144559206 wait complete. 2024-03-19T03:31:06.474 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-19T03:31:06.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.669+0000 7f232de00640 1 -- 172.21.15.117:0/3488780495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2328103940 msgr2=0x7f2328103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:06.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.669+0000 7f232de00640 1 --2- 172.21.15.117:0/3488780495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2328103940 0x7f2328103d60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2310009900 tx=0x7f231002bc30 comp rx=0 tx=0).stop 2024-03-19T03:31:06.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.669+0000 7f232de00640 1 -- 172.21.15.117:0/3488780495 shutdown_connections 2024-03-19T03:31:06.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.669+0000 7f232de00640 1 --2- 172.21.15.117:0/3488780495 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f23281054f0 0x7f232810bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:06.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.669+0000 7f232de00640 1 --2- 172.21.15.117:0/3488780495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2328104b30 0x7f2328104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:06.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.669+0000 7f232de00640 1 --2- 172.21.15.117:0/3488780495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2328103940 0x7f2328103d60 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:06.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.669+0000 7f232de00640 1 -- 172.21.15.117:0/3488780495 >> 172.21.15.117:0/3488780495 conn(0x7f23280feed0 msgr2=0x7f23281012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:06.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.669+0000 7f232de00640 1 -- 172.21.15.117:0/3488780495 shutdown_connections 2024-03-19T03:31:06.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.669+0000 7f232de00640 1 -- 172.21.15.117:0/3488780495 wait complete. 2024-03-19T03:31:06.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.670+0000 7f232de00640 1 Processor -- start 2024-03-19T03:31:06.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.671+0000 7f232de00640 1 -- start start 2024-03-19T03:31:06.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.671+0000 7f232de00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2328103940 0x7f2328199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:06.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.671+0000 7f232de00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2328104b30 0x7f232819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:06.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.671+0000 7f232de00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f23281054f0 0x7f23281a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:06.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.671+0000 7f232de00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f232810f590 con 0x7f2328103940 2024-03-19T03:31:06.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.671+0000 7f232de00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f232810f410 con 0x7f2328104b30 2024-03-19T03:31:06.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.671+0000 7f232de00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f232810f710 con 0x7f23281054f0 2024-03-19T03:31:06.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.671+0000 7f232ca00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2328103940 0x7f2328199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:06.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.671+0000 7f231fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2328104b30 0x7f232819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:06.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.671+0000 7f232ca00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2328103940 0x7f2328199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60806/0 (socket says 172.21.15.117:60806) 2024-03-19T03:31:06.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.671+0000 7f231fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2328104b30 0x7f232819a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:51818/0 (socket says 172.21.15.117:51818) 2024-03-19T03:31:06.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.671+0000 7f232ca00640 1 -- 172.21.15.117:0/4181966569 learned_addr learned my addr 172.21.15.117:0/4181966569 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:06.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.671+0000 7f232d400640 1 --2- 172.21.15.117:0/4181966569 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f23281054f0 0x7f23281a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:06.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.672+0000 7f231fe00640 1 -- 172.21.15.117:0/4181966569 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f23281054f0 msgr2=0x7f23281a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:06.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.672+0000 7f231fe00640 1 --2- 172.21.15.117:0/4181966569 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f23281054f0 0x7f23281a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:06.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.672+0000 7f231fe00640 1 -- 172.21.15.117:0/4181966569 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2328103940 msgr2=0x7f2328199c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:06.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.672+0000 7f231fe00640 1 --2- 172.21.15.117:0/4181966569 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2328103940 0x7f2328199c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:06.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.672+0000 7f231fe00640 1 -- 172.21.15.117:0/4181966569 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23281a1860 con 0x7f2328104b30 2024-03-19T03:31:06.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.672+0000 7f231fe00640 1 --2- 172.21.15.117:0/4181966569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2328104b30 0x7f232819a190 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f231800e9f0 tx=0x7f231800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:06.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.672+0000 7f2326000640 1 -- 172.21.15.117:0/4181966569 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231800cd80 con 0x7f2328104b30 2024-03-19T03:31:06.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.672+0000 7f2326000640 1 -- 172.21.15.117:0/4181966569 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23180035f0 con 0x7f2328104b30 2024-03-19T03:31:06.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.672+0000 7f232de00640 1 -- 172.21.15.117:0/4181966569 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23281a1b50 con 0x7f2328104b30 2024-03-19T03:31:06.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.672+0000 7f2326000640 1 -- 172.21.15.117:0/4181966569 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2318010690 con 0x7f2328104b30 2024-03-19T03:31:06.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.672+0000 7f232de00640 1 -- 172.21.15.117:0/4181966569 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23281a20f0 con 0x7f2328104b30 2024-03-19T03:31:06.675 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.673+0000 7f232de00640 1 -- 172.21.15.117:0/4181966569 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f232810ac60 con 0x7f2328104b30 2024-03-19T03:31:06.676 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.675+0000 7f2326000640 1 -- 172.21.15.117:0/4181966569 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f2318003b30 con 0x7f2328104b30 2024-03-19T03:31:06.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.676+0000 7f2326000640 1 --2- 172.21.15.117:0/4181966569 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f23080a2880 0x7f23080a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:06.677 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.677+0000 7f2326000640 1 -- 172.21.15.117:0/4181966569 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f23180cef70 con 0x7f2328104b30 2024-03-19T03:31:06.678 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.677+0000 7f232ca00640 1 --2- 172.21.15.117:0/4181966569 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f23080a2880 0x7f23080a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:06.678 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.677+0000 7f232ca00640 1 --2- 172.21.15.117:0/4181966569 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f23080a2880 0x7f23080a4d40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2310006010 tx=0x7f2310005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:06.681 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.680+0000 7f2326000640 1 -- 172.21.15.117:0/4181966569 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f231809bdd0 con 0x7f2328104b30 2024-03-19T03:31:06.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:06 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-19T03:31:06.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:06 smithi119 ceph-mon[66861]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T03:31:06.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:06 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/4144559206' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-19T03:31:06.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:06 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-19T03:31:06.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:06 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-19T03:31:06.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:06 smithi117 ceph-mon[31838]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T03:31:06.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:06 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/4144559206' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-19T03:31:06.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:06 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-19T03:31:06.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:06 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-19T03:31:06.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:06 smithi178 ceph-mon[67578]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T03:31:06.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:06 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/4144559206' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-19T03:31:06.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:06 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-19T03:31:06.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:06.923+0000 7f232de00640 1 -- 172.21.15.117:0/4181966569 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f2328062420 con 0x7f2328104b30 2024-03-19T03:31:07.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.456+0000 7f2326000640 1 -- 172.21.15.117:0/4181966569 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7f2318097030 con 0x7f2328104b30 2024-03-19T03:31:07.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.460+0000 7f232de00640 1 -- 172.21.15.117:0/4181966569 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f23080a2880 msgr2=0x7f23080a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:07.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.460+0000 7f232de00640 1 --2- 172.21.15.117:0/4181966569 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f23080a2880 0x7f23080a4d40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2310006010 tx=0x7f2310005b50 comp rx=0 tx=0).stop 2024-03-19T03:31:07.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.460+0000 7f232de00640 1 -- 172.21.15.117:0/4181966569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2328104b30 msgr2=0x7f232819a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:07.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.460+0000 7f232de00640 1 --2- 172.21.15.117:0/4181966569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2328104b30 0x7f232819a190 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f231800e9f0 tx=0x7f231800eeb0 comp rx=0 tx=0).stop 2024-03-19T03:31:07.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.461+0000 7f232de00640 1 -- 172.21.15.117:0/4181966569 shutdown_connections 2024-03-19T03:31:07.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.461+0000 7f232de00640 1 --2- 172.21.15.117:0/4181966569 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f23281054f0 0x7f23281a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:07.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.461+0000 7f232de00640 1 --2- 172.21.15.117:0/4181966569 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f23080a2880 0x7f23080a4d40 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:07.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.461+0000 7f232de00640 1 --2- 172.21.15.117:0/4181966569 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2328104b30 0x7f232819a190 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:07.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.461+0000 7f232de00640 1 --2- 172.21.15.117:0/4181966569 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2328103940 0x7f2328199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:07.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.461+0000 7f232de00640 1 -- 172.21.15.117:0/4181966569 >> 172.21.15.117:0/4181966569 conn(0x7f23280feed0 msgr2=0x7f2328107e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:07.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.461+0000 7f232de00640 1 -- 172.21.15.117:0/4181966569 shutdown_connections 2024-03-19T03:31:07.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.461+0000 7f232de00640 1 -- 172.21.15.117:0/4181966569 wait complete. 2024-03-19T03:31:07.495 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-19T03:31:07.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.685+0000 7f3fa5e00640 1 -- 172.21.15.117:0/3142956249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fa00757e0 msgr2=0x7f3fa0073c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:07.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.685+0000 7f3fa5e00640 1 --2- 172.21.15.117:0/3142956249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fa00757e0 0x7f3fa0073c30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3f94009900 tx=0x7f3f9402bcd0 comp rx=0 tx=0).stop 2024-03-19T03:31:07.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.685+0000 7f3fa5e00640 1 -- 172.21.15.117:0/3142956249 shutdown_connections 2024-03-19T03:31:07.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.685+0000 7f3fa5e00640 1 --2- 172.21.15.117:0/3142956249 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fa01095d0 0x7f3fa010ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:07.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.685+0000 7f3fa5e00640 1 --2- 172.21.15.117:0/3142956249 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fa0074200 0x7f3fa0074660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:07.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.685+0000 7f3fa5e00640 1 --2- 172.21.15.117:0/3142956249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fa00757e0 0x7f3fa0073c30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:07.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.685+0000 7f3fa5e00640 1 -- 172.21.15.117:0/3142956249 >> 172.21.15.117:0/3142956249 conn(0x7f3fa00fcc90 msgr2=0x7f3fa00ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:07.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.685+0000 7f3fa5e00640 1 -- 172.21.15.117:0/3142956249 shutdown_connections 2024-03-19T03:31:07.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.685+0000 7f3fa5e00640 1 -- 172.21.15.117:0/3142956249 wait complete. 2024-03-19T03:31:07.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.686+0000 7f3fa5e00640 1 Processor -- start 2024-03-19T03:31:07.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.687+0000 7f3fa5e00640 1 -- start start 2024-03-19T03:31:07.689 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.687+0000 7f3fa5e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fa0074200 0x7f3fa0199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:07.689 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.687+0000 7f3fa5e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fa00757e0 0x7f3fa019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:07.689 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.687+0000 7f3fa5e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fa01095d0 0x7f3fa01a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:07.689 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.687+0000 7f3fa5e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa010f580 con 0x7f3fa0074200 2024-03-19T03:31:07.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.687+0000 7f3fa5e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa010f400 con 0x7f3fa01095d0 2024-03-19T03:31:07.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.687+0000 7f3fa5e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa010f700 con 0x7f3fa00757e0 2024-03-19T03:31:07.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.687+0000 7f3fa4a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fa0074200 0x7f3fa0199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:07.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.687+0000 7f3fa5400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fa01095d0 0x7f3fa01a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:07.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.687+0000 7f3fa4a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fa0074200 0x7f3fa0199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60826/0 (socket says 172.21.15.117:60826) 2024-03-19T03:31:07.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.687+0000 7f3fa4a00640 1 -- 172.21.15.117:0/1844429760 learned_addr learned my addr 172.21.15.117:0/1844429760 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:07.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.687+0000 7f3f9fe00640 1 --2- 172.21.15.117:0/1844429760 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fa00757e0 0x7f3fa019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:07.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.688+0000 7f3fa5400640 1 -- 172.21.15.117:0/1844429760 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fa00757e0 msgr2=0x7f3fa019a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:07.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.688+0000 7f3fa5400640 1 --2- 172.21.15.117:0/1844429760 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fa00757e0 0x7f3fa019a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:07.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.688+0000 7f3fa5400640 1 -- 172.21.15.117:0/1844429760 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fa0074200 msgr2=0x7f3fa0199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:07.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.688+0000 7f3fa5400640 1 --2- 172.21.15.117:0/1844429760 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fa0074200 0x7f3fa0199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:07.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.688+0000 7f3fa5400640 1 -- 172.21.15.117:0/1844429760 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fa01a1850 con 0x7f3fa01095d0 2024-03-19T03:31:07.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.688+0000 7f3fa5400640 1 --2- 172.21.15.117:0/1844429760 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fa01095d0 0x7f3fa01a11b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3f98009fd0 tx=0x7f3f9800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:07.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.688+0000 7f3f9d600640 1 -- 172.21.15.117:0/1844429760 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9800ce20 con 0x7f3fa01095d0 2024-03-19T03:31:07.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.688+0000 7f3f9d600640 1 -- 172.21.15.117:0/1844429760 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f98005020 con 0x7f3fa01095d0 2024-03-19T03:31:07.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.688+0000 7f3fa5e00640 1 -- 172.21.15.117:0/1844429760 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fa01a1b40 con 0x7f3fa01095d0 2024-03-19T03:31:07.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.688+0000 7f3f9d600640 1 -- 172.21.15.117:0/1844429760 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f98010720 con 0x7f3fa01095d0 2024-03-19T03:31:07.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.689+0000 7f3fa5e00640 1 -- 172.21.15.117:0/1844429760 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fa01a2090 con 0x7f3fa01095d0 2024-03-19T03:31:07.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.690+0000 7f3fa5e00640 1 -- 172.21.15.117:0/1844429760 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fa0108390 con 0x7f3fa01095d0 2024-03-19T03:31:07.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.691+0000 7f3f9d600640 1 -- 172.21.15.117:0/1844429760 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f3f98005540 con 0x7f3fa01095d0 2024-03-19T03:31:07.693 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:07 smithi117 ceph-mon[31838]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-19T03:31:07.693 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:07 smithi117 ceph-mon[31838]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-19T03:31:07.693 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:07 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-19T03:31:07.693 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:07 smithi117 ceph-mon[31838]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-19T03:31:07.693 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:07 smithi117 ceph-mon[31838]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T03:31:07.693 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:07 smithi117 ceph-mon[31838]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-03-19T03:31:07.693 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:07 smithi117 ceph-mon[31838]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-19T03:31:07.693 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:07 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/4181966569' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-19T03:31:07.693 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:07 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-19T03:31:07.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.693+0000 7f3f9d600640 1 --2- 172.21.15.117:0/1844429760 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f3f880a2880 0x7f3f880a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:07.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.694+0000 7f3f9d600640 1 -- 172.21.15.117:0/1844429760 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3f980cf2e0 con 0x7f3fa01095d0 2024-03-19T03:31:07.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.694+0000 7f3fa4a00640 1 --2- 172.21.15.117:0/1844429760 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f3f880a2880 0x7f3f880a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:07.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.695+0000 7f3fa4a00640 1 --2- 172.21.15.117:0/1844429760 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f3f880a2880 0x7f3f880a4d40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3f9402c850 tx=0x7f3f9402cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:07.701 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.701+0000 7f3f9d600640 1 -- 172.21.15.117:0/1844429760 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3f9809c140 con 0x7f3fa01095d0 2024-03-19T03:31:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:07 smithi119 ceph-mon[66861]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-19T03:31:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:07 smithi119 ceph-mon[66861]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-19T03:31:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:07 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-19T03:31:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:07 smithi119 ceph-mon[66861]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-19T03:31:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:07 smithi119 ceph-mon[66861]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T03:31:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:07 smithi119 ceph-mon[66861]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-03-19T03:31:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:07 smithi119 ceph-mon[66861]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-19T03:31:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:07 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/4181966569' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-19T03:31:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:07 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-19T03:31:07.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:07 smithi178 ceph-mon[67578]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-19T03:31:07.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:07 smithi178 ceph-mon[67578]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-19T03:31:07.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:07 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-19T03:31:07.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:07 smithi178 ceph-mon[67578]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-19T03:31:07.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:07 smithi178 ceph-mon[67578]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T03:31:07.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:07 smithi178 ceph-mon[67578]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-03-19T03:31:07.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:07 smithi178 ceph-mon[67578]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-19T03:31:07.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:07 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/4181966569' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-19T03:31:07.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:07 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-19T03:31:07.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:07.934+0000 7f3fa5e00640 1 -- 172.21.15.117:0/1844429760 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f3fa0062420 con 0x7f3fa01095d0 2024-03-19T03:31:08.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.471+0000 7f3f9d600640 1 -- 172.21.15.117:0/1844429760 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7f3f9809ba40 con 0x7f3fa01095d0 2024-03-19T03:31:08.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.475+0000 7f3fa5e00640 1 -- 172.21.15.117:0/1844429760 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f3f880a2880 msgr2=0x7f3f880a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:08.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.475+0000 7f3fa5e00640 1 --2- 172.21.15.117:0/1844429760 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f3f880a2880 0x7f3f880a4d40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3f9402c850 tx=0x7f3f9402cf90 comp rx=0 tx=0).stop 2024-03-19T03:31:08.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.475+0000 7f3fa5e00640 1 -- 172.21.15.117:0/1844429760 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fa01095d0 msgr2=0x7f3fa01a11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:08.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.475+0000 7f3fa5e00640 1 --2- 172.21.15.117:0/1844429760 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fa01095d0 0x7f3fa01a11b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3f98009fd0 tx=0x7f3f9800ef10 comp rx=0 tx=0).stop 2024-03-19T03:31:08.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.475+0000 7f3fa5e00640 1 -- 172.21.15.117:0/1844429760 shutdown_connections 2024-03-19T03:31:08.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.475+0000 7f3fa5e00640 1 --2- 172.21.15.117:0/1844429760 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f3f880a2880 0x7f3f880a4d40 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:08.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.475+0000 7f3fa5e00640 1 --2- 172.21.15.117:0/1844429760 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fa01095d0 0x7f3fa01a11b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:08.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.475+0000 7f3fa5e00640 1 --2- 172.21.15.117:0/1844429760 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fa00757e0 0x7f3fa019a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:08.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.475+0000 7f3fa5e00640 1 --2- 172.21.15.117:0/1844429760 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fa0074200 0x7f3fa0199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:08.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.475+0000 7f3fa5e00640 1 -- 172.21.15.117:0/1844429760 >> 172.21.15.117:0/1844429760 conn(0x7f3fa00fcc90 msgr2=0x7f3fa0105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:08.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.475+0000 7f3fa5e00640 1 -- 172.21.15.117:0/1844429760 shutdown_connections 2024-03-19T03:31:08.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.475+0000 7f3fa5e00640 1 -- 172.21.15.117:0/1844429760 wait complete. 2024-03-19T03:31:08.507 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-03-19T03:31:08.507 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-19T03:31:08.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.698+0000 7f9a63e00640 1 -- 172.21.15.117:0/1506653142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a64102910 msgr2=0x7f9a64102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:08.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.698+0000 7f9a63e00640 1 --2- 172.21.15.117:0/1506653142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a64102910 0x7f9a64102d90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9a50009900 tx=0x7f9a5002bc30 comp rx=0 tx=0).stop 2024-03-19T03:31:08.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.698+0000 7f9a63e00640 1 -- 172.21.15.117:0/1506653142 shutdown_connections 2024-03-19T03:31:08.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.698+0000 7f9a63e00640 1 --2- 172.21.15.117:0/1506653142 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a641032d0 0x7f9a64109b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:08.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.698+0000 7f9a63e00640 1 --2- 172.21.15.117:0/1506653142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a64102910 0x7f9a64102d90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:08.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.698+0000 7f9a63e00640 1 --2- 172.21.15.117:0/1506653142 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9a64101720 0x7f9a64101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:08.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.698+0000 7f9a63e00640 1 -- 172.21.15.117:0/1506653142 >> 172.21.15.117:0/1506653142 conn(0x7f9a640fcc90 msgr2=0x7f9a640ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:08.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.699+0000 7f9a63e00640 1 -- 172.21.15.117:0/1506653142 shutdown_connections 2024-03-19T03:31:08.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.699+0000 7f9a63e00640 1 -- 172.21.15.117:0/1506653142 wait complete. 2024-03-19T03:31:08.701 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.700+0000 7f9a63e00640 1 Processor -- start 2024-03-19T03:31:08.701 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.700+0000 7f9a63e00640 1 -- start start 2024-03-19T03:31:08.702 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.701+0000 7f9a63e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a64101720 0x7f9a64197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:08.702 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.701+0000 7f9a63e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9a64102910 0x7f9a64197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:08.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.701+0000 7f9a63e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a641032d0 0x7f9a6419efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:08.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.701+0000 7f9a63e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6410d3f0 con 0x7f9a64101720 2024-03-19T03:31:08.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.701+0000 7f9a63e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6410d270 con 0x7f9a641032d0 2024-03-19T03:31:08.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.701+0000 7f9a63e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6410d570 con 0x7f9a64102910 2024-03-19T03:31:08.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.701+0000 7f9a62a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a64101720 0x7f9a64197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:08.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.701+0000 7f9a62a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a64101720 0x7f9a64197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60846/0 (socket says 172.21.15.117:60846) 2024-03-19T03:31:08.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.701+0000 7f9a63400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a641032d0 0x7f9a6419efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:08.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.701+0000 7f9a62a00640 1 -- 172.21.15.117:0/1138710052 learned_addr learned my addr 172.21.15.117:0/1138710052 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:08.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.701+0000 7f9a62000640 1 --2- 172.21.15.117:0/1138710052 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9a64102910 0x7f9a64197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:08.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.702+0000 7f9a62a00640 1 -- 172.21.15.117:0/1138710052 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9a64102910 msgr2=0x7f9a64197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:08.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.702+0000 7f9a62a00640 1 --2- 172.21.15.117:0/1138710052 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9a64102910 0x7f9a64197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:08.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.702+0000 7f9a62a00640 1 -- 172.21.15.117:0/1138710052 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a641032d0 msgr2=0x7f9a6419efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:08.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.702+0000 7f9a62a00640 1 --2- 172.21.15.117:0/1138710052 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a641032d0 0x7f9a6419efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:08.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.702+0000 7f9a62a00640 1 -- 172.21.15.117:0/1138710052 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a6419f650 con 0x7f9a64101720 2024-03-19T03:31:08.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.702+0000 7f9a63400640 1 --2- 172.21.15.117:0/1138710052 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a641032d0 0x7f9a6419efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:31:08.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.702+0000 7f9a62a00640 1 --2- 172.21.15.117:0/1138710052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a64101720 0x7f9a64197a30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9a5400e9c0 tx=0x7f9a5400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:08.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.702+0000 7f9a5b400640 1 -- 172.21.15.117:0/1138710052 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5400cd80 con 0x7f9a64101720 2024-03-19T03:31:08.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.702+0000 7f9a5b400640 1 -- 172.21.15.117:0/1138710052 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a540035f0 con 0x7f9a64101720 2024-03-19T03:31:08.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.703+0000 7f9a63e00640 1 -- 172.21.15.117:0/1138710052 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a6419f940 con 0x7f9a64101720 2024-03-19T03:31:08.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.703+0000 7f9a5b400640 1 -- 172.21.15.117:0/1138710052 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a54010690 con 0x7f9a64101720 2024-03-19T03:31:08.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.703+0000 7f9a63e00640 1 -- 172.21.15.117:0/1138710052 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a6419fe60 con 0x7f9a64101720 2024-03-19T03:31:08.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.704+0000 7f9a63e00640 1 -- 172.21.15.117:0/1138710052 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a64108a60 con 0x7f9a64101720 2024-03-19T03:31:08.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.706+0000 7f9a5b400640 1 -- 172.21.15.117:0/1138710052 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f9a540037a0 con 0x7f9a64101720 2024-03-19T03:31:08.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.708+0000 7f9a5b400640 1 --2- 172.21.15.117:0/1138710052 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f9a480a2880 0x7f9a480a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:08.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.708+0000 7f9a5b400640 1 -- 172.21.15.117:0/1138710052 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9a540ceff0 con 0x7f9a64101720 2024-03-19T03:31:08.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.708+0000 7f9a62000640 1 --2- 172.21.15.117:0/1138710052 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f9a480a2880 0x7f9a480a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:08.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.709+0000 7f9a62000640 1 --2- 172.21.15.117:0/1138710052 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f9a480a2880 0x7f9a480a4d40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9a5002c240 tx=0x7f9a5002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:08.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.713+0000 7f9a5b400640 1 -- 172.21.15.117:0/1138710052 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9a5409be50 con 0x7f9a64101720 2024-03-19T03:31:08.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:08 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-19T03:31:08.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:08 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] up:active 2024-03-19T03:31:08.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:08 smithi119 ceph-mon[66861]: daemon mds.g assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-19T03:31:08.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:08 smithi119 ceph-mon[66861]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-03-19T03:31:08.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:08 smithi119 ceph-mon[66861]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:creating} 9 up:standby 2024-03-19T03:31:08.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:08 smithi119 ceph-mon[66861]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-03-19T03:31:08.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:08 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1844429760' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-19T03:31:08.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:08 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-19T03:31:08.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:08 smithi119 ceph-mon[66861]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s wr, 0 op/s 2024-03-19T03:31:08.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:08 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-19T03:31:08.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:08 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] up:active 2024-03-19T03:31:08.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:08 smithi117 ceph-mon[31838]: daemon mds.g assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-19T03:31:08.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:08 smithi117 ceph-mon[31838]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-03-19T03:31:08.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:08 smithi117 ceph-mon[31838]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:creating} 9 up:standby 2024-03-19T03:31:08.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:08 smithi117 ceph-mon[31838]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-03-19T03:31:08.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:08 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1844429760' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-19T03:31:08.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:08 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-19T03:31:08.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:08 smithi117 ceph-mon[31838]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s wr, 0 op/s 2024-03-19T03:31:08.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:08 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-19T03:31:08.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:08 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] up:active 2024-03-19T03:31:08.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:08 smithi178 ceph-mon[67578]: daemon mds.g assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-19T03:31:08.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:08 smithi178 ceph-mon[67578]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-03-19T03:31:08.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:08 smithi178 ceph-mon[67578]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:creating} 9 up:standby 2024-03-19T03:31:08.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:08 smithi178 ceph-mon[67578]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-03-19T03:31:08.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:08 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1844429760' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-19T03:31:08.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:08 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-19T03:31:08.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:08 smithi178 ceph-mon[67578]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s wr, 0 op/s 2024-03-19T03:31:08.953 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.952+0000 7f9a63e00640 1 -- 172.21.15.117:0/1138710052 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f9a64061c60 con 0x7f9a480a2880 2024-03-19T03:31:08.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.989+0000 7f9a5b400640 1 -- 172.21.15.117:0/1138710052 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9a64061c60 con 0x7f9a480a2880 2024-03-19T03:31:08.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.993+0000 7f9a63e00640 1 -- 172.21.15.117:0/1138710052 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f9a480a2880 msgr2=0x7f9a480a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:08.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.993+0000 7f9a63e00640 1 --2- 172.21.15.117:0/1138710052 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f9a480a2880 0x7f9a480a4d40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9a5002c240 tx=0x7f9a5002df90 comp rx=0 tx=0).stop 2024-03-19T03:31:08.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.993+0000 7f9a63e00640 1 -- 172.21.15.117:0/1138710052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a64101720 msgr2=0x7f9a64197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:08.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.993+0000 7f9a63e00640 1 --2- 172.21.15.117:0/1138710052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a64101720 0x7f9a64197a30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9a5400e9c0 tx=0x7f9a5400ee80 comp rx=0 tx=0).stop 2024-03-19T03:31:08.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.993+0000 7f9a63e00640 1 -- 172.21.15.117:0/1138710052 shutdown_connections 2024-03-19T03:31:08.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.993+0000 7f9a63e00640 1 --2- 172.21.15.117:0/1138710052 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f9a480a2880 0x7f9a480a4d40 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:08.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.993+0000 7f9a63e00640 1 --2- 172.21.15.117:0/1138710052 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a641032d0 0x7f9a6419efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:08.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.993+0000 7f9a63e00640 1 --2- 172.21.15.117:0/1138710052 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9a64102910 0x7f9a64197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:08.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.993+0000 7f9a63e00640 1 --2- 172.21.15.117:0/1138710052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a64101720 0x7f9a64197a30 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:08.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.993+0000 7f9a63e00640 1 -- 172.21.15.117:0/1138710052 >> 172.21.15.117:0/1138710052 conn(0x7f9a640fcc90 msgr2=0x7f9a64105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:08.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.993+0000 7f9a63e00640 1 -- 172.21.15.117:0/1138710052 shutdown_connections 2024-03-19T03:31:08.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:08.993+0000 7f9a63e00640 1 -- 172.21.15.117:0/1138710052 wait complete. 2024-03-19T03:31:09.016 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-19T03:31:09.185 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:31:08 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:31:08.961+0000 7f2510400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T03:31:09.185 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:31:08 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:31:08.961+0000 7f2510400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T03:31:09.185 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:31:08 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:31:08.961+0000 7f2510400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T03:31:09.186 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:31:08 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:31:08.961+0000 7f2510400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T03:31:09.186 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:31:08 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:31:08.961+0000 7f2510400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T03:31:09.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.206+0000 7f0c0be00640 1 -- 172.21.15.117:0/1556839436 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c0c0757e0 msgr2=0x7f0c0c073c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:09.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.206+0000 7f0c0be00640 1 --2- 172.21.15.117:0/1556839436 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c0c0757e0 0x7f0c0c073c30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc009900 tx=0x7f0bfc02bcd0 comp rx=0 tx=0).stop 2024-03-19T03:31:09.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.206+0000 7f0c0be00640 1 -- 172.21.15.117:0/1556839436 shutdown_connections 2024-03-19T03:31:09.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.206+0000 7f0c0be00640 1 --2- 172.21.15.117:0/1556839436 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c0c1095d0 0x7f0c0c10ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:09.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.206+0000 7f0c0be00640 1 --2- 172.21.15.117:0/1556839436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c0c074200 0x7f0c0c074660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:09.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.206+0000 7f0c0be00640 1 --2- 172.21.15.117:0/1556839436 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c0c0757e0 0x7f0c0c073c30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:09.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.206+0000 7f0c0be00640 1 -- 172.21.15.117:0/1556839436 >> 172.21.15.117:0/1556839436 conn(0x7f0c0c0fcc90 msgr2=0x7f0c0c0ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:09.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.206+0000 7f0c0be00640 1 -- 172.21.15.117:0/1556839436 shutdown_connections 2024-03-19T03:31:09.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.206+0000 7f0c0be00640 1 -- 172.21.15.117:0/1556839436 wait complete. 2024-03-19T03:31:09.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.207+0000 7f0c0be00640 1 Processor -- start 2024-03-19T03:31:09.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.207+0000 7f0c0be00640 1 -- start start 2024-03-19T03:31:09.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.208+0000 7f0c0be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c0c074200 0x7f0c0c199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:09.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.208+0000 7f0c0be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c0c0757e0 0x7f0c0c19a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:09.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.208+0000 7f0c0be00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c0c1095d0 0x7f0c0c1a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:09.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.208+0000 7f0c0be00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0c10f5f0 con 0x7f0c0c0757e0 2024-03-19T03:31:09.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.208+0000 7f0c0be00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0c10f470 con 0x7f0c0c074200 2024-03-19T03:31:09.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.208+0000 7f0c0be00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0c10f770 con 0x7f0c0c1095d0 2024-03-19T03:31:09.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.208+0000 7f0c0aa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c0c074200 0x7f0c0c199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:09.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.208+0000 7f0c0a000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c0c0757e0 0x7f0c0c19a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:09.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.208+0000 7f0c0aa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c0c074200 0x7f0c0c199ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:51880/0 (socket says 172.21.15.117:51880) 2024-03-19T03:31:09.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.208+0000 7f0c0aa00640 1 -- 172.21.15.117:0/2386030174 learned_addr learned my addr 172.21.15.117:0/2386030174 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:09.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.208+0000 7f0c0b400640 1 --2- 172.21.15.117:0/2386030174 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c0c1095d0 0x7f0c0c1a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:09.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.209+0000 7f0c0aa00640 1 -- 172.21.15.117:0/2386030174 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c0c1095d0 msgr2=0x7f0c0c1a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:09.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.209+0000 7f0c0aa00640 1 --2- 172.21.15.117:0/2386030174 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c0c1095d0 0x7f0c0c1a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:09.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.209+0000 7f0c0aa00640 1 -- 172.21.15.117:0/2386030174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c0c0757e0 msgr2=0x7f0c0c19a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:09.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.209+0000 7f0c0aa00640 1 --2- 172.21.15.117:0/2386030174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c0c0757e0 0x7f0c0c19a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:09.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.209+0000 7f0c0aa00640 1 -- 172.21.15.117:0/2386030174 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c0c1a1860 con 0x7f0c0c074200 2024-03-19T03:31:09.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.209+0000 7f0c0b400640 1 --2- 172.21.15.117:0/2386030174 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c0c1095d0 0x7f0c0c1a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:31:09.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.209+0000 7f0c0a000640 1 --2- 172.21.15.117:0/2386030174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c0c0757e0 0x7f0c0c19a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:31:09.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.209+0000 7f0c0aa00640 1 --2- 172.21.15.117:0/2386030174 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c0c074200 0x7f0c0c199ca0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc000c00 tx=0x7f0bfc02c6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:09.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.209+0000 7f0c10a00640 1 -- 172.21.15.117:0/2386030174 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc043020 con 0x7f0c0c074200 2024-03-19T03:31:09.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.209+0000 7f0c10a00640 1 -- 172.21.15.117:0/2386030174 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bfc0054e0 con 0x7f0c0c074200 2024-03-19T03:31:09.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.209+0000 7f0c10a00640 1 -- 172.21.15.117:0/2386030174 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc003c20 con 0x7f0c0c074200 2024-03-19T03:31:09.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.209+0000 7f0c0be00640 1 -- 172.21.15.117:0/2386030174 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c0c1a1af0 con 0x7f0c0c074200 2024-03-19T03:31:09.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.210+0000 7f0c0be00640 1 -- 172.21.15.117:0/2386030174 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c0c1a1ff0 con 0x7f0c0c074200 2024-03-19T03:31:09.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.211+0000 7f0c0be00640 1 -- 172.21.15.117:0/2386030174 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c0c068530 con 0x7f0c0c074200 2024-03-19T03:31:09.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.213+0000 7f0c10a00640 1 -- 172.21.15.117:0/2386030174 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f0bfc046020 con 0x7f0c0c074200 2024-03-19T03:31:09.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.216+0000 7f0c10a00640 1 --2- 172.21.15.117:0/2386030174 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0bf00a2880 0x7f0bf00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:09.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.216+0000 7f0c10a00640 1 -- 172.21.15.117:0/2386030174 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0bfc0f0e10 con 0x7f0c0c074200 2024-03-19T03:31:09.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.216+0000 7f0c0a000640 1 --2- 172.21.15.117:0/2386030174 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0bf00a2880 0x7f0bf00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:09.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.217+0000 7f0c0a000640 1 --2- 172.21.15.117:0/2386030174 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0bf00a2880 0x7f0bf00a4d40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0bf80095f0 tx=0x7f0bf8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:09.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.219+0000 7f0c10a00640 1 -- 172.21.15.117:0/2386030174 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0bfc0f4030 con 0x7f0c0c074200 2024-03-19T03:31:09.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.454+0000 7f0c0be00640 1 -- 172.21.15.117:0/2386030174 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f0c0c061cf0 con 0x7f0bf00a2880 2024-03-19T03:31:09.472 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:09 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-19T03:31:09.472 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:09 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] up:active 2024-03-19T03:31:09.472 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:09 smithi117 ceph-mon[31838]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-19T03:31:09.473 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:09 smithi117 ceph-mon[31838]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 9 up:standby 2024-03-19T03:31:09.473 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:09 smithi117 ceph-mon[31838]: fsmap cephfs:4 {0=b=up:active,1=a=up:active,2=g=up:active,3=h=up:creating} 8 up:standby 2024-03-19T03:31:09.473 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:09 smithi117 ceph-mon[31838]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-19T03:31:09.473 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:09 smithi117 ceph-mon[31838]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:31:09.473 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:09 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] up:active 2024-03-19T03:31:09.473 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:09 smithi117 ceph-mon[31838]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-19T03:31:09.473 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:09 smithi117 ceph-mon[31838]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-19T03:31:09.473 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:09 smithi117 ceph-mon[31838]: Cluster is now healthy 2024-03-19T03:31:09.473 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:09 smithi117 ceph-mon[31838]: fsmap cephfs:4 {0=b=up:active,1=a=up:active,2=g=up:active,3=h=up:active} 4 up:standby-replay 4 up:standby 2024-03-19T03:31:09.473 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:09 smithi117 ceph-mon[31838]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-03-19T03:31:09.473 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:09 smithi117 ceph-mon[31838]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-03-19T03:31:09.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.481+0000 7f0c10a00640 1 -- 172.21.15.117:0/2386030174 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0c0c061cf0 con 0x7f0bf00a2880 2024-03-19T03:31:09.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.484+0000 7f0c0be00640 1 -- 172.21.15.117:0/2386030174 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0bf00a2880 msgr2=0x7f0bf00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:09.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.484+0000 7f0c0be00640 1 --2- 172.21.15.117:0/2386030174 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0bf00a2880 0x7f0bf00a4d40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0bf80095f0 tx=0x7f0bf8005f90 comp rx=0 tx=0).stop 2024-03-19T03:31:09.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.484+0000 7f0c0be00640 1 -- 172.21.15.117:0/2386030174 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c0c074200 msgr2=0x7f0c0c199ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:09.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.484+0000 7f0c0be00640 1 --2- 172.21.15.117:0/2386030174 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c0c074200 0x7f0c0c199ca0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc000c00 tx=0x7f0bfc02c6f0 comp rx=0 tx=0).stop 2024-03-19T03:31:09.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.485+0000 7f0c0be00640 1 -- 172.21.15.117:0/2386030174 shutdown_connections 2024-03-19T03:31:09.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.485+0000 7f0c0be00640 1 --2- 172.21.15.117:0/2386030174 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c0c1095d0 0x7f0c0c1a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:09.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.485+0000 7f0c0be00640 1 --2- 172.21.15.117:0/2386030174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c0c0757e0 0x7f0c0c19a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:09.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.485+0000 7f0c0be00640 1 --2- 172.21.15.117:0/2386030174 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0bf00a2880 0x7f0bf00a4d40 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:09.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.485+0000 7f0c0be00640 1 --2- 172.21.15.117:0/2386030174 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0c0c074200 0x7f0c0c199ca0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:09.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.485+0000 7f0c0be00640 1 -- 172.21.15.117:0/2386030174 >> 172.21.15.117:0/2386030174 conn(0x7f0c0c0fcc90 msgr2=0x7f0c0c105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:09.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.485+0000 7f0c0be00640 1 -- 172.21.15.117:0/2386030174 shutdown_connections 2024-03-19T03:31:09.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.485+0000 7f0c0be00640 1 -- 172.21.15.117:0/2386030174 wait complete. 2024-03-19T03:31:09.504 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-19T03:31:09.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.720+0000 7f01a3a00640 1 -- 172.21.15.117:0/1113405379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f019c101830 msgr2=0x7f019c101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:09.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.720+0000 7f01a3a00640 1 --2- 172.21.15.117:0/1113405379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f019c101830 0x7f019c101c50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0190009900 tx=0x7f019002bc30 comp rx=0 tx=0).stop 2024-03-19T03:31:09.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.720+0000 7f01a3a00640 1 -- 172.21.15.117:0/1113405379 shutdown_connections 2024-03-19T03:31:09.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.720+0000 7f01a3a00640 1 --2- 172.21.15.117:0/1113405379 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f019c1033e0 0x7f019c109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:09.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.720+0000 7f01a3a00640 1 --2- 172.21.15.117:0/1113405379 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f019c102a20 0x7f019c102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:09.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.720+0000 7f01a3a00640 1 --2- 172.21.15.117:0/1113405379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f019c101830 0x7f019c101c50 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:09.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.720+0000 7f01a3a00640 1 -- 172.21.15.117:0/1113405379 >> 172.21.15.117:0/1113405379 conn(0x7f019c0fcdc0 msgr2=0x7f019c0ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:09.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.720+0000 7f01a3a00640 1 -- 172.21.15.117:0/1113405379 shutdown_connections 2024-03-19T03:31:09.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.720+0000 7f01a3a00640 1 -- 172.21.15.117:0/1113405379 wait complete. 2024-03-19T03:31:09.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.721+0000 7f01a3a00640 1 Processor -- start 2024-03-19T03:31:09.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.721+0000 7f01a3a00640 1 -- start start 2024-03-19T03:31:09.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.722+0000 7f01a3a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f019c101830 0x7f019c0755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.722+0000 7f01a3a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f019c102a20 0x7f019c073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.722+0000 7f01a3a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f019c1033e0 0x7f019c074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.722+0000 7f01a3a00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f019c10d500 con 0x7f019c1033e0 2024-03-19T03:31:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.722+0000 7f01a3a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f019c10d380 con 0x7f019c101830 2024-03-19T03:31:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.722+0000 7f01a3a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f019c10d680 con 0x7f019c102a20 2024-03-19T03:31:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.722+0000 7f01a1000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f019c101830 0x7f019c0755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.722+0000 7f01a1000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f019c101830 0x7f019c0755d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:51890/0 (socket says 172.21.15.117:51890) 2024-03-19T03:31:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.722+0000 7f01a1000640 1 -- 172.21.15.117:0/3434569708 learned_addr learned my addr 172.21.15.117:0/3434569708 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.722+0000 7f01a1a00640 1 --2- 172.21.15.117:0/3434569708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f019c1033e0 0x7f019c074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.722+0000 7f019be00640 1 --2- 172.21.15.117:0/3434569708 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f019c102a20 0x7f019c073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.722+0000 7f01a1000640 1 -- 172.21.15.117:0/3434569708 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f019c102a20 msgr2=0x7f019c073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.722+0000 7f01a1000640 1 --2- 172.21.15.117:0/3434569708 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f019c102a20 0x7f019c073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.722+0000 7f01a1000640 1 -- 172.21.15.117:0/3434569708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f019c1033e0 msgr2=0x7f019c074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.722+0000 7f01a1000640 1 --2- 172.21.15.117:0/3434569708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f019c1033e0 0x7f019c074150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:09.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.722+0000 7f01a1000640 1 -- 172.21.15.117:0/3434569708 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f019c0749e0 con 0x7f019c101830 2024-03-19T03:31:09.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.722+0000 7f01a1a00640 1 --2- 172.21.15.117:0/3434569708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f019c1033e0 0x7f019c074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:31:09.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.722+0000 7f019be00640 1 --2- 172.21.15.117:0/3434569708 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f019c102a20 0x7f019c073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:31:09.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.723+0000 7f01a1000640 1 --2- 172.21.15.117:0/3434569708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f019c101830 0x7f019c0755d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0190000c00 tx=0x7f019002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:09.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.723+0000 7f0199600640 1 -- 172.21.15.117:0/3434569708 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0190043020 con 0x7f019c101830 2024-03-19T03:31:09.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.723+0000 7f0199600640 1 -- 172.21.15.117:0/3434569708 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01900054e0 con 0x7f019c101830 2024-03-19T03:31:09.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.723+0000 7f0199600640 1 -- 172.21.15.117:0/3434569708 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0190003c40 con 0x7f019c101830 2024-03-19T03:31:09.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.723+0000 7f01a3a00640 1 -- 172.21.15.117:0/3434569708 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f019c1a1c90 con 0x7f019c101830 2024-03-19T03:31:09.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.723+0000 7f01a3a00640 1 -- 172.21.15.117:0/3434569708 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f019c1a2180 con 0x7f019c101830 2024-03-19T03:31:09.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.724+0000 7f01a3a00640 1 -- 172.21.15.117:0/3434569708 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f019c108ba0 con 0x7f019c101830 2024-03-19T03:31:09.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.726+0000 7f0199600640 1 -- 172.21.15.117:0/3434569708 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f0190046020 con 0x7f019c101830 2024-03-19T03:31:09.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.727+0000 7f0199600640 1 --2- 172.21.15.117:0/3434569708 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f01840a2880 0x7f01840a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:09.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.728+0000 7f0199600640 1 -- 172.21.15.117:0/3434569708 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f01900f12d0 con 0x7f019c101830 2024-03-19T03:31:09.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.728+0000 7f019be00640 1 --2- 172.21.15.117:0/3434569708 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f01840a2880 0x7f01840a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:09.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.729+0000 7f019be00640 1 --2- 172.21.15.117:0/3434569708 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f01840a2880 0x7f01840a4d40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f019c075340 tx=0x7f0188005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:09.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:09.730+0000 7f0199600640 1 -- 172.21.15.117:0/3434569708 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f019004d020 con 0x7f019c101830 2024-03-19T03:31:09.734 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 19 03:31:09 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-b[97021]: 2024-03-19T03:31:09.436+0000 7f2c847ae700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-19T03:31:09.734 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 19 03:31:09 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-b[97021]: 2024-03-19T03:31:09.436+0000 7f2c847ae700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-19T03:31:09.734 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 19 03:31:09 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-b[97021]: 2024-03-19T03:31:09.436+0000 7f2c847ae700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-19T03:31:09.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:09 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-19T03:31:09.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:09 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] up:active 2024-03-19T03:31:09.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:09 smithi119 ceph-mon[66861]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-19T03:31:09.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:09 smithi119 ceph-mon[66861]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 9 up:standby 2024-03-19T03:31:09.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:09 smithi119 ceph-mon[66861]: fsmap cephfs:4 {0=b=up:active,1=a=up:active,2=g=up:active,3=h=up:creating} 8 up:standby 2024-03-19T03:31:09.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:09 smithi119 ceph-mon[66861]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-19T03:31:09.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:09 smithi119 ceph-mon[66861]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:31:09.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:09 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] up:active 2024-03-19T03:31:09.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:09 smithi119 ceph-mon[66861]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-19T03:31:09.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:09 smithi119 ceph-mon[66861]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-19T03:31:09.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:09 smithi119 ceph-mon[66861]: Cluster is now healthy 2024-03-19T03:31:09.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:09 smithi119 ceph-mon[66861]: fsmap cephfs:4 {0=b=up:active,1=a=up:active,2=g=up:active,3=h=up:active} 4 up:standby-replay 4 up:standby 2024-03-19T03:31:09.736 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:09 smithi119 ceph-mon[66861]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-03-19T03:31:09.736 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:09 smithi119 ceph-mon[66861]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-03-19T03:31:09.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:09 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-19T03:31:09.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:09 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] up:active 2024-03-19T03:31:09.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:09 smithi178 ceph-mon[67578]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-19T03:31:09.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:09 smithi178 ceph-mon[67578]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 9 up:standby 2024-03-19T03:31:09.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:09 smithi178 ceph-mon[67578]: fsmap cephfs:4 {0=b=up:active,1=a=up:active,2=g=up:active,3=h=up:creating} 8 up:standby 2024-03-19T03:31:09.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:09 smithi178 ceph-mon[67578]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-19T03:31:09.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:09 smithi178 ceph-mon[67578]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:31:09.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:09 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] up:active 2024-03-19T03:31:09.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:09 smithi178 ceph-mon[67578]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-19T03:31:09.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:09 smithi178 ceph-mon[67578]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-19T03:31:09.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:09 smithi178 ceph-mon[67578]: Cluster is now healthy 2024-03-19T03:31:09.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:09 smithi178 ceph-mon[67578]: fsmap cephfs:4 {0=b=up:active,1=a=up:active,2=g=up:active,3=h=up:active} 4 up:standby-replay 4 up:standby 2024-03-19T03:31:09.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:09 smithi178 ceph-mon[67578]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-03-19T03:31:09.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:09 smithi178 ceph-mon[67578]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-03-19T03:31:10.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.132+0000 7f01a3a00640 1 -- 172.21.15.117:0/3434569708 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f019c061cf0 con 0x7f01840a2880 2024-03-19T03:31:10.139 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.138+0000 7f0199600640 1 -- 172.21.15.117:0/3434569708 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f019c061cf0 con 0x7f01840a2880 2024-03-19T03:31:10.139 INFO:teuthology.orchestra.run.smithi117.stdout:/volumes/qa/sv_0/72ec05c7-865f-43f5-a93a-81f1218ac285 2024-03-19T03:31:10.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.141+0000 7f01a3a00640 1 -- 172.21.15.117:0/3434569708 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f01840a2880 msgr2=0x7f01840a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:10.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.141+0000 7f01a3a00640 1 --2- 172.21.15.117:0/3434569708 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f01840a2880 0x7f01840a4d40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f019c075340 tx=0x7f0188005f90 comp rx=0 tx=0).stop 2024-03-19T03:31:10.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.141+0000 7f01a3a00640 1 -- 172.21.15.117:0/3434569708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f019c101830 msgr2=0x7f019c0755d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:10.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.141+0000 7f01a3a00640 1 --2- 172.21.15.117:0/3434569708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f019c101830 0x7f019c0755d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0190000c00 tx=0x7f019002c650 comp rx=0 tx=0).stop 2024-03-19T03:31:10.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.141+0000 7f01a3a00640 1 -- 172.21.15.117:0/3434569708 shutdown_connections 2024-03-19T03:31:10.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.141+0000 7f01a3a00640 1 --2- 172.21.15.117:0/3434569708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f019c1033e0 0x7f019c074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:10.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.141+0000 7f01a3a00640 1 --2- 172.21.15.117:0/3434569708 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f019c102a20 0x7f019c073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:10.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.141+0000 7f01a3a00640 1 --2- 172.21.15.117:0/3434569708 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f01840a2880 0x7f01840a4d40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:10.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.141+0000 7f01a3a00640 1 --2- 172.21.15.117:0/3434569708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f019c101830 0x7f019c0755d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:10.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.141+0000 7f01a3a00640 1 -- 172.21.15.117:0/3434569708 >> 172.21.15.117:0/3434569708 conn(0x7f019c0fcdc0 msgr2=0x7f019c105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:10.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.141+0000 7f01a3a00640 1 -- 172.21.15.117:0/3434569708 shutdown_connections 2024-03-19T03:31:10.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.142+0000 7f01a3a00640 1 -- 172.21.15.117:0/3434569708 wait complete. 2024-03-19T03:31:10.163 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-19T03:31:10.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.354+0000 7fac9b200640 1 -- 172.21.15.117:0/2023668939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac9c102910 msgr2=0x7fac9c102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:10.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.354+0000 7fac9b200640 1 --2- 172.21.15.117:0/2023668939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac9c102910 0x7fac9c102d90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fac88009900 tx=0x7fac8802bc30 comp rx=0 tx=0).stop 2024-03-19T03:31:10.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.354+0000 7fac9b200640 1 -- 172.21.15.117:0/2023668939 shutdown_connections 2024-03-19T03:31:10.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.354+0000 7fac9b200640 1 --2- 172.21.15.117:0/2023668939 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fac9c1032d0 0x7fac9c109b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:10.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.354+0000 7fac9b200640 1 --2- 172.21.15.117:0/2023668939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac9c102910 0x7fac9c102d90 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:10.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.354+0000 7fac9b200640 1 --2- 172.21.15.117:0/2023668939 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac9c101720 0x7fac9c101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:10.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.354+0000 7fac9b200640 1 -- 172.21.15.117:0/2023668939 >> 172.21.15.117:0/2023668939 conn(0x7fac9c0fcc90 msgr2=0x7fac9c0ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:10.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.354+0000 7fac9b200640 1 -- 172.21.15.117:0/2023668939 shutdown_connections 2024-03-19T03:31:10.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.354+0000 7fac9b200640 1 -- 172.21.15.117:0/2023668939 wait complete. 2024-03-19T03:31:10.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.355+0000 7fac9b200640 1 Processor -- start 2024-03-19T03:31:10.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.355+0000 7fac9b200640 1 -- start start 2024-03-19T03:31:10.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.356+0000 7fac9b200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fac9c101720 0x7fac9c0755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:10.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.356+0000 7fac9b200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac9c102910 0x7fac9c073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:10.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.356+0000 7fac9b200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac9c1032d0 0x7fac9c074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:10.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.356+0000 7fac9b200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fac9c10d3f0 con 0x7fac9c102910 2024-03-19T03:31:10.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.356+0000 7fac9b200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fac9c10d270 con 0x7fac9c1032d0 2024-03-19T03:31:10.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.356+0000 7fac9b200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fac9c10d570 con 0x7fac9c101720 2024-03-19T03:31:10.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.356+0000 7fac99400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac9c102910 0x7fac9c073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:10.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.356+0000 7fac9a800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac9c1032d0 0x7fac9c074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:10.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.356+0000 7fac99400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac9c102910 0x7fac9c073c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60888/0 (socket says 172.21.15.117:60888) 2024-03-19T03:31:10.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.356+0000 7fac99400640 1 -- 172.21.15.117:0/194780069 learned_addr learned my addr 172.21.15.117:0/194780069 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:10.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.356+0000 7fac99e00640 1 --2- 172.21.15.117:0/194780069 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fac9c101720 0x7fac9c0755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:10.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.357+0000 7fac9a800640 1 -- 172.21.15.117:0/194780069 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fac9c101720 msgr2=0x7fac9c0755d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:10.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.357+0000 7fac9a800640 1 --2- 172.21.15.117:0/194780069 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fac9c101720 0x7fac9c0755d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:10.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.357+0000 7fac9a800640 1 -- 172.21.15.117:0/194780069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac9c102910 msgr2=0x7fac9c073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:10.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.357+0000 7fac9a800640 1 --2- 172.21.15.117:0/194780069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac9c102910 0x7fac9c073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:10.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.357+0000 7fac9a800640 1 -- 172.21.15.117:0/194780069 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac9c0749e0 con 0x7fac9c1032d0 2024-03-19T03:31:10.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.357+0000 7fac9a800640 1 --2- 172.21.15.117:0/194780069 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac9c1032d0 0x7fac9c074150 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fac9400d680 tx=0x7fac9400db40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:10.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.357+0000 7fac92a00640 1 -- 172.21.15.117:0/194780069 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac94017070 con 0x7fac9c1032d0 2024-03-19T03:31:10.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.357+0000 7fac92a00640 1 -- 172.21.15.117:0/194780069 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac94009820 con 0x7fac9c1032d0 2024-03-19T03:31:10.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.357+0000 7fac9b200640 1 -- 172.21.15.117:0/194780069 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac9c1a1b80 con 0x7fac9c1032d0 2024-03-19T03:31:10.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.357+0000 7fac92a00640 1 -- 172.21.15.117:0/194780069 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac94004e00 con 0x7fac9c1032d0 2024-03-19T03:31:10.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.357+0000 7fac9b200640 1 -- 172.21.15.117:0/194780069 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac9c1a20f0 con 0x7fac9c1032d0 2024-03-19T03:31:10.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.358+0000 7fac9b200640 1 -- 172.21.15.117:0/194780069 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac9c108a40 con 0x7fac9c1032d0 2024-03-19T03:31:10.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.360+0000 7fac92a00640 1 -- 172.21.15.117:0/194780069 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fac940099d0 con 0x7fac9c1032d0 2024-03-19T03:31:10.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.361+0000 7fac92a00640 1 --2- 172.21.15.117:0/194780069 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fac800a2880 0x7fac800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:10.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.361+0000 7fac92a00640 1 -- 172.21.15.117:0/194780069 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fac940ce430 con 0x7fac9c1032d0 2024-03-19T03:31:10.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.361+0000 7fac99e00640 1 --2- 172.21.15.117:0/194780069 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fac800a2880 0x7fac800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:10.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.362+0000 7fac99e00640 1 --2- 172.21.15.117:0/194780069 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fac800a2880 0x7fac800a4d40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fac9c102770 tx=0x7fac8c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:10.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.365+0000 7fac92a00640 1 -- 172.21.15.117:0/194780069 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fac9409abe0 con 0x7fac9c1032d0 2024-03-19T03:31:10.615 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.614+0000 7fac9b200640 1 -- 172.21.15.117:0/194780069 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fac9c061cf0 con 0x7fac800a2880 2024-03-19T03:31:10.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.641+0000 7fac92a00640 1 -- 172.21.15.117:0/194780069 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fac9c061cf0 con 0x7fac800a2880 2024-03-19T03:31:10.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.643+0000 7fac9b200640 1 -- 172.21.15.117:0/194780069 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fac800a2880 msgr2=0x7fac800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:10.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.643+0000 7fac9b200640 1 --2- 172.21.15.117:0/194780069 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fac800a2880 0x7fac800a4d40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fac9c102770 tx=0x7fac8c009500 comp rx=0 tx=0).stop 2024-03-19T03:31:10.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.643+0000 7fac9b200640 1 -- 172.21.15.117:0/194780069 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac9c1032d0 msgr2=0x7fac9c074150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:10.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.643+0000 7fac9b200640 1 --2- 172.21.15.117:0/194780069 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac9c1032d0 0x7fac9c074150 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fac9400d680 tx=0x7fac9400db40 comp rx=0 tx=0).stop 2024-03-19T03:31:10.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.643+0000 7fac9b200640 1 -- 172.21.15.117:0/194780069 shutdown_connections 2024-03-19T03:31:10.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.643+0000 7fac9b200640 1 --2- 172.21.15.117:0/194780069 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fac800a2880 0x7fac800a4d40 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:10.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.643+0000 7fac9b200640 1 --2- 172.21.15.117:0/194780069 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac9c1032d0 0x7fac9c074150 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:10.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.643+0000 7fac9b200640 1 --2- 172.21.15.117:0/194780069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac9c102910 0x7fac9c073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:10.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.643+0000 7fac9b200640 1 --2- 172.21.15.117:0/194780069 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fac9c101720 0x7fac9c0755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:10.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.643+0000 7fac9b200640 1 -- 172.21.15.117:0/194780069 >> 172.21.15.117:0/194780069 conn(0x7fac9c0fcc90 msgr2=0x7fac9c105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:10.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.644+0000 7fac9b200640 1 -- 172.21.15.117:0/194780069 shutdown_connections 2024-03-19T03:31:10.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.644+0000 7fac9b200640 1 -- 172.21.15.117:0/194780069 wait complete. 2024-03-19T03:31:10.666 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-19T03:31:10.863 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.861+0000 7f2d7be00640 1 -- 172.21.15.117:0/2698262000 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d7c1021b0 msgr2=0x7f2d7c108a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:10.863 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.861+0000 7f2d7be00640 1 --2- 172.21.15.117:0/2698262000 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d7c1021b0 0x7f2d7c108a40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2d74009900 tx=0x7f2d7402bc30 comp rx=0 tx=0).stop 2024-03-19T03:31:10.863 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.862+0000 7f2d7be00640 1 -- 172.21.15.117:0/2698262000 shutdown_connections 2024-03-19T03:31:10.863 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.862+0000 7f2d7be00640 1 --2- 172.21.15.117:0/2698262000 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d7c1021b0 0x7f2d7c108a40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:10.863 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.862+0000 7f2d7be00640 1 --2- 172.21.15.117:0/2698262000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d7c1017f0 0x7f2d7c101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:10.863 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.862+0000 7f2d7be00640 1 --2- 172.21.15.117:0/2698262000 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d7c100600 0x7f2d7c100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:10.863 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.862+0000 7f2d7be00640 1 -- 172.21.15.117:0/2698262000 >> 172.21.15.117:0/2698262000 conn(0x7f2d7c0fbb90 msgr2=0x7f2d7c0fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:10.863 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.862+0000 7f2d7be00640 1 -- 172.21.15.117:0/2698262000 shutdown_connections 2024-03-19T03:31:10.863 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.862+0000 7f2d7be00640 1 -- 172.21.15.117:0/2698262000 wait complete. 2024-03-19T03:31:10.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.863+0000 7f2d7be00640 1 Processor -- start 2024-03-19T03:31:10.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.863+0000 7f2d7be00640 1 -- start start 2024-03-19T03:31:10.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.863+0000 7f2d7be00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d7c100600 0x7f2d7c1979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:10.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.863+0000 7f2d7be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d7c1017f0 0x7f2d7c197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:10.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.863+0000 7f2d7be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d7c1021b0 0x7f2d7c19ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:10.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.863+0000 7f2d7be00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7c10c280 con 0x7f2d7c1017f0 2024-03-19T03:31:10.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.863+0000 7f2d7be00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7c10c100 con 0x7f2d7c1021b0 2024-03-19T03:31:10.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.863+0000 7f2d7be00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7c10c400 con 0x7f2d7c100600 2024-03-19T03:31:10.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.863+0000 7f2d72000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d7c1017f0 0x7f2d7c197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:10.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.863+0000 7f2d7aa00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d7c100600 0x7f2d7c1979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:10.866 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.863+0000 7f2d7b400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d7c1021b0 0x7f2d7c19ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:10.866 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.863+0000 7f2d7aa00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d7c100600 0x7f2d7c1979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:49368/0 (socket says 172.21.15.117:49368) 2024-03-19T03:31:10.866 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.863+0000 7f2d72000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d7c1017f0 0x7f2d7c197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60898/0 (socket says 172.21.15.117:60898) 2024-03-19T03:31:10.866 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.863+0000 7f2d7aa00640 1 -- 172.21.15.117:0/784699333 learned_addr learned my addr 172.21.15.117:0/784699333 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:10.866 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.864+0000 7f2d7aa00640 1 -- 172.21.15.117:0/784699333 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d7c1021b0 msgr2=0x7f2d7c19ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:10.866 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.864+0000 7f2d7aa00640 1 --2- 172.21.15.117:0/784699333 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d7c1021b0 0x7f2d7c19ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:10.866 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.864+0000 7f2d7aa00640 1 -- 172.21.15.117:0/784699333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d7c1017f0 msgr2=0x7f2d7c197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:10.866 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.864+0000 7f2d7aa00640 1 --2- 172.21.15.117:0/784699333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d7c1017f0 0x7f2d7c197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:10.866 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.864+0000 7f2d7aa00640 1 -- 172.21.15.117:0/784699333 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d7c19f5d0 con 0x7f2d7c100600 2024-03-19T03:31:10.866 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.864+0000 7f2d7b400640 1 --2- 172.21.15.117:0/784699333 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d7c1021b0 0x7f2d7c19ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:31:10.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.864+0000 7f2d7aa00640 1 --2- 172.21.15.117:0/784699333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d7c100600 0x7f2d7c1979b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00eb70 tx=0x7f2d6c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:10.867 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.864+0000 7f2d80a00640 1 -- 172.21.15.117:0/784699333 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c003b70 con 0x7f2d7c100600 2024-03-19T03:31:10.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.865+0000 7f2d80a00640 1 -- 172.21.15.117:0/784699333 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d6c00ee70 con 0x7f2d7c100600 2024-03-19T03:31:10.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.865+0000 7f2d7be00640 1 -- 172.21.15.117:0/784699333 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d7c102a70 con 0x7f2d7c100600 2024-03-19T03:31:10.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.865+0000 7f2d80a00640 1 -- 172.21.15.117:0/784699333 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c010830 con 0x7f2d7c100600 2024-03-19T03:31:10.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.865+0000 7f2d7be00640 1 -- 172.21.15.117:0/784699333 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d7c102fc0 con 0x7f2d7c100600 2024-03-19T03:31:10.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.866+0000 7f2d7be00640 1 -- 172.21.15.117:0/784699333 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d7c103280 con 0x7f2d7c100600 2024-03-19T03:31:10.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.868+0000 7f2d80a00640 1 -- 172.21.15.117:0/784699333 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f2d6c004a40 con 0x7f2d7c100600 2024-03-19T03:31:10.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.869+0000 7f2d80a00640 1 --2- 172.21.15.117:0/784699333 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f2d600a2880 0x7f2d600a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:10.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.869+0000 7f2d80a00640 1 -- 172.21.15.117:0/784699333 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2d6c0cf2a0 con 0x7f2d7c100600 2024-03-19T03:31:10.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.869+0000 7f2d72000640 1 --2- 172.21.15.117:0/784699333 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f2d600a2880 0x7f2d600a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:10.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.870+0000 7f2d72000640 1 --2- 172.21.15.117:0/784699333 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f2d600a2880 0x7f2d600a4d40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2d640095f0 tx=0x7f2d64005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:10.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:10.872+0000 7f2d80a00640 1 -- 172.21.15.117:0/784699333 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2d6c09c100 con 0x7f2d7c100600 2024-03-19T03:31:11.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.017+0000 7f2d80a00640 1 -- 172.21.15.117:0/784699333 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f2d6c098070 con 0x7f2d7c100600 2024-03-19T03:31:11.118 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.117+0000 7f2d7be00640 1 -- 172.21.15.117:0/784699333 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f2d7c0fdd10 con 0x7f2d600a2880 2024-03-19T03:31:11.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.123+0000 7f2d80a00640 1 -- 172.21.15.117:0/784699333 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f2d7c0fdd10 con 0x7f2d600a2880 2024-03-19T03:31:11.124 INFO:teuthology.orchestra.run.smithi117.stdout:/volumes/qa/sv_1/8f1a3abf-f504-494e-80fd-bcaac3a6e05c 2024-03-19T03:31:11.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.126+0000 7f2d7be00640 1 -- 172.21.15.117:0/784699333 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f2d600a2880 msgr2=0x7f2d600a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:11.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.126+0000 7f2d7be00640 1 --2- 172.21.15.117:0/784699333 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f2d600a2880 0x7f2d600a4d40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2d640095f0 tx=0x7f2d64005f90 comp rx=0 tx=0).stop 2024-03-19T03:31:11.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.126+0000 7f2d7be00640 1 -- 172.21.15.117:0/784699333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d7c100600 msgr2=0x7f2d7c1979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:11.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.126+0000 7f2d7be00640 1 --2- 172.21.15.117:0/784699333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d7c100600 0x7f2d7c1979b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00eb70 tx=0x7f2d6c00c6a0 comp rx=0 tx=0).stop 2024-03-19T03:31:11.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.126+0000 7f2d7be00640 1 -- 172.21.15.117:0/784699333 shutdown_connections 2024-03-19T03:31:11.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.126+0000 7f2d7be00640 1 --2- 172.21.15.117:0/784699333 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f2d600a2880 0x7f2d600a4d40 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:11.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.126+0000 7f2d7be00640 1 --2- 172.21.15.117:0/784699333 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d7c1021b0 0x7f2d7c19ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:11.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.126+0000 7f2d7be00640 1 --2- 172.21.15.117:0/784699333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d7c1017f0 0x7f2d7c197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:11.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.126+0000 7f2d7be00640 1 --2- 172.21.15.117:0/784699333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d7c100600 0x7f2d7c1979b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:11.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.126+0000 7f2d7be00640 1 -- 172.21.15.117:0/784699333 >> 172.21.15.117:0/784699333 conn(0x7f2d7c0fbb90 msgr2=0x7f2d7c0fd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:11.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.126+0000 7f2d7be00640 1 -- 172.21.15.117:0/784699333 shutdown_connections 2024-03-19T03:31:11.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.126+0000 7f2d7be00640 1 -- 172.21.15.117:0/784699333 wait complete. 2024-03-19T03:31:11.145 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:31:11.291 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:10 smithi117 ceph-mon[31838]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:31:11.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:10 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] up:active 2024-03-19T03:31:11.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:10 smithi117 ceph-mon[31838]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-19T03:31:11.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:10 smithi117 ceph-mon[31838]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:31:11.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:10 smithi117 ceph-mon[31838]: from='client.24857 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:31:11.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:10 smithi117 ceph-mon[31838]: pgmap v335: 129 pgs: 129 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 1023 B/s wr, 3 op/s 2024-03-19T03:31:11.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.340+0000 7f3023000640 1 -- 172.21.15.117:0/3956637218 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f301c102a30 msgr2=0x7f301c102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:11.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.340+0000 7f3023000640 1 --2- 172.21.15.117:0/3956637218 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f301c102a30 0x7f301c102eb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3004009900 tx=0x7f300402bc30 comp rx=0 tx=0).stop 2024-03-19T03:31:11.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.340+0000 7f3023000640 1 -- 172.21.15.117:0/3956637218 shutdown_connections 2024-03-19T03:31:11.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.340+0000 7f3023000640 1 --2- 172.21.15.117:0/3956637218 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f301c1033f0 0x7f301c109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:11.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.340+0000 7f3023000640 1 --2- 172.21.15.117:0/3956637218 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f301c102a30 0x7f301c102eb0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:11.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.340+0000 7f3023000640 1 --2- 172.21.15.117:0/3956637218 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f301c101840 0x7f301c101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:11.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.340+0000 7f3023000640 1 -- 172.21.15.117:0/3956637218 >> 172.21.15.117:0/3956637218 conn(0x7f301c0fcdd0 msgr2=0x7f301c0ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:11.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.340+0000 7f3023000640 1 -- 172.21.15.117:0/3956637218 shutdown_connections 2024-03-19T03:31:11.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.340+0000 7f3023000640 1 -- 172.21.15.117:0/3956637218 wait complete. 2024-03-19T03:31:11.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.341+0000 7f3023000640 1 Processor -- start 2024-03-19T03:31:11.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.341+0000 7f3023000640 1 -- start start 2024-03-19T03:31:11.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.341+0000 7f3023000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f301c101840 0x7f301c199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:11.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.342+0000 7f3023000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f301c102a30 0x7f301c19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:11.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.342+0000 7f3023000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f301c1033f0 0x7f301c1a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:11.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.342+0000 7f3023000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f301c10d4f0 con 0x7f301c101840 2024-03-19T03:31:11.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.342+0000 7f3023000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f301c10d370 con 0x7f301c1033f0 2024-03-19T03:31:11.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.342+0000 7f3023000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f301c10d670 con 0x7f301c102a30 2024-03-19T03:31:11.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.342+0000 7f301be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f301c101840 0x7f301c199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:11.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.342+0000 7f3021000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f301c1033f0 0x7f301c1a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:11.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.342+0000 7f301be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f301c101840 0x7f301c199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60906/0 (socket says 172.21.15.117:60906) 2024-03-19T03:31:11.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.342+0000 7f301be00640 1 -- 172.21.15.117:0/3969593586 learned_addr learned my addr 172.21.15.117:0/3969593586 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.342+0000 7f3013400640 1 --2- 172.21.15.117:0/3969593586 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f301c102a30 0x7f301c19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.342+0000 7f3021000640 1 -- 172.21.15.117:0/3969593586 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f301c102a30 msgr2=0x7f301c19a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.342+0000 7f3021000640 1 --2- 172.21.15.117:0/3969593586 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f301c102a30 0x7f301c19a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.342+0000 7f3021000640 1 -- 172.21.15.117:0/3969593586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f301c101840 msgr2=0x7f301c199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.342+0000 7f3021000640 1 --2- 172.21.15.117:0/3969593586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f301c101840 0x7f301c199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.342+0000 7f3021000640 1 -- 172.21.15.117:0/3969593586 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f301c1a1970 con 0x7f301c1033f0 2024-03-19T03:31:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.342+0000 7f301be00640 1 --2- 172.21.15.117:0/3969593586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f301c101840 0x7f301c199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:31:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.343+0000 7f3021000640 1 --2- 172.21.15.117:0/3969593586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f301c1033f0 0x7f301c1a1300 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f301400e970 tx=0x7f301400ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.343+0000 7f3019600640 1 -- 172.21.15.117:0/3969593586 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f301400ccc0 con 0x7f301c1033f0 2024-03-19T03:31:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.343+0000 7f3019600640 1 -- 172.21.15.117:0/3969593586 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3014005020 con 0x7f301c1033f0 2024-03-19T03:31:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.343+0000 7f3019600640 1 -- 172.21.15.117:0/3969593586 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3014004a00 con 0x7f301c1033f0 2024-03-19T03:31:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.343+0000 7f3023000640 1 -- 172.21.15.117:0/3969593586 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f301c1a1c60 con 0x7f301c1033f0 2024-03-19T03:31:11.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.343+0000 7f3023000640 1 -- 172.21.15.117:0/3969593586 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f301c1a2130 con 0x7f301c1033f0 2024-03-19T03:31:11.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:10 smithi178 ceph-mon[67578]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:31:11.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:10 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] up:active 2024-03-19T03:31:11.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:10 smithi178 ceph-mon[67578]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-19T03:31:11.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:10 smithi178 ceph-mon[67578]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:31:11.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:10 smithi178 ceph-mon[67578]: from='client.24857 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:31:11.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:10 smithi178 ceph-mon[67578]: pgmap v335: 129 pgs: 129 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 1023 B/s wr, 3 op/s 2024-03-19T03:31:11.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.345+0000 7f3023000640 1 -- 172.21.15.117:0/3969593586 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f301c1a2360 con 0x7f301c1033f0 2024-03-19T03:31:11.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.346+0000 7f3019600640 1 -- 172.21.15.117:0/3969593586 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f30140051d0 con 0x7f301c1033f0 2024-03-19T03:31:11.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.347+0000 7f3019600640 1 --2- 172.21.15.117:0/3969593586 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f30000a2890 0x7f30000a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:11.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.348+0000 7f3019600640 1 -- 172.21.15.117:0/3969593586 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f30140ced00 con 0x7f301c1033f0 2024-03-19T03:31:11.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.348+0000 7f301be00640 1 --2- 172.21.15.117:0/3969593586 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f30000a2890 0x7f30000a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:11.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.348+0000 7f301be00640 1 --2- 172.21.15.117:0/3969593586 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f30000a2890 0x7f30000a4d50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f300c005c40 tx=0x7f300c0097a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:11.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.352+0000 7f3019600640 1 -- 172.21.15.117:0/3969593586 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f301409ba00 con 0x7f301c1033f0 2024-03-19T03:31:11.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:10 smithi119 ceph-mon[66861]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:31:11.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:10 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] up:active 2024-03-19T03:31:11.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:10 smithi119 ceph-mon[66861]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-19T03:31:11.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:10 smithi119 ceph-mon[66861]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:31:11.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:10 smithi119 ceph-mon[66861]: from='client.24857 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:31:11.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:10 smithi119 ceph-mon[66861]: pgmap v335: 129 pgs: 129 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 1023 B/s wr, 3 op/s 2024-03-19T03:31:11.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.592+0000 7f3023000640 1 -- 172.21.15.117:0/3969593586 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f301c1a2530 con 0x7f301c1033f0 2024-03-19T03:31:11.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.593+0000 7f3019600640 1 -- 172.21.15.117:0/3969593586 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11953 (secure 0 0 0) 0x7f301409b300 con 0x7f301c1033f0 2024-03-19T03:31:11.594 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:31:11.594 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/54638259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":54638259},{"type":"v1","addr":"172.21.15.117:6835","nonce":54638259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3608185039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3608185039},{"type":"v1","addr":"172.21.15.117:6837","nonce":3608185039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T03:30:35.081740+0000","modified":"2024-03-19T03:31:09.995133+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24599,"mds_1":24581,"mds_2":24569,"mds_3":24545,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":20,"addr":"172.21.15.119:6833/1291833214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1291833214},{"type":"v1","addr":"172.21.15.119:6833","nonce":1291833214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6835/1579860804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1579860804},{"type":"v1","addr":"172.21.15.178:6835","nonce":1579860804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6835/1686938941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1686938941},{"type":"v1","addr":"172.21.15.119:6835","nonce":1686938941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6837/1770913538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":1770913538},{"type":"v1","addr":"172.21.15.178:6837","nonce":1770913538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6839/2655593383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2655593383},{"type":"v1","addr":"172.21.15.178:6839","nonce":2655593383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.119:6837/2054992738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2054992738},{"type":"v1","addr":"172.21.15.119:6837","nonce":2054992738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"g","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.117:6839/684335078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":684335078},{"type":"v1","addr":"172.21.15.117:6839","nonce":684335078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.117:6841/1329933892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6840","nonce":1329933892},{"type":"v1","addr":"172.21.15.117:6841","nonce":1329933892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/2383354833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2383354833},{"type":"v1","addr":"172.21.15.119:6839","nonce":2383354833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24581,24599,24569,24545,24467]},"id":1}]} 2024-03-19T03:31:11.598 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 28 2024-03-19T03:31:11.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.597+0000 7f3023000640 1 -- 172.21.15.117:0/3969593586 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f30000a2890 msgr2=0x7f30000a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:11.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.597+0000 7f3023000640 1 --2- 172.21.15.117:0/3969593586 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f30000a2890 0x7f30000a4d50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f300c005c40 tx=0x7f300c0097a0 comp rx=0 tx=0).stop 2024-03-19T03:31:11.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.597+0000 7f3023000640 1 -- 172.21.15.117:0/3969593586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f301c1033f0 msgr2=0x7f301c1a1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:11.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.597+0000 7f3023000640 1 --2- 172.21.15.117:0/3969593586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f301c1033f0 0x7f301c1a1300 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f301400e970 tx=0x7f301400ee30 comp rx=0 tx=0).stop 2024-03-19T03:31:11.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.597+0000 7f3023000640 1 -- 172.21.15.117:0/3969593586 shutdown_connections 2024-03-19T03:31:11.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.597+0000 7f3023000640 1 --2- 172.21.15.117:0/3969593586 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f30000a2890 0x7f30000a4d50 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:11.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.597+0000 7f3023000640 1 --2- 172.21.15.117:0/3969593586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f301c1033f0 0x7f301c1a1300 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:11.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.597+0000 7f3023000640 1 --2- 172.21.15.117:0/3969593586 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f301c102a30 0x7f301c19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:11.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.597+0000 7f3023000640 1 --2- 172.21.15.117:0/3969593586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f301c101840 0x7f301c199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:11.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.597+0000 7f3023000640 1 -- 172.21.15.117:0/3969593586 >> 172.21.15.117:0/3969593586 conn(0x7f301c0fcdd0 msgr2=0x7f301c073a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:11.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.598+0000 7f3023000640 1 -- 172.21.15.117:0/3969593586 shutdown_connections 2024-03-19T03:31:11.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.598+0000 7f3023000640 1 -- 172.21.15.117:0/3969593586 wait complete. 2024-03-19T03:31:11.617 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:31:11.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.840+0000 7f773d800640 1 -- 172.21.15.117:0/1173622024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7738101840 msgr2=0x7f7738101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:11.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.840+0000 7f773d800640 1 --2- 172.21.15.117:0/1173622024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7738101840 0x7f7738101c60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7728009900 tx=0x7f772802bc30 comp rx=0 tx=0).stop 2024-03-19T03:31:11.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.840+0000 7f773d800640 1 -- 172.21.15.117:0/1173622024 shutdown_connections 2024-03-19T03:31:11.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.840+0000 7f773d800640 1 --2- 172.21.15.117:0/1173622024 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f77381033f0 0x7f7738109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:11.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.840+0000 7f773d800640 1 --2- 172.21.15.117:0/1173622024 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7738102a30 0x7f7738102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:11.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.840+0000 7f773d800640 1 --2- 172.21.15.117:0/1173622024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7738101840 0x7f7738101c60 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:11.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.840+0000 7f773d800640 1 -- 172.21.15.117:0/1173622024 >> 172.21.15.117:0/1173622024 conn(0x7f77380fcdd0 msgr2=0x7f77380ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:11.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.840+0000 7f773d800640 1 -- 172.21.15.117:0/1173622024 shutdown_connections 2024-03-19T03:31:11.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.840+0000 7f773d800640 1 -- 172.21.15.117:0/1173622024 wait complete. 2024-03-19T03:31:11.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.841+0000 7f773d800640 1 Processor -- start 2024-03-19T03:31:11.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.841+0000 7f773d800640 1 -- start start 2024-03-19T03:31:11.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.841+0000 7f773d800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7738101840 0x7f77380755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:11.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.841+0000 7f773d800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7738102a30 0x7f7738073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:11.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.841+0000 7f773d800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f77381033f0 0x7f7738074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:11.846 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.841+0000 7f773d800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f773810d510 con 0x7f7738102a30 2024-03-19T03:31:11.846 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.841+0000 7f773d800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f773810d390 con 0x7f7738101840 2024-03-19T03:31:11.846 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.841+0000 7f773d800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f773810d690 con 0x7f77381033f0 2024-03-19T03:31:11.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.842+0000 7f7736a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7738101840 0x7f77380755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:11.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.842+0000 7f7736a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7738101840 0x7f77380755d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:51956/0 (socket says 172.21.15.117:51956) 2024-03-19T03:31:11.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.842+0000 7f7736a00640 1 -- 172.21.15.117:0/2294578502 learned_addr learned my addr 172.21.15.117:0/2294578502 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:11.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.842+0000 7f7736000640 1 --2- 172.21.15.117:0/2294578502 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7738102a30 0x7f7738073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:11.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.842+0000 7f7737400640 1 --2- 172.21.15.117:0/2294578502 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f77381033f0 0x7f7738074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:11.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.842+0000 7f7736a00640 1 -- 172.21.15.117:0/2294578502 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f77381033f0 msgr2=0x7f7738074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:11.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.842+0000 7f7736a00640 1 --2- 172.21.15.117:0/2294578502 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f77381033f0 0x7f7738074150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:11.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.842+0000 7f7736a00640 1 -- 172.21.15.117:0/2294578502 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7738102a30 msgr2=0x7f7738073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:11.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.842+0000 7f7736a00640 1 --2- 172.21.15.117:0/2294578502 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7738102a30 0x7f7738073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:11.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.842+0000 7f7736a00640 1 -- 172.21.15.117:0/2294578502 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77380749e0 con 0x7f7738101840 2024-03-19T03:31:11.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.843+0000 7f7736a00640 1 --2- 172.21.15.117:0/2294578502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7738101840 0x7f77380755d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7728000c00 tx=0x7f772802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:11.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.843+0000 7f772f400640 1 -- 172.21.15.117:0/2294578502 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7728043020 con 0x7f7738101840 2024-03-19T03:31:11.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.843+0000 7f772f400640 1 -- 172.21.15.117:0/2294578502 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77280054e0 con 0x7f7738101840 2024-03-19T03:31:11.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.843+0000 7f772f400640 1 -- 172.21.15.117:0/2294578502 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7728003c40 con 0x7f7738101840 2024-03-19T03:31:11.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.843+0000 7f773d800640 1 -- 172.21.15.117:0/2294578502 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77381a1ca0 con 0x7f7738101840 2024-03-19T03:31:11.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.843+0000 7f773d800640 1 -- 172.21.15.117:0/2294578502 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77381a21a0 con 0x7f7738101840 2024-03-19T03:31:11.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.844+0000 7f773d800640 1 -- 172.21.15.117:0/2294578502 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7738108b60 con 0x7f7738101840 2024-03-19T03:31:11.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.846+0000 7f772f400640 1 -- 172.21.15.117:0/2294578502 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f7728046020 con 0x7f7738101840 2024-03-19T03:31:11.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.847+0000 7f772f400640 1 --2- 172.21.15.117:0/2294578502 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f771c0a2910 0x7f771c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:11.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.847+0000 7f772f400640 1 -- 172.21.15.117:0/2294578502 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f77280f0ce0 con 0x7f7738101840 2024-03-19T03:31:11.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.848+0000 7f7736000640 1 --2- 172.21.15.117:0/2294578502 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f771c0a2910 0x7f771c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:11.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.848+0000 7f7736000640 1 --2- 172.21.15.117:0/2294578502 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f771c0a2910 0x7f771c0a4dd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7738075340 tx=0x7f7724005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:11.856 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:11.856+0000 7f772f400640 1 -- 172.21.15.117:0/2294578502 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f772804d020 con 0x7f7738101840 2024-03-19T03:31:12.083 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.082+0000 7f773d800640 1 -- 172.21.15.117:0/2294578502 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f773802d430 con 0x7f7738101840 2024-03-19T03:31:12.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.084+0000 7f772f400640 1 -- 172.21.15.117:0/2294578502 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24210 (secure 0 0 0) 0x7f77280bda90 con 0x7f7738101840 2024-03-19T03:31:12.085 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:31:12.085 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":89,"fsid":"a3a82456-e59f-11ee-95c9-87774f69a715","created":"2024-03-19T03:21:30.427136+0000","modified":"2024-03-19T03:30:40.180745+0000","last_up_change":"2024-03-19T03:29:32.967505+0000","last_in_change":"2024-03-19T03:29:12.848740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:25:18.824270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T03:30:33.137170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T03:30:34.598838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T03:30:38.230023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b32eb0ee-9d4b-4748-87ec-eda02c907415","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6803","nonce":3014934900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6805","nonce":3014934900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6809","nonce":3014934900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6807","nonce":3014934900}]},"public_addr":"172.21.15.117:6803/3014934900","cluster_addr":"172.21.15.117:6805/3014934900","heartbeat_back_addr":"172.21.15.117:6809/3014934900","heartbeat_front_addr":"172.21.15.117:6807/3014934900","state":["exists","up"]},{"osd":1,"uuid":"e07585be-c016-4466-b07c-abbc22a58de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6801","nonce":2425927341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6803","nonce":2425927341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6807","nonce":2425927341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6805","nonce":2425927341}]},"public_addr":"172.21.15.119:6801/2425927341","cluster_addr":"172.21.15.119:6803/2425927341","heartbeat_back_addr":"172.21.15.119:6807/2425927341","heartbeat_front_addr":"172.21.15.119:6805/2425927341","state":["exists","up"]},{"osd":2,"uuid":"ede9edda-3b9f-41ca-ad04-5dcb738ab757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6801","nonce":3733812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6803","nonce":3733812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6807","nonce":3733812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6805","nonce":3733812989}]},"public_addr":"172.21.15.178:6801/3733812989","cluster_addr":"172.21.15.178:6803/3733812989","heartbeat_back_addr":"172.21.15.178:6807/3733812989","heartbeat_front_addr":"172.21.15.178:6805/3733812989","state":["exists","up"]},{"osd":3,"uuid":"f81c41e1-3a69-40db-a90f-58151286b081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6811","nonce":2178310135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6813","nonce":2178310135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6817","nonce":2178310135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6815","nonce":2178310135}]},"public_addr":"172.21.15.117:6811/2178310135","cluster_addr":"172.21.15.117:6813/2178310135","heartbeat_back_addr":"172.21.15.117:6817/2178310135","heartbeat_front_addr":"172.21.15.117:6815/2178310135","state":["exists","up"]},{"osd":4,"uuid":"caf7ce86-caf0-4eaf-bbab-bd2d93e63c33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6809","nonce":3455398086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6811","nonce":3455398086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6815","nonce":3455398086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6813","nonce":3455398086}]},"public_addr":"172.21.15.119:6809/3455398086","cluster_addr":"172.21.15.119:6811/3455398086","heartbeat_back_addr":"172.21.15.119:6815/3455398086","heartbeat_front_addr":"172.21.15.119:6813/3455398086","state":["exists","up"]},{"osd":5,"uuid":"20ba0b8a-2e0f-4ea7-b435-430e44766200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6809","nonce":1259631912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6811","nonce":1259631912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6815","nonce":1259631912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6813","nonce":1259631912}]},"public_addr":"172.21.15.178:6809/1259631912","cluster_addr":"172.21.15.178:6811/1259631912","heartbeat_back_addr":"172.21.15.178:6815/1259631912","heartbeat_front_addr":"172.21.15.178:6813/1259631912","state":["exists","up"]},{"osd":6,"uuid":"75c2ddd7-f837-42e0-ac47-8cfbcf97a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6819","nonce":864146757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6821","nonce":864146757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6825","nonce":864146757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6823","nonce":864146757}]},"public_addr":"172.21.15.117:6819/864146757","cluster_addr":"172.21.15.117:6821/864146757","heartbeat_back_addr":"172.21.15.117:6825/864146757","heartbeat_front_addr":"172.21.15.117:6823/864146757","state":["exists","up"]},{"osd":7,"uuid":"929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6817","nonce":1273365545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6819","nonce":1273365545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6823","nonce":1273365545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6821","nonce":1273365545}]},"public_addr":"172.21.15.119:6817/1273365545","cluster_addr":"172.21.15.119:6819/1273365545","heartbeat_back_addr":"172.21.15.119:6823/1273365545","heartbeat_front_addr":"172.21.15.119:6821/1273365545","state":["exists","up"]},{"osd":8,"uuid":"fd371461-a538-4781-92e1-5eead4ca3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6817","nonce":1059587501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6819","nonce":1059587501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6823","nonce":1059587501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6821","nonce":1059587501}]},"public_addr":"172.21.15.178:6817/1059587501","cluster_addr":"172.21.15.178:6819/1059587501","heartbeat_back_addr":"172.21.15.178:6823/1059587501","heartbeat_front_addr":"172.21.15.178:6821/1059587501","state":["exists","up"]},{"osd":9,"uuid":"0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6827","nonce":3653875600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6829","nonce":3653875600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6833","nonce":3653875600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6831","nonce":3653875600}]},"public_addr":"172.21.15.117:6827/3653875600","cluster_addr":"172.21.15.117:6829/3653875600","heartbeat_back_addr":"172.21.15.117:6833/3653875600","heartbeat_front_addr":"172.21.15.117:6831/3653875600","state":["exists","up"]},{"osd":10,"uuid":"de2049da-d19e-4386-afbf-fd6b5fddbcba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6825","nonce":1050976194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6827","nonce":1050976194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6831","nonce":1050976194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6829","nonce":1050976194}]},"public_addr":"172.21.15.119:6825/1050976194","cluster_addr":"172.21.15.119:6827/1050976194","heartbeat_back_addr":"172.21.15.119:6831/1050976194","heartbeat_front_addr":"172.21.15.119:6829/1050976194","state":["exists","up"]},{"osd":11,"uuid":"3cccc0f9-d273-4b6b-9b8e-c926371472dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6825","nonce":2915080368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6827","nonce":2915080368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6831","nonce":2915080368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6829","nonce":2915080368}]},"public_addr":"172.21.15.178:6825/2915080368","cluster_addr":"172.21.15.178:6827/2915080368","heartbeat_back_addr":"172.21.15.178:6831/2915080368","heartbeat_front_addr":"172.21.15.178:6829/2915080368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:27.876274+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:53.094706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:16.598813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:48.160419+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:14.048925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:38.814304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:12.542328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:38.942747+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:04.488963+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:38.196053+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:05.254728+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:30.874915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.117:0/185684447":"2024-03-20T03:22:38.134578+0000","172.21.15.117:0/361367101":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/2788067603":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3506200675":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6800/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3283471541":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3750920169":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/2780813298":"2024-03-20T03:22:03.905461+0000","172.21.15.117:6800/2788067603":"2024-03-20T03:22:38.134578+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:31:12.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.088+0000 7f773d800640 1 -- 172.21.15.117:0/2294578502 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f771c0a2910 msgr2=0x7f771c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:12.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.088+0000 7f773d800640 1 --2- 172.21.15.117:0/2294578502 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f771c0a2910 0x7f771c0a4dd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7738075340 tx=0x7f7724005f90 comp rx=0 tx=0).stop 2024-03-19T03:31:12.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.088+0000 7f773d800640 1 -- 172.21.15.117:0/2294578502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7738101840 msgr2=0x7f77380755d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:12.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.088+0000 7f773d800640 1 --2- 172.21.15.117:0/2294578502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7738101840 0x7f77380755d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7728000c00 tx=0x7f772802c650 comp rx=0 tx=0).stop 2024-03-19T03:31:12.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.088+0000 7f773d800640 1 -- 172.21.15.117:0/2294578502 shutdown_connections 2024-03-19T03:31:12.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.088+0000 7f773d800640 1 --2- 172.21.15.117:0/2294578502 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f77381033f0 0x7f7738074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:12.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.088+0000 7f773d800640 1 --2- 172.21.15.117:0/2294578502 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7738102a30 0x7f7738073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:12.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.088+0000 7f773d800640 1 --2- 172.21.15.117:0/2294578502 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f771c0a2910 0x7f771c0a4dd0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:12.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.088+0000 7f773d800640 1 --2- 172.21.15.117:0/2294578502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7738101840 0x7f77380755d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:12.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.088+0000 7f773d800640 1 -- 172.21.15.117:0/2294578502 >> 172.21.15.117:0/2294578502 conn(0x7f77380fcdd0 msgr2=0x7f7738105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:12.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.088+0000 7f773d800640 1 -- 172.21.15.117:0/2294578502 shutdown_connections 2024-03-19T03:31:12.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.088+0000 7f773d800640 1 -- 172.21.15.117:0/2294578502 wait complete. 2024-03-19T03:31:12.109 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-19T03:31:12.109 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T03:31:12.291 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:12 smithi117 ceph-mon[31838]: from='client.24863 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:31:12.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:12 smithi117 ceph-mon[31838]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-19T03:31:12.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:12 smithi117 ceph-mon[31838]: from='client.24847 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:31:12.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:12 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3969593586' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:31:12.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.312+0000 7fb1cfe00640 1 -- 172.21.15.117:0/648593798 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb1d0101760 msgr2=0x7fb1d0101be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:12.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.312+0000 7fb1cfe00640 1 --2- 172.21.15.117:0/648593798 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb1d0101760 0x7fb1d0101be0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc009900 tx=0x7fb1bc02bc30 comp rx=0 tx=0).stop 2024-03-19T03:31:12.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.312+0000 7fb1cfe00640 1 -- 172.21.15.117:0/648593798 shutdown_connections 2024-03-19T03:31:12.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.312+0000 7fb1cfe00640 1 --2- 172.21.15.117:0/648593798 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1d0102120 0x7fb1d01089b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:12.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.312+0000 7fb1cfe00640 1 --2- 172.21.15.117:0/648593798 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb1d0101760 0x7fb1d0101be0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:12.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.312+0000 7fb1cfe00640 1 --2- 172.21.15.117:0/648593798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1d0100570 0x7fb1d0100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:12.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.312+0000 7fb1cfe00640 1 -- 172.21.15.117:0/648593798 >> 172.21.15.117:0/648593798 conn(0x7fb1d00fbb00 msgr2=0x7fb1d00fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:12.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.313+0000 7fb1cfe00640 1 -- 172.21.15.117:0/648593798 shutdown_connections 2024-03-19T03:31:12.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.313+0000 7fb1cfe00640 1 -- 172.21.15.117:0/648593798 wait complete. 2024-03-19T03:31:12.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.314+0000 7fb1cfe00640 1 Processor -- start 2024-03-19T03:31:12.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.314+0000 7fb1cfe00640 1 -- start start 2024-03-19T03:31:12.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.315+0000 7fb1cfe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb1d0100570 0x7fb1d0104500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:12.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.315+0000 7fb1cfe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1d0101760 0x7fb1d0102b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:12.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.315+0000 7fb1cfe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1d0102120 0x7fb1d0103080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:12.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.315+0000 7fb1cfe00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d010c240 con 0x7fb1d0101760 2024-03-19T03:31:12.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.315+0000 7fb1cfe00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d010c0c0 con 0x7fb1d0100570 2024-03-19T03:31:12.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.315+0000 7fb1cfe00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d010c3c0 con 0x7fb1d0102120 2024-03-19T03:31:12.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.315+0000 7fb1cea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb1d0100570 0x7fb1d0104500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:12.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.315+0000 7fb1cea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb1d0100570 0x7fb1d0104500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:51974/0 (socket says 172.21.15.117:51974) 2024-03-19T03:31:12.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.315+0000 7fb1cea00640 1 -- 172.21.15.117:0/285649237 learned_addr learned my addr 172.21.15.117:0/285649237 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:12.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.315+0000 7fb1ce000640 1 --2- 172.21.15.117:0/285649237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1d0101760 0x7fb1d0102b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:12.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.315+0000 7fb1cf400640 1 --2- 172.21.15.117:0/285649237 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1d0102120 0x7fb1d0103080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:12.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.315+0000 7fb1cea00640 1 -- 172.21.15.117:0/285649237 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1d0102120 msgr2=0x7fb1d0103080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:12.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.315+0000 7fb1cea00640 1 --2- 172.21.15.117:0/285649237 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1d0102120 0x7fb1d0103080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:12.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.315+0000 7fb1cea00640 1 -- 172.21.15.117:0/285649237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1d0101760 msgr2=0x7fb1d0102b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:12.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.315+0000 7fb1cea00640 1 --2- 172.21.15.117:0/285649237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1d0101760 0x7fb1d0102b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:12.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.315+0000 7fb1cea00640 1 -- 172.21.15.117:0/285649237 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1d0103910 con 0x7fb1d0100570 2024-03-19T03:31:12.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.315+0000 7fb1cf400640 1 --2- 172.21.15.117:0/285649237 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1d0102120 0x7fb1d0103080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:31:12.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.316+0000 7fb1cea00640 1 --2- 172.21.15.117:0/285649237 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb1d0100570 0x7fb1d0104500 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb1c000e9c0 tx=0x7fb1c000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:12.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.316+0000 7fb1c7400640 1 -- 172.21.15.117:0/285649237 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c000cd80 con 0x7fb1d0100570 2024-03-19T03:31:12.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.316+0000 7fb1c7400640 1 -- 172.21.15.117:0/285649237 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1c00035f0 con 0x7fb1d0100570 2024-03-19T03:31:12.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.316+0000 7fb1cfe00640 1 -- 172.21.15.117:0/285649237 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1d01a1a70 con 0x7fb1d0100570 2024-03-19T03:31:12.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.316+0000 7fb1c7400640 1 -- 172.21.15.117:0/285649237 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c0010690 con 0x7fb1d0100570 2024-03-19T03:31:12.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.316+0000 7fb1cfe00640 1 -- 172.21.15.117:0/285649237 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1d01a1f90 con 0x7fb1d0100570 2024-03-19T03:31:12.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.317+0000 7fb1cfe00640 1 -- 172.21.15.117:0/285649237 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1d0068530 con 0x7fb1d0100570 2024-03-19T03:31:12.319 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.319+0000 7fb1c7400640 1 -- 172.21.15.117:0/285649237 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fb1c00037a0 con 0x7fb1d0100570 2024-03-19T03:31:12.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.320+0000 7fb1c7400640 1 --2- 172.21.15.117:0/285649237 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fb1b40a2890 0x7fb1b40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:12.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.320+0000 7fb1c7400640 1 -- 172.21.15.117:0/285649237 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb1c00cf350 con 0x7fb1d0100570 2024-03-19T03:31:12.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.321+0000 7fb1ce000640 1 --2- 172.21.15.117:0/285649237 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fb1b40a2890 0x7fb1b40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:12.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.321+0000 7fb1ce000640 1 --2- 172.21.15.117:0/285649237 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fb1b40a2890 0x7fb1b40a4d50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc02c240 tx=0x7fb1bc02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:12.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.329+0000 7fb1c7400640 1 -- 172.21.15.117:0/285649237 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb1c009bff0 con 0x7fb1d0100570 2024-03-19T03:31:12.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:12 smithi178 ceph-mon[67578]: from='client.24863 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:31:12.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:12 smithi178 ceph-mon[67578]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-19T03:31:12.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:12 smithi178 ceph-mon[67578]: from='client.24847 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:31:12.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:12 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3969593586' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:31:12.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:12 smithi119 ceph-mon[66861]: from='client.24863 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:31:12.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:12 smithi119 ceph-mon[66861]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-19T03:31:12.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:12 smithi119 ceph-mon[66861]: from='client.24847 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:31:12.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:12 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3969593586' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:31:12.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.530+0000 7fb1cfe00640 1 -- 172.21.15.117:0/285649237 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb1d00fdc80 con 0x7fb1b40a2890 2024-03-19T03:31:12.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.552+0000 7fb1c7400640 1 -- 172.21.15.117:0/285649237 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340705 (secure 0 0 0) 0x7fb1d00fdc80 con 0x7fb1b40a2890 2024-03-19T03:31:12.558 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:31:12.560 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-03-19T03:31:12.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.558+0000 7fb1cfe00640 1 -- 172.21.15.117:0/285649237 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fb1b40a2890 msgr2=0x7fb1b40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:12.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.558+0000 7fb1cfe00640 1 --2- 172.21.15.117:0/285649237 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fb1b40a2890 0x7fb1b40a4d50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc02c240 tx=0x7fb1bc02df90 comp rx=0 tx=0).stop 2024-03-19T03:31:12.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.559+0000 7fb1cfe00640 1 -- 172.21.15.117:0/285649237 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb1d0100570 msgr2=0x7fb1d0104500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:12.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.559+0000 7fb1cfe00640 1 --2- 172.21.15.117:0/285649237 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb1d0100570 0x7fb1d0104500 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb1c000e9c0 tx=0x7fb1c000ee80 comp rx=0 tx=0).stop 2024-03-19T03:31:12.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.559+0000 7fb1cfe00640 1 -- 172.21.15.117:0/285649237 shutdown_connections 2024-03-19T03:31:12.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.559+0000 7fb1cfe00640 1 --2- 172.21.15.117:0/285649237 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1d0102120 0x7fb1d0103080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:12.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.559+0000 7fb1cfe00640 1 --2- 172.21.15.117:0/285649237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1d0101760 0x7fb1d0102b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:12.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.559+0000 7fb1cfe00640 1 --2- 172.21.15.117:0/285649237 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fb1b40a2890 0x7fb1b40a4d50 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:12.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.559+0000 7fb1cfe00640 1 --2- 172.21.15.117:0/285649237 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb1d0100570 0x7fb1d0104500 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:12.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.559+0000 7fb1cfe00640 1 -- 172.21.15.117:0/285649237 >> 172.21.15.117:0/285649237 conn(0x7fb1d00fbb00 msgr2=0x7fb1d00fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:12.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.559+0000 7fb1cfe00640 1 -- 172.21.15.117:0/285649237 shutdown_connections 2024-03-19T03:31:12.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.559+0000 7fb1cfe00640 1 -- 172.21.15.117:0/285649237 wait complete. 2024-03-19T03:31:12.579 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-03-19T03:31:12.291038+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":601558,"num_objects":97,"num_object_clones":0,"num_object_copies":293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":167,"num_read_kb":143,"num_write":387,"num_write_kb":3010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":62,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":318,"ondisk_log_size":318,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360988,"kb_used_data":11836,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580836,"statfs":{"total":1151940427776,"available":1151570776064,"internally_reserved":0,"allocated":12120064,"data_stored":7576042,"data_compressed":284628,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19138,"internal_metadata":357414206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6550,"num_objects":73,"num_object_clones":0,"num_object_copies":221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":157,"num_write_kb":53,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005702"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800069+0000","last_change":"2024-03-19T03:30:35.676095+0000","last_active":"2024-03-19T03:30:40.800069+0000","last_peered":"2024-03-19T03:30:40.800069+0000","last_clean":"2024-03-19T03:30:40.800069+0000","last_became_active":"2024-03-19T03:30:35.675772+0000","last_became_peered":"2024-03-19T03:30:35.675772+0000","last_unstale":"2024-03-19T03:30:40.800069+0000","last_undegraded":"2024-03-19T03:30:40.800069+0000","last_fullsized":"2024-03-19T03:30:40.800069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:12:31.902482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.729653+0000","last_change":"2024-03-19T03:30:35.672451+0000","last_active":"2024-03-19T03:30:40.729653+0000","last_peered":"2024-03-19T03:30:40.729653+0000","last_clean":"2024-03-19T03:30:40.729653+0000","last_became_active":"2024-03-19T03:30:35.671516+0000","last_became_peered":"2024-03-19T03:30:35.671516+0000","last_unstale":"2024-03-19T03:30:40.729653+0000","last_undegraded":"2024-03-19T03:30:40.729653+0000","last_fullsized":"2024-03-19T03:30:40.729653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:14:07.021704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.729332+0000","last_change":"2024-03-19T03:30:35.674425+0000","last_active":"2024-03-19T03:30:40.729332+0000","last_peered":"2024-03-19T03:30:40.729332+0000","last_clean":"2024-03-19T03:30:40.729332+0000","last_became_active":"2024-03-19T03:30:35.674012+0000","last_became_peered":"2024-03-19T03:30:35.674012+0000","last_unstale":"2024-03-19T03:30:40.729332+0000","last_undegraded":"2024-03-19T03:30:40.729332+0000","last_fullsized":"2024-03-19T03:30:40.729332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:51:44.850847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799603+0000","last_change":"2024-03-19T03:30:35.649218+0000","last_active":"2024-03-19T03:30:40.799603+0000","last_peered":"2024-03-19T03:30:40.799603+0000","last_clean":"2024-03-19T03:30:40.799603+0000","last_became_active":"2024-03-19T03:30:35.648848+0000","last_became_peered":"2024-03-19T03:30:35.648848+0000","last_unstale":"2024-03-19T03:30:40.799603+0000","last_undegraded":"2024-03-19T03:30:40.799603+0000","last_fullsized":"2024-03-19T03:30:40.799603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:24.204126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799449+0000","last_change":"2024-03-19T03:30:35.670208+0000","last_active":"2024-03-19T03:30:40.799449+0000","last_peered":"2024-03-19T03:30:40.799449+0000","last_clean":"2024-03-19T03:30:40.799449+0000","last_became_active":"2024-03-19T03:30:35.669939+0000","last_became_peered":"2024-03-19T03:30:35.669939+0000","last_unstale":"2024-03-19T03:30:40.799449+0000","last_undegraded":"2024-03-19T03:30:40.799449+0000","last_fullsized":"2024-03-19T03:30:40.799449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:59.687262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800396+0000","last_change":"2024-03-19T03:30:35.662329+0000","last_active":"2024-03-19T03:30:40.800396+0000","last_peered":"2024-03-19T03:30:40.800396+0000","last_clean":"2024-03-19T03:30:40.800396+0000","last_became_active":"2024-03-19T03:30:35.661969+0000","last_became_peered":"2024-03-19T03:30:35.661969+0000","last_unstale":"2024-03-19T03:30:40.800396+0000","last_undegraded":"2024-03-19T03:30:40.800396+0000","last_fullsized":"2024-03-19T03:30:40.800396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:29.265017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.723926+0000","last_change":"2024-03-19T03:30:35.662586+0000","last_active":"2024-03-19T03:30:40.723926+0000","last_peered":"2024-03-19T03:30:40.723926+0000","last_clean":"2024-03-19T03:30:40.723926+0000","last_became_active":"2024-03-19T03:30:35.661774+0000","last_became_peered":"2024-03-19T03:30:35.661774+0000","last_unstale":"2024-03-19T03:30:40.723926+0000","last_undegraded":"2024-03-19T03:30:40.723926+0000","last_fullsized":"2024-03-19T03:30:40.723926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:49.258178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800781+0000","last_change":"2024-03-19T03:30:35.638512+0000","last_active":"2024-03-19T03:30:40.800781+0000","last_peered":"2024-03-19T03:30:40.800781+0000","last_clean":"2024-03-19T03:30:40.800781+0000","last_became_active":"2024-03-19T03:30:35.637205+0000","last_became_peered":"2024-03-19T03:30:35.637205+0000","last_unstale":"2024-03-19T03:30:40.800781+0000","last_undegraded":"2024-03-19T03:30:40.800781+0000","last_fullsized":"2024-03-19T03:30:40.800781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:29:34.578520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799770+0000","last_change":"2024-03-19T03:30:35.650681+0000","last_active":"2024-03-19T03:30:40.799770+0000","last_peered":"2024-03-19T03:30:40.799770+0000","last_clean":"2024-03-19T03:30:40.799770+0000","last_became_active":"2024-03-19T03:30:35.650324+0000","last_became_peered":"2024-03-19T03:30:35.650324+0000","last_unstale":"2024-03-19T03:30:40.799770+0000","last_undegraded":"2024-03-19T03:30:40.799770+0000","last_fullsized":"2024-03-19T03:30:40.799770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:48:23.413530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800642+0000","last_change":"2024-03-19T03:30:35.650240+0000","last_active":"2024-03-19T03:30:40.800642+0000","last_peered":"2024-03-19T03:30:40.800642+0000","last_clean":"2024-03-19T03:30:40.800642+0000","last_became_active":"2024-03-19T03:30:35.649888+0000","last_became_peered":"2024-03-19T03:30:35.649888+0000","last_unstale":"2024-03-19T03:30:40.800642+0000","last_undegraded":"2024-03-19T03:30:40.800642+0000","last_fullsized":"2024-03-19T03:30:40.800642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:55:30.110289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.724094+0000","last_change":"2024-03-19T03:30:35.657058+0000","last_active":"2024-03-19T03:30:40.724094+0000","last_peered":"2024-03-19T03:30:40.724094+0000","last_clean":"2024-03-19T03:30:40.724094+0000","last_became_active":"2024-03-19T03:30:35.656678+0000","last_became_peered":"2024-03-19T03:30:35.656678+0000","last_unstale":"2024-03-19T03:30:40.724094+0000","last_undegraded":"2024-03-19T03:30:40.724094+0000","last_fullsized":"2024-03-19T03:30:40.724094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:10.211852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799551+0000","last_change":"2024-03-19T03:30:35.651543+0000","last_active":"2024-03-19T03:30:40.799551+0000","last_peered":"2024-03-19T03:30:40.799551+0000","last_clean":"2024-03-19T03:30:40.799551+0000","last_became_active":"2024-03-19T03:30:35.649514+0000","last_became_peered":"2024-03-19T03:30:35.649514+0000","last_unstale":"2024-03-19T03:30:40.799551+0000","last_undegraded":"2024-03-19T03:30:40.799551+0000","last_fullsized":"2024-03-19T03:30:40.799551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:54:37.607873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.801550+0000","last_change":"2024-03-19T03:30:35.662494+0000","last_active":"2024-03-19T03:30:40.801550+0000","last_peered":"2024-03-19T03:30:40.801550+0000","last_clean":"2024-03-19T03:30:40.801550+0000","last_became_active":"2024-03-19T03:30:35.660596+0000","last_became_peered":"2024-03-19T03:30:35.660596+0000","last_unstale":"2024-03-19T03:30:40.801550+0000","last_undegraded":"2024-03-19T03:30:40.801550+0000","last_fullsized":"2024-03-19T03:30:40.801550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:25:59.184025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.728184+0000","last_change":"2024-03-19T03:30:35.675751+0000","last_active":"2024-03-19T03:30:40.728184+0000","last_peered":"2024-03-19T03:30:40.728184+0000","last_clean":"2024-03-19T03:30:40.728184+0000","last_became_active":"2024-03-19T03:30:35.674313+0000","last_became_peered":"2024-03-19T03:30:35.674313+0000","last_unstale":"2024-03-19T03:30:40.728184+0000","last_undegraded":"2024-03-19T03:30:40.728184+0000","last_fullsized":"2024-03-19T03:30:40.728184+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:26:26.433958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.728403+0000","last_change":"2024-03-19T03:30:35.672628+0000","last_active":"2024-03-19T03:30:40.728403+0000","last_peered":"2024-03-19T03:30:40.728403+0000","last_clean":"2024-03-19T03:30:40.728403+0000","last_became_active":"2024-03-19T03:30:35.672246+0000","last_became_peered":"2024-03-19T03:30:35.672246+0000","last_unstale":"2024-03-19T03:30:40.728403+0000","last_undegraded":"2024-03-19T03:30:40.728403+0000","last_fullsized":"2024-03-19T03:30:40.728403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:07:59.891108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.798739+0000","last_change":"2024-03-19T03:30:35.678718+0000","last_active":"2024-03-19T03:30:40.798739+0000","last_peered":"2024-03-19T03:30:40.798739+0000","last_clean":"2024-03-19T03:30:40.798739+0000","last_became_active":"2024-03-19T03:30:35.678461+0000","last_became_peered":"2024-03-19T03:30:35.678461+0000","last_unstale":"2024-03-19T03:30:40.798739+0000","last_undegraded":"2024-03-19T03:30:40.798739+0000","last_fullsized":"2024-03-19T03:30:40.798739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:19:39.079048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799956+0000","last_change":"2024-03-19T03:30:35.675847+0000","last_active":"2024-03-19T03:30:40.799956+0000","last_peered":"2024-03-19T03:30:40.799956+0000","last_clean":"2024-03-19T03:30:40.799956+0000","last_became_active":"2024-03-19T03:30:35.675204+0000","last_became_peered":"2024-03-19T03:30:35.675204+0000","last_unstale":"2024-03-19T03:30:40.799956+0000","last_undegraded":"2024-03-19T03:30:40.799956+0000","last_fullsized":"2024-03-19T03:30:40.799956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:08.732536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.802017+0000","last_change":"2024-03-19T03:30:35.663110+0000","last_active":"2024-03-19T03:30:40.802017+0000","last_peered":"2024-03-19T03:30:40.802017+0000","last_clean":"2024-03-19T03:30:40.802017+0000","last_became_active":"2024-03-19T03:30:35.658376+0000","last_became_peered":"2024-03-19T03:30:35.658376+0000","last_unstale":"2024-03-19T03:30:40.802017+0000","last_undegraded":"2024-03-19T03:30:40.802017+0000","last_fullsized":"2024-03-19T03:30:40.802017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:29:44.658681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.724720+0000","last_change":"2024-03-19T03:30:35.661154+0000","last_active":"2024-03-19T03:30:40.724720+0000","last_peered":"2024-03-19T03:30:40.724720+0000","last_clean":"2024-03-19T03:30:40.724720+0000","last_became_active":"2024-03-19T03:30:35.660794+0000","last_became_peered":"2024-03-19T03:30:35.660794+0000","last_unstale":"2024-03-19T03:30:40.724720+0000","last_undegraded":"2024-03-19T03:30:40.724720+0000","last_fullsized":"2024-03-19T03:30:40.724720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:26:29.803827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.724176+0000","last_change":"2024-03-19T03:30:39.735971+0000","last_active":"2024-03-19T03:30:40.724176+0000","last_peered":"2024-03-19T03:30:40.724176+0000","last_clean":"2024-03-19T03:30:40.724176+0000","last_became_active":"2024-03-19T03:30:39.735585+0000","last_became_peered":"2024-03-19T03:30:39.735585+0000","last_unstale":"2024-03-19T03:30:40.724176+0000","last_undegraded":"2024-03-19T03:30:40.724176+0000","last_fullsized":"2024-03-19T03:30:40.724176+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:30:15.969148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:10.003637+0000","last_change":"2024-03-19T03:30:34.632177+0000","last_active":"2024-03-19T03:31:10.003637+0000","last_peered":"2024-03-19T03:31:10.003637+0000","last_clean":"2024-03-19T03:31:10.003637+0000","last_became_active":"2024-03-19T03:30:34.631836+0000","last_became_peered":"2024-03-19T03:30:34.631836+0000","last_unstale":"2024-03-19T03:31:10.003637+0000","last_undegraded":"2024-03-19T03:31:10.003637+0000","last_fullsized":"2024-03-19T03:31:10.003637+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:15:42.883089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.803099+0000","last_change":"2024-03-19T03:30:35.654203+0000","last_active":"2024-03-19T03:30:40.803099+0000","last_peered":"2024-03-19T03:30:40.803099+0000","last_clean":"2024-03-19T03:30:40.803099+0000","last_became_active":"2024-03-19T03:30:35.653665+0000","last_became_peered":"2024-03-19T03:30:35.653665+0000","last_unstale":"2024-03-19T03:30:40.803099+0000","last_undegraded":"2024-03-19T03:30:40.803099+0000","last_fullsized":"2024-03-19T03:30:40.803099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:06:49.153811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799196+0000","last_change":"2024-03-19T03:30:39.733744+0000","last_active":"2024-03-19T03:30:40.799196+0000","last_peered":"2024-03-19T03:30:40.799196+0000","last_clean":"2024-03-19T03:30:40.799196+0000","last_became_active":"2024-03-19T03:30:39.733373+0000","last_became_peered":"2024-03-19T03:30:39.733373+0000","last_unstale":"2024-03-19T03:30:40.799196+0000","last_undegraded":"2024-03-19T03:30:40.799196+0000","last_fullsized":"2024-03-19T03:30:40.799196+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:48:48.186995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.803237+0000","last_change":"2024-03-19T03:30:34.632369+0000","last_active":"2024-03-19T03:30:40.803237+0000","last_peered":"2024-03-19T03:30:40.803237+0000","last_clean":"2024-03-19T03:30:40.803237+0000","last_became_active":"2024-03-19T03:30:34.632027+0000","last_became_peered":"2024-03-19T03:30:34.632027+0000","last_unstale":"2024-03-19T03:30:40.803237+0000","last_undegraded":"2024-03-19T03:30:40.803237+0000","last_fullsized":"2024-03-19T03:30:40.803237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:04:22.575864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.729798+0000","last_change":"2024-03-19T03:30:35.677250+0000","last_active":"2024-03-19T03:30:40.729798+0000","last_peered":"2024-03-19T03:30:40.729798+0000","last_clean":"2024-03-19T03:30:40.729798+0000","last_became_active":"2024-03-19T03:30:35.676741+0000","last_became_peered":"2024-03-19T03:30:35.676741+0000","last_unstale":"2024-03-19T03:30:40.729798+0000","last_undegraded":"2024-03-19T03:30:40.729798+0000","last_fullsized":"2024-03-19T03:30:40.729798+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:52:06.095738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799273+0000","last_change":"2024-03-19T03:30:39.733501+0000","last_active":"2024-03-19T03:30:40.799273+0000","last_peered":"2024-03-19T03:30:40.799273+0000","last_clean":"2024-03-19T03:30:40.799273+0000","last_became_active":"2024-03-19T03:30:39.733102+0000","last_became_peered":"2024-03-19T03:30:39.733102+0000","last_unstale":"2024-03-19T03:30:40.799273+0000","last_undegraded":"2024-03-19T03:30:40.799273+0000","last_fullsized":"2024-03-19T03:30:40.799273+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:35:06.004381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:09.003240+0000","last_change":"2024-03-19T03:30:34.627192+0000","last_active":"2024-03-19T03:31:09.003240+0000","last_peered":"2024-03-19T03:31:09.003240+0000","last_clean":"2024-03-19T03:31:09.003240+0000","last_became_active":"2024-03-19T03:30:34.626730+0000","last_became_peered":"2024-03-19T03:30:34.626730+0000","last_unstale":"2024-03-19T03:31:09.003240+0000","last_undegraded":"2024-03-19T03:31:09.003240+0000","last_fullsized":"2024-03-19T03:31:09.003240+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:20:25.816500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799710+0000","last_change":"2024-03-19T03:30:35.663783+0000","last_active":"2024-03-19T03:30:40.799710+0000","last_peered":"2024-03-19T03:30:40.799710+0000","last_clean":"2024-03-19T03:30:40.799710+0000","last_became_active":"2024-03-19T03:30:35.663472+0000","last_became_peered":"2024-03-19T03:30:35.663472+0000","last_unstale":"2024-03-19T03:30:40.799710+0000","last_undegraded":"2024-03-19T03:30:40.799710+0000","last_fullsized":"2024-03-19T03:30:40.799710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:51:15.211113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799308+0000","last_change":"2024-03-19T03:30:39.731492+0000","last_active":"2024-03-19T03:30:40.799308+0000","last_peered":"2024-03-19T03:30:40.799308+0000","last_clean":"2024-03-19T03:30:40.799308+0000","last_became_active":"2024-03-19T03:30:39.731168+0000","last_became_peered":"2024-03-19T03:30:39.731168+0000","last_unstale":"2024-03-19T03:30:40.799308+0000","last_undegraded":"2024-03-19T03:30:40.799308+0000","last_fullsized":"2024-03-19T03:30:40.799308+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:11:48.281607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:09.003327+0000","last_change":"2024-03-19T03:30:34.627589+0000","last_active":"2024-03-19T03:31:09.003327+0000","last_peered":"2024-03-19T03:31:09.003327+0000","last_clean":"2024-03-19T03:31:09.003327+0000","last_became_active":"2024-03-19T03:30:34.627229+0000","last_became_peered":"2024-03-19T03:30:34.627229+0000","last_unstale":"2024-03-19T03:31:09.003327+0000","last_undegraded":"2024-03-19T03:31:09.003327+0000","last_fullsized":"2024-03-19T03:31:09.003327+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:53:23.128932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800936+0000","last_change":"2024-03-19T03:30:35.650954+0000","last_active":"2024-03-19T03:30:40.800936+0000","last_peered":"2024-03-19T03:30:40.800936+0000","last_clean":"2024-03-19T03:30:40.800936+0000","last_became_active":"2024-03-19T03:30:35.650268+0000","last_became_peered":"2024-03-19T03:30:35.650268+0000","last_unstale":"2024-03-19T03:30:40.800936+0000","last_undegraded":"2024-03-19T03:30:40.800936+0000","last_fullsized":"2024-03-19T03:30:40.800936+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T15:22:28.132438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.724455+0000","last_change":"2024-03-19T03:30:39.727351+0000","last_active":"2024-03-19T03:30:40.724455+0000","last_peered":"2024-03-19T03:30:40.724455+0000","last_clean":"2024-03-19T03:30:40.724455+0000","last_became_active":"2024-03-19T03:30:39.726868+0000","last_became_peered":"2024-03-19T03:30:39.726868+0000","last_unstale":"2024-03-19T03:30:40.724455+0000","last_undegraded":"2024-03-19T03:30:40.724455+0000","last_fullsized":"2024-03-19T03:30:40.724455+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:40:59.003064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:08.996425+0000","last_change":"2024-03-19T03:30:34.623911+0000","last_active":"2024-03-19T03:31:08.996425+0000","last_peered":"2024-03-19T03:31:08.996425+0000","last_clean":"2024-03-19T03:31:08.996425+0000","last_became_active":"2024-03-19T03:30:34.623548+0000","last_became_peered":"2024-03-19T03:30:34.623548+0000","last_unstale":"2024-03-19T03:31:08.996425+0000","last_undegraded":"2024-03-19T03:31:08.996425+0000","last_fullsized":"2024-03-19T03:31:08.996425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:29:56.616353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799771+0000","last_change":"2024-03-19T03:30:35.661874+0000","last_active":"2024-03-19T03:30:40.799771+0000","last_peered":"2024-03-19T03:30:40.799771+0000","last_clean":"2024-03-19T03:30:40.799771+0000","last_became_active":"2024-03-19T03:30:35.661521+0000","last_became_peered":"2024-03-19T03:30:35.661521+0000","last_unstale":"2024-03-19T03:30:40.799771+0000","last_undegraded":"2024-03-19T03:30:40.799771+0000","last_fullsized":"2024-03-19T03:30:40.799771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:19:35.017020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.724476+0000","last_change":"2024-03-19T03:30:39.748197+0000","last_active":"2024-03-19T03:30:40.724476+0000","last_peered":"2024-03-19T03:30:40.724476+0000","last_clean":"2024-03-19T03:30:40.724476+0000","last_became_active":"2024-03-19T03:30:39.747833+0000","last_became_peered":"2024-03-19T03:30:39.747833+0000","last_unstale":"2024-03-19T03:30:40.724476+0000","last_undegraded":"2024-03-19T03:30:40.724476+0000","last_fullsized":"2024-03-19T03:30:40.724476+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:29:19.213403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:07.477261+0000","last_change":"2024-03-19T03:30:34.623491+0000","last_active":"2024-03-19T03:31:07.477261+0000","last_peered":"2024-03-19T03:31:07.477261+0000","last_clean":"2024-03-19T03:31:07.477261+0000","last_became_active":"2024-03-19T03:30:34.623152+0000","last_became_peered":"2024-03-19T03:30:34.623152+0000","last_unstale":"2024-03-19T03:31:07.477261+0000","last_undegraded":"2024-03-19T03:31:07.477261+0000","last_fullsized":"2024-03-19T03:31:07.477261+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:00:25.530926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800823+0000","last_change":"2024-03-19T03:30:35.651528+0000","last_active":"2024-03-19T03:30:40.800823+0000","last_peered":"2024-03-19T03:30:40.800823+0000","last_clean":"2024-03-19T03:30:40.800823+0000","last_became_active":"2024-03-19T03:30:35.650240+0000","last_became_peered":"2024-03-19T03:30:35.650240+0000","last_unstale":"2024-03-19T03:30:40.800823+0000","last_undegraded":"2024-03-19T03:30:40.800823+0000","last_fullsized":"2024-03-19T03:30:40.800823+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:25.123570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799874+0000","last_change":"2024-03-19T03:30:39.740630+0000","last_active":"2024-03-19T03:30:40.799874+0000","last_peered":"2024-03-19T03:30:40.799874+0000","last_clean":"2024-03-19T03:30:40.799874+0000","last_became_active":"2024-03-19T03:30:39.740170+0000","last_became_peered":"2024-03-19T03:30:39.740170+0000","last_unstale":"2024-03-19T03:30:40.799874+0000","last_undegraded":"2024-03-19T03:30:40.799874+0000","last_fullsized":"2024-03-19T03:30:40.799874+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:23:49.415718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:07.483948+0000","last_change":"2024-03-19T03:30:34.635173+0000","last_active":"2024-03-19T03:31:07.483948+0000","last_peered":"2024-03-19T03:31:07.483948+0000","last_clean":"2024-03-19T03:31:07.483948+0000","last_became_active":"2024-03-19T03:30:34.633809+0000","last_became_peered":"2024-03-19T03:30:34.633809+0000","last_unstale":"2024-03-19T03:31:07.483948+0000","last_undegraded":"2024-03-19T03:31:07.483948+0000","last_fullsized":"2024-03-19T03:31:07.483948+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:03:57.409707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800659+0000","last_change":"2024-03-19T03:30:35.665660+0000","last_active":"2024-03-19T03:30:40.800659+0000","last_peered":"2024-03-19T03:30:40.800659+0000","last_clean":"2024-03-19T03:30:40.800659+0000","last_became_active":"2024-03-19T03:30:35.665290+0000","last_became_peered":"2024-03-19T03:30:35.665290+0000","last_unstale":"2024-03-19T03:30:40.800659+0000","last_undegraded":"2024-03-19T03:30:40.800659+0000","last_fullsized":"2024-03-19T03:30:40.800659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:20.333009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.729721+0000","last_change":"2024-03-19T03:30:39.744391+0000","last_active":"2024-03-19T03:30:40.729721+0000","last_peered":"2024-03-19T03:30:40.729721+0000","last_clean":"2024-03-19T03:30:40.729721+0000","last_became_active":"2024-03-19T03:30:39.744000+0000","last_became_peered":"2024-03-19T03:30:39.744000+0000","last_unstale":"2024-03-19T03:30:40.729721+0000","last_undegraded":"2024-03-19T03:30:40.729721+0000","last_fullsized":"2024-03-19T03:30:40.729721+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:17:06.124286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:08.994577+0000","last_change":"2024-03-19T03:30:34.621811+0000","last_active":"2024-03-19T03:31:08.994577+0000","last_peered":"2024-03-19T03:31:08.994577+0000","last_clean":"2024-03-19T03:31:08.994577+0000","last_became_active":"2024-03-19T03:30:34.621308+0000","last_became_peered":"2024-03-19T03:30:34.621308+0000","last_unstale":"2024-03-19T03:31:08.994577+0000","last_undegraded":"2024-03-19T03:31:08.994577+0000","last_fullsized":"2024-03-19T03:31:08.994577+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:36:45.895720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.802438+0000","last_change":"2024-03-19T03:30:35.673683+0000","last_active":"2024-03-19T03:30:40.802438+0000","last_peered":"2024-03-19T03:30:40.802438+0000","last_clean":"2024-03-19T03:30:40.802438+0000","last_became_active":"2024-03-19T03:30:35.670489+0000","last_became_peered":"2024-03-19T03:30:35.670489+0000","last_unstale":"2024-03-19T03:30:40.802438+0000","last_undegraded":"2024-03-19T03:30:40.802438+0000","last_fullsized":"2024-03-19T03:30:40.802438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:48:30.521454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.801202+0000","last_change":"2024-03-19T03:30:39.726358+0000","last_active":"2024-03-19T03:30:40.801202+0000","last_peered":"2024-03-19T03:30:40.801202+0000","last_clean":"2024-03-19T03:30:40.801202+0000","last_became_active":"2024-03-19T03:30:39.725720+0000","last_became_peered":"2024-03-19T03:30:39.725720+0000","last_unstale":"2024-03-19T03:30:40.801202+0000","last_undegraded":"2024-03-19T03:30:40.801202+0000","last_fullsized":"2024-03-19T03:30:40.801202+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:01:08.759794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:06.474278+0000","last_change":"2024-03-19T03:30:34.626336+0000","last_active":"2024-03-19T03:31:06.474278+0000","last_peered":"2024-03-19T03:31:06.474278+0000","last_clean":"2024-03-19T03:31:06.474278+0000","last_became_active":"2024-03-19T03:30:34.625991+0000","last_became_peered":"2024-03-19T03:30:34.625991+0000","last_unstale":"2024-03-19T03:31:06.474278+0000","last_undegraded":"2024-03-19T03:31:06.474278+0000","last_fullsized":"2024-03-19T03:31:06.474278+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:40:58.791759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799988+0000","last_change":"2024-03-19T03:30:35.667001+0000","last_active":"2024-03-19T03:30:40.799988+0000","last_peered":"2024-03-19T03:30:40.799988+0000","last_clean":"2024-03-19T03:30:40.799988+0000","last_became_active":"2024-03-19T03:30:35.666586+0000","last_became_peered":"2024-03-19T03:30:35.666586+0000","last_unstale":"2024-03-19T03:30:40.799988+0000","last_undegraded":"2024-03-19T03:30:40.799988+0000","last_fullsized":"2024-03-19T03:30:40.799988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:40:30.100860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.802488+0000","last_change":"2024-03-19T03:30:39.745267+0000","last_active":"2024-03-19T03:30:40.802488+0000","last_peered":"2024-03-19T03:30:40.802488+0000","last_clean":"2024-03-19T03:30:40.802488+0000","last_became_active":"2024-03-19T03:30:39.743434+0000","last_became_peered":"2024-03-19T03:30:39.743434+0000","last_unstale":"2024-03-19T03:30:40.802488+0000","last_undegraded":"2024-03-19T03:30:40.802488+0000","last_fullsized":"2024-03-19T03:30:40.802488+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:53:37.583886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:09.001673+0000","last_change":"2024-03-19T03:30:34.617769+0000","last_active":"2024-03-19T03:31:09.001673+0000","last_peered":"2024-03-19T03:31:09.001673+0000","last_clean":"2024-03-19T03:31:09.001673+0000","last_became_active":"2024-03-19T03:30:34.617428+0000","last_became_peered":"2024-03-19T03:30:34.617428+0000","last_unstale":"2024-03-19T03:31:09.001673+0000","last_undegraded":"2024-03-19T03:31:09.001673+0000","last_fullsized":"2024-03-19T03:31:09.001673+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:58:09.181320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.729801+0000","last_change":"2024-03-19T03:30:35.676278+0000","last_active":"2024-03-19T03:30:40.729801+0000","last_peered":"2024-03-19T03:30:40.729801+0000","last_clean":"2024-03-19T03:30:40.729801+0000","last_became_active":"2024-03-19T03:30:35.673773+0000","last_became_peered":"2024-03-19T03:30:35.673773+0000","last_unstale":"2024-03-19T03:30:40.729801+0000","last_undegraded":"2024-03-19T03:30:40.729801+0000","last_fullsized":"2024-03-19T03:30:40.729801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:39:12.983732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800083+0000","last_change":"2024-03-19T03:30:39.733273+0000","last_active":"2024-03-19T03:30:40.800083+0000","last_peered":"2024-03-19T03:30:40.800083+0000","last_clean":"2024-03-19T03:30:40.800083+0000","last_became_active":"2024-03-19T03:30:39.732885+0000","last_became_peered":"2024-03-19T03:30:39.732885+0000","last_unstale":"2024-03-19T03:30:40.800083+0000","last_undegraded":"2024-03-19T03:30:40.800083+0000","last_fullsized":"2024-03-19T03:30:40.800083+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:48:10.446577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:08.494914+0000","last_change":"2024-03-19T03:30:34.637076+0000","last_active":"2024-03-19T03:31:08.494914+0000","last_peered":"2024-03-19T03:31:08.494914+0000","last_clean":"2024-03-19T03:31:08.494914+0000","last_became_active":"2024-03-19T03:30:34.636642+0000","last_became_peered":"2024-03-19T03:30:34.636642+0000","last_unstale":"2024-03-19T03:31:08.494914+0000","last_undegraded":"2024-03-19T03:31:08.494914+0000","last_fullsized":"2024-03-19T03:31:08.494914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:26:17.494081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799830+0000","last_change":"2024-03-19T03:30:35.651739+0000","last_active":"2024-03-19T03:30:40.799830+0000","last_peered":"2024-03-19T03:30:40.799830+0000","last_clean":"2024-03-19T03:30:40.799830+0000","last_became_active":"2024-03-19T03:30:35.651419+0000","last_became_peered":"2024-03-19T03:30:35.651419+0000","last_unstale":"2024-03-19T03:30:40.799830+0000","last_undegraded":"2024-03-19T03:30:40.799830+0000","last_fullsized":"2024-03-19T03:30:40.799830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T15:17:55.022050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800590+0000","last_change":"2024-03-19T03:30:39.733228+0000","last_active":"2024-03-19T03:30:40.800590+0000","last_peered":"2024-03-19T03:30:40.800590+0000","last_clean":"2024-03-19T03:30:40.800590+0000","last_became_active":"2024-03-19T03:30:39.732819+0000","last_became_peered":"2024-03-19T03:30:39.732819+0000","last_unstale":"2024-03-19T03:30:40.800590+0000","last_undegraded":"2024-03-19T03:30:40.800590+0000","last_fullsized":"2024-03-19T03:30:40.800590+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:03:43.968209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:06.469000+0000","last_change":"2024-03-19T03:30:34.618323+0000","last_active":"2024-03-19T03:31:06.469000+0000","last_peered":"2024-03-19T03:31:06.469000+0000","last_clean":"2024-03-19T03:31:06.469000+0000","last_became_active":"2024-03-19T03:30:34.617975+0000","last_became_peered":"2024-03-19T03:30:34.617975+0000","last_unstale":"2024-03-19T03:31:06.469000+0000","last_undegraded":"2024-03-19T03:31:06.469000+0000","last_fullsized":"2024-03-19T03:31:06.469000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:37:15.261998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800119+0000","last_change":"2024-03-19T03:30:35.652754+0000","last_active":"2024-03-19T03:30:40.800119+0000","last_peered":"2024-03-19T03:30:40.800119+0000","last_clean":"2024-03-19T03:30:40.800119+0000","last_became_active":"2024-03-19T03:30:35.652415+0000","last_became_peered":"2024-03-19T03:30:35.652415+0000","last_unstale":"2024-03-19T03:30:40.800119+0000","last_undegraded":"2024-03-19T03:30:40.800119+0000","last_fullsized":"2024-03-19T03:30:40.800119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:29:31.838302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.801350+0000","last_change":"2024-03-19T03:30:39.732885+0000","last_active":"2024-03-19T03:30:40.801350+0000","last_peered":"2024-03-19T03:30:40.801350+0000","last_clean":"2024-03-19T03:30:40.801350+0000","last_became_active":"2024-03-19T03:30:39.732449+0000","last_became_peered":"2024-03-19T03:30:39.732449+0000","last_unstale":"2024-03-19T03:30:40.801350+0000","last_undegraded":"2024-03-19T03:30:40.801350+0000","last_fullsized":"2024-03-19T03:30:40.801350+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:08:01.304805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:09.007040+0000","last_change":"2024-03-19T03:30:34.617560+0000","last_active":"2024-03-19T03:31:09.007040+0000","last_peered":"2024-03-19T03:31:09.007040+0000","last_clean":"2024-03-19T03:31:09.007040+0000","last_became_active":"2024-03-19T03:30:34.617234+0000","last_became_peered":"2024-03-19T03:30:34.617234+0000","last_unstale":"2024-03-19T03:31:09.007040+0000","last_undegraded":"2024-03-19T03:31:09.007040+0000","last_fullsized":"2024-03-19T03:31:09.007040+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:00:46.206589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800748+0000","last_change":"2024-03-19T03:30:35.672723+0000","last_active":"2024-03-19T03:30:40.800748+0000","last_peered":"2024-03-19T03:30:40.800748+0000","last_clean":"2024-03-19T03:30:40.800748+0000","last_became_active":"2024-03-19T03:30:35.670828+0000","last_became_peered":"2024-03-19T03:30:35.670828+0000","last_unstale":"2024-03-19T03:30:40.800748+0000","last_undegraded":"2024-03-19T03:30:40.800748+0000","last_fullsized":"2024-03-19T03:30:40.800748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:22:19.254329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800833+0000","last_change":"2024-03-19T03:30:39.738887+0000","last_active":"2024-03-19T03:30:40.800833+0000","last_peered":"2024-03-19T03:30:40.800833+0000","last_clean":"2024-03-19T03:30:40.800833+0000","last_became_active":"2024-03-19T03:30:39.738450+0000","last_became_peered":"2024-03-19T03:30:39.738450+0000","last_unstale":"2024-03-19T03:30:40.800833+0000","last_undegraded":"2024-03-19T03:30:40.800833+0000","last_fullsized":"2024-03-19T03:30:40.800833+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:23:57.550464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:09.007078+0000","last_change":"2024-03-19T03:30:34.634810+0000","last_active":"2024-03-19T03:31:09.007078+0000","last_peered":"2024-03-19T03:31:09.007078+0000","last_clean":"2024-03-19T03:31:09.007078+0000","last_became_active":"2024-03-19T03:30:34.634404+0000","last_became_peered":"2024-03-19T03:30:34.634404+0000","last_unstale":"2024-03-19T03:31:09.007078+0000","last_undegraded":"2024-03-19T03:31:09.007078+0000","last_fullsized":"2024-03-19T03:31:09.007078+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:19:09.231175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.723928+0000","last_change":"2024-03-19T03:30:35.664808+0000","last_active":"2024-03-19T03:30:40.723928+0000","last_peered":"2024-03-19T03:30:40.723928+0000","last_clean":"2024-03-19T03:30:40.723928+0000","last_became_active":"2024-03-19T03:30:35.664431+0000","last_became_peered":"2024-03-19T03:30:35.664431+0000","last_unstale":"2024-03-19T03:30:40.723928+0000","last_undegraded":"2024-03-19T03:30:40.723928+0000","last_fullsized":"2024-03-19T03:30:40.723928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:18:47.623528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.801170+0000","last_change":"2024-03-19T03:30:39.736905+0000","last_active":"2024-03-19T03:30:40.801170+0000","last_peered":"2024-03-19T03:30:40.801170+0000","last_clean":"2024-03-19T03:30:40.801170+0000","last_became_active":"2024-03-19T03:30:39.736329+0000","last_became_peered":"2024-03-19T03:30:39.736329+0000","last_unstale":"2024-03-19T03:30:40.801170+0000","last_undegraded":"2024-03-19T03:30:40.801170+0000","last_fullsized":"2024-03-19T03:30:40.801170+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:12:44.745999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:10.001301+0000","last_change":"2024-03-19T03:30:34.619890+0000","last_active":"2024-03-19T03:31:10.001301+0000","last_peered":"2024-03-19T03:31:10.001301+0000","last_clean":"2024-03-19T03:31:10.001301+0000","last_became_active":"2024-03-19T03:30:34.619518+0000","last_became_peered":"2024-03-19T03:30:34.619518+0000","last_unstale":"2024-03-19T03:31:10.001301+0000","last_undegraded":"2024-03-19T03:31:10.001301+0000","last_fullsized":"2024-03-19T03:31:10.001301+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T15:19:41.314321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.723621+0000","last_change":"2024-03-19T03:30:35.649241+0000","last_active":"2024-03-19T03:30:40.723621+0000","last_peered":"2024-03-19T03:30:40.723621+0000","last_clean":"2024-03-19T03:30:40.723621+0000","last_became_active":"2024-03-19T03:30:35.648883+0000","last_became_peered":"2024-03-19T03:30:35.648883+0000","last_unstale":"2024-03-19T03:30:40.723621+0000","last_undegraded":"2024-03-19T03:30:40.723621+0000","last_fullsized":"2024-03-19T03:30:40.723621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:43:44.219322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.801794+0000","last_change":"2024-03-19T03:30:39.741401+0000","last_active":"2024-03-19T03:30:40.801794+0000","last_peered":"2024-03-19T03:30:40.801794+0000","last_clean":"2024-03-19T03:30:40.801794+0000","last_became_active":"2024-03-19T03:30:39.741019+0000","last_became_peered":"2024-03-19T03:30:39.741019+0000","last_unstale":"2024-03-19T03:30:40.801794+0000","last_undegraded":"2024-03-19T03:30:40.801794+0000","last_fullsized":"2024-03-19T03:30:40.801794+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:29:12.824867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:08.990202+0000","last_change":"2024-03-19T03:30:34.635076+0000","last_active":"2024-03-19T03:31:08.990202+0000","last_peered":"2024-03-19T03:31:08.990202+0000","last_clean":"2024-03-19T03:31:08.990202+0000","last_became_active":"2024-03-19T03:30:34.634680+0000","last_became_peered":"2024-03-19T03:30:34.634680+0000","last_unstale":"2024-03-19T03:31:08.990202+0000","last_undegraded":"2024-03-19T03:31:08.990202+0000","last_fullsized":"2024-03-19T03:31:08.990202+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:16:38.874086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799993+0000","last_change":"2024-03-19T03:30:35.651912+0000","last_active":"2024-03-19T03:30:40.799993+0000","last_peered":"2024-03-19T03:30:40.799993+0000","last_clean":"2024-03-19T03:30:40.799993+0000","last_became_active":"2024-03-19T03:30:35.651563+0000","last_became_peered":"2024-03-19T03:30:35.651563+0000","last_unstale":"2024-03-19T03:30:40.799993+0000","last_undegraded":"2024-03-19T03:30:40.799993+0000","last_fullsized":"2024-03-19T03:30:40.799993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:41:05.677230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799949+0000","last_change":"2024-03-19T03:30:39.737466+0000","last_active":"2024-03-19T03:30:40.799949+0000","last_peered":"2024-03-19T03:30:40.799949+0000","last_clean":"2024-03-19T03:30:40.799949+0000","last_became_active":"2024-03-19T03:30:39.737126+0000","last_became_peered":"2024-03-19T03:30:39.737126+0000","last_unstale":"2024-03-19T03:30:40.799949+0000","last_undegraded":"2024-03-19T03:30:40.799949+0000","last_fullsized":"2024-03-19T03:30:40.799949+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:52:25.030107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:08.474804+0000","last_change":"2024-03-19T03:30:34.622729+0000","last_active":"2024-03-19T03:31:08.474804+0000","last_peered":"2024-03-19T03:31:08.474804+0000","last_clean":"2024-03-19T03:31:08.474804+0000","last_became_active":"2024-03-19T03:30:34.622349+0000","last_became_peered":"2024-03-19T03:30:34.622349+0000","last_unstale":"2024-03-19T03:31:08.474804+0000","last_undegraded":"2024-03-19T03:31:08.474804+0000","last_fullsized":"2024-03-19T03:31:08.474804+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:28.893702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.801057+0000","last_change":"2024-03-19T03:30:35.654167+0000","last_active":"2024-03-19T03:30:40.801057+0000","last_peered":"2024-03-19T03:30:40.801057+0000","last_clean":"2024-03-19T03:30:40.801057+0000","last_became_active":"2024-03-19T03:30:35.653782+0000","last_became_peered":"2024-03-19T03:30:35.653782+0000","last_unstale":"2024-03-19T03:30:40.801057+0000","last_undegraded":"2024-03-19T03:30:40.801057+0000","last_fullsized":"2024-03-19T03:30:40.801057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:26:47.584353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799710+0000","last_change":"2024-03-19T03:30:39.728505+0000","last_active":"2024-03-19T03:30:40.799710+0000","last_peered":"2024-03-19T03:30:40.799710+0000","last_clean":"2024-03-19T03:30:40.799710+0000","last_became_active":"2024-03-19T03:30:39.728095+0000","last_became_peered":"2024-03-19T03:30:39.728095+0000","last_unstale":"2024-03-19T03:30:40.799710+0000","last_undegraded":"2024-03-19T03:30:40.799710+0000","last_fullsized":"2024-03-19T03:30:40.799710+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:59:44.189992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.802826+0000","last_change":"2024-03-19T03:30:34.642738+0000","last_active":"2024-03-19T03:30:40.802826+0000","last_peered":"2024-03-19T03:30:40.802826+0000","last_clean":"2024-03-19T03:30:40.802826+0000","last_became_active":"2024-03-19T03:30:34.642173+0000","last_became_peered":"2024-03-19T03:30:34.642173+0000","last_unstale":"2024-03-19T03:30:40.802826+0000","last_undegraded":"2024-03-19T03:30:40.802826+0000","last_fullsized":"2024-03-19T03:30:40.802826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:17:51.633284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800650+0000","last_change":"2024-03-19T03:30:35.663676+0000","last_active":"2024-03-19T03:30:40.800650+0000","last_peered":"2024-03-19T03:30:40.800650+0000","last_clean":"2024-03-19T03:30:40.800650+0000","last_became_active":"2024-03-19T03:30:35.663313+0000","last_became_peered":"2024-03-19T03:30:35.663313+0000","last_unstale":"2024-03-19T03:30:40.800650+0000","last_undegraded":"2024-03-19T03:30:40.800650+0000","last_fullsized":"2024-03-19T03:30:40.800650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:02:25.933784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.801529+0000","last_change":"2024-03-19T03:30:39.744967+0000","last_active":"2024-03-19T03:30:40.801529+0000","last_peered":"2024-03-19T03:30:40.801529+0000","last_clean":"2024-03-19T03:30:40.801529+0000","last_became_active":"2024-03-19T03:30:39.744542+0000","last_became_peered":"2024-03-19T03:30:39.744542+0000","last_unstale":"2024-03-19T03:30:40.801529+0000","last_undegraded":"2024-03-19T03:30:40.801529+0000","last_fullsized":"2024-03-19T03:30:40.801529+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:56:54.699749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:10.004351+0000","last_change":"2024-03-19T03:30:34.622225+0000","last_active":"2024-03-19T03:31:10.004351+0000","last_peered":"2024-03-19T03:31:10.004351+0000","last_clean":"2024-03-19T03:31:10.004351+0000","last_became_active":"2024-03-19T03:30:34.621836+0000","last_became_peered":"2024-03-19T03:30:34.621836+0000","last_unstale":"2024-03-19T03:31:10.004351+0000","last_undegraded":"2024-03-19T03:31:10.004351+0000","last_fullsized":"2024-03-19T03:31:10.004351+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:20:35.854632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800182+0000","last_change":"2024-03-19T03:30:35.675961+0000","last_active":"2024-03-19T03:30:40.800182+0000","last_peered":"2024-03-19T03:30:40.800182+0000","last_clean":"2024-03-19T03:30:40.800182+0000","last_became_active":"2024-03-19T03:30:35.675603+0000","last_became_peered":"2024-03-19T03:30:35.675603+0000","last_unstale":"2024-03-19T03:30:40.800182+0000","last_undegraded":"2024-03-19T03:30:40.800182+0000","last_fullsized":"2024-03-19T03:30:40.800182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:39:47.429027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.801751+0000","last_change":"2024-03-19T03:30:39.746747+0000","last_active":"2024-03-19T03:30:40.801751+0000","last_peered":"2024-03-19T03:30:40.801751+0000","last_clean":"2024-03-19T03:30:40.801751+0000","last_became_active":"2024-03-19T03:30:39.745553+0000","last_became_peered":"2024-03-19T03:30:39.745553+0000","last_unstale":"2024-03-19T03:30:40.801751+0000","last_undegraded":"2024-03-19T03:30:40.801751+0000","last_fullsized":"2024-03-19T03:30:40.801751+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:14:56.211061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800676+0000","last_change":"2024-03-19T03:30:35.649795+0000","last_active":"2024-03-19T03:30:40.800676+0000","last_peered":"2024-03-19T03:30:40.800676+0000","last_clean":"2024-03-19T03:30:40.800676+0000","last_became_active":"2024-03-19T03:30:35.649417+0000","last_became_peered":"2024-03-19T03:30:35.649417+0000","last_unstale":"2024-03-19T03:30:40.800676+0000","last_undegraded":"2024-03-19T03:30:40.800676+0000","last_fullsized":"2024-03-19T03:30:40.800676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:24.065817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:10.001028+0000","last_change":"2024-03-19T03:30:34.637446+0000","last_active":"2024-03-19T03:31:10.001028+0000","last_peered":"2024-03-19T03:31:10.001028+0000","last_clean":"2024-03-19T03:31:10.001028+0000","last_became_active":"2024-03-19T03:30:34.637083+0000","last_became_peered":"2024-03-19T03:30:34.637083+0000","last_unstale":"2024-03-19T03:31:10.001028+0000","last_undegraded":"2024-03-19T03:31:10.001028+0000","last_fullsized":"2024-03-19T03:31:10.001028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:51:02.090890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.987872+0000","last_change":"2024-03-19T03:30:39.726337+0000","last_active":"2024-03-19T03:30:40.987872+0000","last_peered":"2024-03-19T03:30:40.987872+0000","last_clean":"2024-03-19T03:30:40.987872+0000","last_became_active":"2024-03-19T03:30:39.725919+0000","last_became_peered":"2024-03-19T03:30:39.725919+0000","last_unstale":"2024-03-19T03:30:40.987872+0000","last_undegraded":"2024-03-19T03:30:40.987872+0000","last_fullsized":"2024-03-19T03:30:40.987872+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:19:48.993790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.802078+0000","last_change":"2024-03-19T03:30:35.670002+0000","last_active":"2024-03-19T03:30:40.802078+0000","last_peered":"2024-03-19T03:30:40.802078+0000","last_clean":"2024-03-19T03:30:40.802078+0000","last_became_active":"2024-03-19T03:30:35.653208+0000","last_became_peered":"2024-03-19T03:30:35.653208+0000","last_unstale":"2024-03-19T03:30:40.802078+0000","last_undegraded":"2024-03-19T03:30:40.802078+0000","last_fullsized":"2024-03-19T03:30:40.802078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:37:15.246977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:09.005888+0000","last_change":"2024-03-19T03:30:34.625107+0000","last_active":"2024-03-19T03:31:09.005888+0000","last_peered":"2024-03-19T03:31:09.005888+0000","last_clean":"2024-03-19T03:31:09.005888+0000","last_became_active":"2024-03-19T03:30:34.624711+0000","last_became_peered":"2024-03-19T03:30:34.624711+0000","last_unstale":"2024-03-19T03:31:09.005888+0000","last_undegraded":"2024-03-19T03:31:09.005888+0000","last_fullsized":"2024-03-19T03:31:09.005888+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:39:33.880100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":346,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.723512+0000","last_change":"2024-03-19T03:29:11.374136+0000","last_active":"2024-03-19T03:30:40.723512+0000","last_peered":"2024-03-19T03:30:40.723512+0000","last_clean":"2024-03-19T03:30:40.723512+0000","last_became_active":"2024-03-19T03:29:11.373691+0000","last_became_peered":"2024-03-19T03:29:11.373691+0000","last_unstale":"2024-03-19T03:30:40.723512+0000","last_undegraded":"2024-03-19T03:30:40.723512+0000","last_fullsized":"2024-03-19T03:30:40.723512+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:25:19.274380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:25:19.274380+0000","last_clean_scrub_stamp":"2024-03-19T03:25:19.274380+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:29:59.731772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.801172+0000","last_change":"2024-03-19T03:30:39.744391+0000","last_active":"2024-03-19T03:30:40.801172+0000","last_peered":"2024-03-19T03:30:40.801172+0000","last_clean":"2024-03-19T03:30:40.801172+0000","last_became_active":"2024-03-19T03:30:39.743995+0000","last_became_peered":"2024-03-19T03:30:39.743995+0000","last_unstale":"2024-03-19T03:30:40.801172+0000","last_undegraded":"2024-03-19T03:30:40.801172+0000","last_fullsized":"2024-03-19T03:30:40.801172+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:41:02.714132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.801257+0000","last_change":"2024-03-19T03:30:35.662528+0000","last_active":"2024-03-19T03:30:40.801257+0000","last_peered":"2024-03-19T03:30:40.801257+0000","last_clean":"2024-03-19T03:30:40.801257+0000","last_became_active":"2024-03-19T03:30:35.659782+0000","last_became_peered":"2024-03-19T03:30:35.659782+0000","last_unstale":"2024-03-19T03:30:40.801257+0000","last_undegraded":"2024-03-19T03:30:40.801257+0000","last_fullsized":"2024-03-19T03:30:40.801257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:18:12.838234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:07.477796+0000","last_change":"2024-03-19T03:30:34.620988+0000","last_active":"2024-03-19T03:31:07.477796+0000","last_peered":"2024-03-19T03:31:07.477796+0000","last_clean":"2024-03-19T03:31:07.477796+0000","last_became_active":"2024-03-19T03:30:34.620411+0000","last_became_peered":"2024-03-19T03:30:34.620411+0000","last_unstale":"2024-03-19T03:31:07.477796+0000","last_undegraded":"2024-03-19T03:31:07.477796+0000","last_fullsized":"2024-03-19T03:31:07.477796+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:37:13.275121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800830+0000","last_change":"2024-03-19T03:30:39.733325+0000","last_active":"2024-03-19T03:30:40.800830+0000","last_peered":"2024-03-19T03:30:40.800830+0000","last_clean":"2024-03-19T03:30:40.800830+0000","last_became_active":"2024-03-19T03:30:39.732553+0000","last_became_peered":"2024-03-19T03:30:39.732553+0000","last_unstale":"2024-03-19T03:30:40.800830+0000","last_undegraded":"2024-03-19T03:30:40.800830+0000","last_fullsized":"2024-03-19T03:30:40.800830+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:52:47.403673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:08.991819+0000","last_change":"2024-03-19T03:30:34.637100+0000","last_active":"2024-03-19T03:31:08.991819+0000","last_peered":"2024-03-19T03:31:08.991819+0000","last_clean":"2024-03-19T03:31:08.991819+0000","last_became_active":"2024-03-19T03:30:34.636715+0000","last_became_peered":"2024-03-19T03:30:34.636715+0000","last_unstale":"2024-03-19T03:31:08.991819+0000","last_undegraded":"2024-03-19T03:31:08.991819+0000","last_fullsized":"2024-03-19T03:31:08.991819+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:48:16.212158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.802061+0000","last_change":"2024-03-19T03:30:35.668405+0000","last_active":"2024-03-19T03:30:40.802061+0000","last_peered":"2024-03-19T03:30:40.802061+0000","last_clean":"2024-03-19T03:30:40.802061+0000","last_became_active":"2024-03-19T03:30:35.667972+0000","last_became_peered":"2024-03-19T03:30:35.667972+0000","last_unstale":"2024-03-19T03:30:40.802061+0000","last_undegraded":"2024-03-19T03:30:40.802061+0000","last_fullsized":"2024-03-19T03:30:40.802061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:01:52.816110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800510+0000","last_change":"2024-03-19T03:30:39.747535+0000","last_active":"2024-03-19T03:30:40.800510+0000","last_peered":"2024-03-19T03:30:40.800510+0000","last_clean":"2024-03-19T03:30:40.800510+0000","last_became_active":"2024-03-19T03:30:39.747122+0000","last_became_peered":"2024-03-19T03:30:39.747122+0000","last_unstale":"2024-03-19T03:30:40.800510+0000","last_undegraded":"2024-03-19T03:30:40.800510+0000","last_fullsized":"2024-03-19T03:30:40.800510+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:07:35.642809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'9","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:09.675963+0000","last_change":"2024-03-19T03:30:34.637768+0000","last_active":"2024-03-19T03:31:09.675963+0000","last_peered":"2024-03-19T03:31:09.675963+0000","last_clean":"2024-03-19T03:31:09.675963+0000","last_became_active":"2024-03-19T03:30:34.637431+0000","last_became_peered":"2024-03-19T03:30:34.637431+0000","last_unstale":"2024-03-19T03:31:09.675963+0000","last_undegraded":"2024-03-19T03:31:09.675963+0000","last_fullsized":"2024-03-19T03:31:09.675963+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:22:50.487209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.803278+0000","last_change":"2024-03-19T03:30:35.661352+0000","last_active":"2024-03-19T03:30:40.803278+0000","last_peered":"2024-03-19T03:30:40.803278+0000","last_clean":"2024-03-19T03:30:40.803278+0000","last_became_active":"2024-03-19T03:30:35.658656+0000","last_became_peered":"2024-03-19T03:30:35.658656+0000","last_unstale":"2024-03-19T03:30:40.803278+0000","last_undegraded":"2024-03-19T03:30:40.803278+0000","last_fullsized":"2024-03-19T03:30:40.803278+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:36.532748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.723999+0000","last_change":"2024-03-19T03:30:39.724637+0000","last_active":"2024-03-19T03:30:40.723999+0000","last_peered":"2024-03-19T03:30:40.723999+0000","last_clean":"2024-03-19T03:30:40.723999+0000","last_became_active":"2024-03-19T03:30:39.723842+0000","last_became_peered":"2024-03-19T03:30:39.723842+0000","last_unstale":"2024-03-19T03:30:40.723999+0000","last_undegraded":"2024-03-19T03:30:40.723999+0000","last_fullsized":"2024-03-19T03:30:40.723999+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:04:02.446185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:09.003790+0000","last_change":"2024-03-19T03:30:34.620404+0000","last_active":"2024-03-19T03:31:09.003790+0000","last_peered":"2024-03-19T03:31:09.003790+0000","last_clean":"2024-03-19T03:31:09.003790+0000","last_became_active":"2024-03-19T03:30:34.620056+0000","last_became_peered":"2024-03-19T03:30:34.620056+0000","last_unstale":"2024-03-19T03:31:09.003790+0000","last_undegraded":"2024-03-19T03:31:09.003790+0000","last_fullsized":"2024-03-19T03:31:09.003790+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:54:42.203912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.730004+0000","last_change":"2024-03-19T03:30:35.686618+0000","last_active":"2024-03-19T03:30:40.730004+0000","last_peered":"2024-03-19T03:30:40.730004+0000","last_clean":"2024-03-19T03:30:40.730004+0000","last_became_active":"2024-03-19T03:30:35.685814+0000","last_became_peered":"2024-03-19T03:30:35.685814+0000","last_unstale":"2024-03-19T03:30:40.730004+0000","last_undegraded":"2024-03-19T03:30:40.730004+0000","last_fullsized":"2024-03-19T03:30:40.730004+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:08.401307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799486+0000","last_change":"2024-03-19T03:30:39.744039+0000","last_active":"2024-03-19T03:30:40.799486+0000","last_peered":"2024-03-19T03:30:40.799486+0000","last_clean":"2024-03-19T03:30:40.799486+0000","last_became_active":"2024-03-19T03:30:39.743637+0000","last_became_peered":"2024-03-19T03:30:39.743637+0000","last_unstale":"2024-03-19T03:30:40.799486+0000","last_undegraded":"2024-03-19T03:30:40.799486+0000","last_fullsized":"2024-03-19T03:30:40.799486+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:02:22.498333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:06.473944+0000","last_change":"2024-03-19T03:30:34.629059+0000","last_active":"2024-03-19T03:31:06.473944+0000","last_peered":"2024-03-19T03:31:06.473944+0000","last_clean":"2024-03-19T03:31:06.473944+0000","last_became_active":"2024-03-19T03:30:34.628617+0000","last_became_peered":"2024-03-19T03:30:34.628617+0000","last_unstale":"2024-03-19T03:31:06.473944+0000","last_undegraded":"2024-03-19T03:31:06.473944+0000","last_fullsized":"2024-03-19T03:31:06.473944+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:47:09.064502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799393+0000","last_change":"2024-03-19T03:30:35.650256+0000","last_active":"2024-03-19T03:30:40.799393+0000","last_peered":"2024-03-19T03:30:40.799393+0000","last_clean":"2024-03-19T03:30:40.799393+0000","last_became_active":"2024-03-19T03:30:35.649086+0000","last_became_peered":"2024-03-19T03:30:35.649086+0000","last_unstale":"2024-03-19T03:30:40.799393+0000","last_undegraded":"2024-03-19T03:30:40.799393+0000","last_fullsized":"2024-03-19T03:30:40.799393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:00:49.204307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.803484+0000","last_change":"2024-03-19T03:30:39.736253+0000","last_active":"2024-03-19T03:30:40.803484+0000","last_peered":"2024-03-19T03:30:40.803484+0000","last_clean":"2024-03-19T03:30:40.803484+0000","last_became_active":"2024-03-19T03:30:39.735699+0000","last_became_peered":"2024-03-19T03:30:39.735699+0000","last_unstale":"2024-03-19T03:30:40.803484+0000","last_undegraded":"2024-03-19T03:30:40.803484+0000","last_fullsized":"2024-03-19T03:30:40.803484+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:46:01.926210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:09.006187+0000","last_change":"2024-03-19T03:30:34.634935+0000","last_active":"2024-03-19T03:31:09.006187+0000","last_peered":"2024-03-19T03:31:09.006187+0000","last_clean":"2024-03-19T03:31:09.006187+0000","last_became_active":"2024-03-19T03:30:34.634602+0000","last_became_peered":"2024-03-19T03:30:34.634602+0000","last_unstale":"2024-03-19T03:31:09.006187+0000","last_undegraded":"2024-03-19T03:31:09.006187+0000","last_fullsized":"2024-03-19T03:31:09.006187+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:16:57.158105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800551+0000","last_change":"2024-03-19T03:30:35.675650+0000","last_active":"2024-03-19T03:30:40.800551+0000","last_peered":"2024-03-19T03:30:40.800551+0000","last_clean":"2024-03-19T03:30:40.800551+0000","last_became_active":"2024-03-19T03:30:35.675255+0000","last_became_peered":"2024-03-19T03:30:35.675255+0000","last_unstale":"2024-03-19T03:30:40.800551+0000","last_undegraded":"2024-03-19T03:30:40.800551+0000","last_fullsized":"2024-03-19T03:30:40.800551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:55:03.201526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:41.558163+0000","last_change":"2024-03-19T03:30:39.733413+0000","last_active":"2024-03-19T03:30:41.558163+0000","last_peered":"2024-03-19T03:30:41.558163+0000","last_clean":"2024-03-19T03:30:41.558163+0000","last_became_active":"2024-03-19T03:30:39.732998+0000","last_became_peered":"2024-03-19T03:30:39.732998+0000","last_unstale":"2024-03-19T03:30:41.558163+0000","last_undegraded":"2024-03-19T03:30:41.558163+0000","last_fullsized":"2024-03-19T03:30:41.558163+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:07:39.512355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:07.467552+0000","last_change":"2024-03-19T03:30:34.623749+0000","last_active":"2024-03-19T03:31:07.467552+0000","last_peered":"2024-03-19T03:31:07.467552+0000","last_clean":"2024-03-19T03:31:07.467552+0000","last_became_active":"2024-03-19T03:30:34.623378+0000","last_became_peered":"2024-03-19T03:30:34.623378+0000","last_unstale":"2024-03-19T03:31:07.467552+0000","last_undegraded":"2024-03-19T03:31:07.467552+0000","last_fullsized":"2024-03-19T03:31:07.467552+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:06:01.939685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.802593+0000","last_change":"2024-03-19T03:30:35.685702+0000","last_active":"2024-03-19T03:30:40.802593+0000","last_peered":"2024-03-19T03:30:40.802593+0000","last_clean":"2024-03-19T03:30:40.802593+0000","last_became_active":"2024-03-19T03:30:35.683340+0000","last_became_peered":"2024-03-19T03:30:35.683340+0000","last_unstale":"2024-03-19T03:30:40.802593+0000","last_undegraded":"2024-03-19T03:30:40.802593+0000","last_fullsized":"2024-03-19T03:30:40.802593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:12.246255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799661+0000","last_change":"2024-03-19T03:30:39.726747+0000","last_active":"2024-03-19T03:30:40.799661+0000","last_peered":"2024-03-19T03:30:40.799661+0000","last_clean":"2024-03-19T03:30:40.799661+0000","last_became_active":"2024-03-19T03:30:39.726236+0000","last_became_peered":"2024-03-19T03:30:39.726236+0000","last_unstale":"2024-03-19T03:30:40.799661+0000","last_undegraded":"2024-03-19T03:30:40.799661+0000","last_fullsized":"2024-03-19T03:30:40.799661+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:02.333544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:06.471949+0000","last_change":"2024-03-19T03:30:34.627828+0000","last_active":"2024-03-19T03:31:06.471949+0000","last_peered":"2024-03-19T03:31:06.471949+0000","last_clean":"2024-03-19T03:31:06.471949+0000","last_became_active":"2024-03-19T03:30:34.627290+0000","last_became_peered":"2024-03-19T03:30:34.627290+0000","last_unstale":"2024-03-19T03:31:06.471949+0000","last_undegraded":"2024-03-19T03:31:06.471949+0000","last_fullsized":"2024-03-19T03:31:06.471949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:07:50.140087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.723835+0000","last_change":"2024-03-19T03:30:35.661180+0000","last_active":"2024-03-19T03:30:40.723835+0000","last_peered":"2024-03-19T03:30:40.723835+0000","last_clean":"2024-03-19T03:30:40.723835+0000","last_became_active":"2024-03-19T03:30:35.660797+0000","last_became_peered":"2024-03-19T03:30:35.660797+0000","last_unstale":"2024-03-19T03:30:40.723835+0000","last_undegraded":"2024-03-19T03:30:40.723835+0000","last_fullsized":"2024-03-19T03:30:40.723835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:54:36.034281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.798935+0000","last_change":"2024-03-19T03:30:39.732832+0000","last_active":"2024-03-19T03:30:40.798935+0000","last_peered":"2024-03-19T03:30:40.798935+0000","last_clean":"2024-03-19T03:30:40.798935+0000","last_became_active":"2024-03-19T03:30:39.732434+0000","last_became_peered":"2024-03-19T03:30:39.732434+0000","last_unstale":"2024-03-19T03:30:40.798935+0000","last_undegraded":"2024-03-19T03:30:40.798935+0000","last_fullsized":"2024-03-19T03:30:40.798935+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:49.737361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:08.995286+0000","last_change":"2024-03-19T03:30:34.625655+0000","last_active":"2024-03-19T03:31:08.995286+0000","last_peered":"2024-03-19T03:31:08.995286+0000","last_clean":"2024-03-19T03:31:08.995286+0000","last_became_active":"2024-03-19T03:30:34.625251+0000","last_became_peered":"2024-03-19T03:30:34.625251+0000","last_unstale":"2024-03-19T03:31:08.995286+0000","last_undegraded":"2024-03-19T03:31:08.995286+0000","last_fullsized":"2024-03-19T03:31:08.995286+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:25.592670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.724140+0000","last_change":"2024-03-19T03:30:35.651147+0000","last_active":"2024-03-19T03:30:40.724140+0000","last_peered":"2024-03-19T03:30:40.724140+0000","last_clean":"2024-03-19T03:30:40.724140+0000","last_became_active":"2024-03-19T03:30:35.650817+0000","last_became_peered":"2024-03-19T03:30:35.650817+0000","last_unstale":"2024-03-19T03:30:40.724140+0000","last_undegraded":"2024-03-19T03:30:40.724140+0000","last_fullsized":"2024-03-19T03:30:40.724140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:35.529226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.728902+0000","last_change":"2024-03-19T03:30:39.749895+0000","last_active":"2024-03-19T03:30:40.728902+0000","last_peered":"2024-03-19T03:30:40.728902+0000","last_clean":"2024-03-19T03:30:40.728902+0000","last_became_active":"2024-03-19T03:30:39.749555+0000","last_became_peered":"2024-03-19T03:30:39.749555+0000","last_unstale":"2024-03-19T03:30:40.728902+0000","last_undegraded":"2024-03-19T03:30:40.728902+0000","last_fullsized":"2024-03-19T03:30:40.728902+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:10:14.383271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:08.991668+0000","last_change":"2024-03-19T03:30:34.635456+0000","last_active":"2024-03-19T03:31:08.991668+0000","last_peered":"2024-03-19T03:31:08.991668+0000","last_clean":"2024-03-19T03:31:08.991668+0000","last_became_active":"2024-03-19T03:30:34.635099+0000","last_became_peered":"2024-03-19T03:30:34.635099+0000","last_unstale":"2024-03-19T03:31:08.991668+0000","last_undegraded":"2024-03-19T03:31:08.991668+0000","last_fullsized":"2024-03-19T03:31:08.991668+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:39:30.083678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.723709+0000","last_change":"2024-03-19T03:30:35.659490+0000","last_active":"2024-03-19T03:30:40.723709+0000","last_peered":"2024-03-19T03:30:40.723709+0000","last_clean":"2024-03-19T03:30:40.723709+0000","last_became_active":"2024-03-19T03:30:35.659208+0000","last_became_peered":"2024-03-19T03:30:35.659208+0000","last_unstale":"2024-03-19T03:30:40.723709+0000","last_undegraded":"2024-03-19T03:30:40.723709+0000","last_fullsized":"2024-03-19T03:30:40.723709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:18.788981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799669+0000","last_change":"2024-03-19T03:30:39.725193+0000","last_active":"2024-03-19T03:30:40.799669+0000","last_peered":"2024-03-19T03:30:40.799669+0000","last_clean":"2024-03-19T03:30:40.799669+0000","last_became_active":"2024-03-19T03:30:39.724745+0000","last_became_peered":"2024-03-19T03:30:39.724745+0000","last_unstale":"2024-03-19T03:30:40.799669+0000","last_undegraded":"2024-03-19T03:30:40.799669+0000","last_fullsized":"2024-03-19T03:30:40.799669+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:30:07.621852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:09.005192+0000","last_change":"2024-03-19T03:30:34.636532+0000","last_active":"2024-03-19T03:31:09.005192+0000","last_peered":"2024-03-19T03:31:09.005192+0000","last_clean":"2024-03-19T03:31:09.005192+0000","last_became_active":"2024-03-19T03:30:34.634104+0000","last_became_peered":"2024-03-19T03:30:34.634104+0000","last_unstale":"2024-03-19T03:31:09.005192+0000","last_undegraded":"2024-03-19T03:31:09.005192+0000","last_fullsized":"2024-03-19T03:31:09.005192+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:41:18.308628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.723909+0000","last_change":"2024-03-19T03:30:35.653036+0000","last_active":"2024-03-19T03:30:40.723909+0000","last_peered":"2024-03-19T03:30:40.723909+0000","last_clean":"2024-03-19T03:30:40.723909+0000","last_became_active":"2024-03-19T03:30:35.652564+0000","last_became_peered":"2024-03-19T03:30:35.652564+0000","last_unstale":"2024-03-19T03:30:40.723909+0000","last_undegraded":"2024-03-19T03:30:40.723909+0000","last_fullsized":"2024-03-19T03:30:40.723909+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:01.812525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800454+0000","last_change":"2024-03-19T03:30:35.650724+0000","last_active":"2024-03-19T03:30:40.800454+0000","last_peered":"2024-03-19T03:30:40.800454+0000","last_clean":"2024-03-19T03:30:40.800454+0000","last_became_active":"2024-03-19T03:30:35.650204+0000","last_became_peered":"2024-03-19T03:30:35.650204+0000","last_unstale":"2024-03-19T03:30:40.800454+0000","last_undegraded":"2024-03-19T03:30:40.800454+0000","last_fullsized":"2024-03-19T03:30:40.800454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:06:56.159133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.724460+0000","last_change":"2024-03-19T03:30:35.650241+0000","last_active":"2024-03-19T03:30:40.724460+0000","last_peered":"2024-03-19T03:30:40.724460+0000","last_clean":"2024-03-19T03:30:40.724460+0000","last_became_active":"2024-03-19T03:30:35.649901+0000","last_became_peered":"2024-03-19T03:30:35.649901+0000","last_unstale":"2024-03-19T03:30:40.724460+0000","last_undegraded":"2024-03-19T03:30:40.724460+0000","last_fullsized":"2024-03-19T03:30:40.724460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:35:53.333403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.724030+0000","last_change":"2024-03-19T03:30:35.679267+0000","last_active":"2024-03-19T03:30:40.724030+0000","last_peered":"2024-03-19T03:30:40.724030+0000","last_clean":"2024-03-19T03:30:40.724030+0000","last_became_active":"2024-03-19T03:30:35.678871+0000","last_became_peered":"2024-03-19T03:30:35.678871+0000","last_unstale":"2024-03-19T03:30:40.724030+0000","last_undegraded":"2024-03-19T03:30:40.724030+0000","last_fullsized":"2024-03-19T03:30:40.724030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:23:17.681854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.729073+0000","last_change":"2024-03-19T03:30:35.683060+0000","last_active":"2024-03-19T03:30:40.729073+0000","last_peered":"2024-03-19T03:30:40.729073+0000","last_clean":"2024-03-19T03:30:40.729073+0000","last_became_active":"2024-03-19T03:30:35.676872+0000","last_became_peered":"2024-03-19T03:30:35.676872+0000","last_unstale":"2024-03-19T03:30:40.729073+0000","last_undegraded":"2024-03-19T03:30:40.729073+0000","last_fullsized":"2024-03-19T03:30:40.729073+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T15:13:51.530914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.802168+0000","last_change":"2024-03-19T03:30:35.682685+0000","last_active":"2024-03-19T03:30:40.802168+0000","last_peered":"2024-03-19T03:30:40.802168+0000","last_clean":"2024-03-19T03:30:40.802168+0000","last_became_active":"2024-03-19T03:30:35.680309+0000","last_became_peered":"2024-03-19T03:30:35.680309+0000","last_unstale":"2024-03-19T03:30:40.802168+0000","last_undegraded":"2024-03-19T03:30:40.802168+0000","last_fullsized":"2024-03-19T03:30:40.802168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:31:59.925521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800307+0000","last_change":"2024-03-19T03:30:35.652656+0000","last_active":"2024-03-19T03:30:40.800307+0000","last_peered":"2024-03-19T03:30:40.800307+0000","last_clean":"2024-03-19T03:30:40.800307+0000","last_became_active":"2024-03-19T03:30:35.650744+0000","last_became_peered":"2024-03-19T03:30:35.650744+0000","last_unstale":"2024-03-19T03:30:40.800307+0000","last_undegraded":"2024-03-19T03:30:40.800307+0000","last_fullsized":"2024-03-19T03:30:40.800307+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:49:01.450719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.724650+0000","last_change":"2024-03-19T03:30:35.650092+0000","last_active":"2024-03-19T03:30:40.724650+0000","last_peered":"2024-03-19T03:30:40.724650+0000","last_clean":"2024-03-19T03:30:40.724650+0000","last_became_active":"2024-03-19T03:30:35.649727+0000","last_became_peered":"2024-03-19T03:30:35.649727+0000","last_unstale":"2024-03-19T03:30:40.724650+0000","last_undegraded":"2024-03-19T03:30:40.724650+0000","last_fullsized":"2024-03-19T03:30:40.724650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:22:13.968226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.803587+0000","last_change":"2024-03-19T03:30:35.664723+0000","last_active":"2024-03-19T03:30:40.803587+0000","last_peered":"2024-03-19T03:30:40.803587+0000","last_clean":"2024-03-19T03:30:40.803587+0000","last_became_active":"2024-03-19T03:30:35.655429+0000","last_became_peered":"2024-03-19T03:30:35.655429+0000","last_unstale":"2024-03-19T03:30:40.803587+0000","last_undegraded":"2024-03-19T03:30:40.803587+0000","last_fullsized":"2024-03-19T03:30:40.803587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:09.230235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800537+0000","last_change":"2024-03-19T03:30:35.660289+0000","last_active":"2024-03-19T03:30:40.800537+0000","last_peered":"2024-03-19T03:30:40.800537+0000","last_clean":"2024-03-19T03:30:40.800537+0000","last_became_active":"2024-03-19T03:30:35.660005+0000","last_became_peered":"2024-03-19T03:30:35.660005+0000","last_unstale":"2024-03-19T03:30:40.800537+0000","last_undegraded":"2024-03-19T03:30:40.800537+0000","last_fullsized":"2024-03-19T03:30:40.800537+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:36:16.033629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.801327+0000","last_change":"2024-03-19T03:30:35.660461+0000","last_active":"2024-03-19T03:30:40.801327+0000","last_peered":"2024-03-19T03:30:40.801327+0000","last_clean":"2024-03-19T03:30:40.801327+0000","last_became_active":"2024-03-19T03:30:35.660058+0000","last_became_peered":"2024-03-19T03:30:35.660058+0000","last_unstale":"2024-03-19T03:30:40.801327+0000","last_undegraded":"2024-03-19T03:30:40.801327+0000","last_fullsized":"2024-03-19T03:30:40.801327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:08:40.326314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799421+0000","last_change":"2024-03-19T03:30:35.665634+0000","last_active":"2024-03-19T03:30:40.799421+0000","last_peered":"2024-03-19T03:30:40.799421+0000","last_clean":"2024-03-19T03:30:40.799421+0000","last_became_active":"2024-03-19T03:30:35.665264+0000","last_became_peered":"2024-03-19T03:30:35.665264+0000","last_unstale":"2024-03-19T03:30:40.799421+0000","last_undegraded":"2024-03-19T03:30:40.799421+0000","last_fullsized":"2024-03-19T03:30:40.799421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:33:56.931412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.801353+0000","last_change":"2024-03-19T03:30:35.681887+0000","last_active":"2024-03-19T03:30:40.801353+0000","last_peered":"2024-03-19T03:30:40.801353+0000","last_clean":"2024-03-19T03:30:40.801353+0000","last_became_active":"2024-03-19T03:30:35.680332+0000","last_became_peered":"2024-03-19T03:30:35.680332+0000","last_unstale":"2024-03-19T03:30:40.801353+0000","last_undegraded":"2024-03-19T03:30:40.801353+0000","last_fullsized":"2024-03-19T03:30:40.801353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:50:24.243641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800987+0000","last_change":"2024-03-19T03:30:35.652988+0000","last_active":"2024-03-19T03:30:40.800987+0000","last_peered":"2024-03-19T03:30:40.800987+0000","last_clean":"2024-03-19T03:30:40.800987+0000","last_became_active":"2024-03-19T03:30:35.652608+0000","last_became_peered":"2024-03-19T03:30:35.652608+0000","last_unstale":"2024-03-19T03:30:40.800987+0000","last_undegraded":"2024-03-19T03:30:40.800987+0000","last_fullsized":"2024-03-19T03:30:40.800987+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:44:35.619094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283260,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10956,"num_objects":93,"num_object_clones":0,"num_object_copies":279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":204,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":62,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":76643,"data_compressed":1826,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},"log_size":179,"ondisk_log_size":179,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383702,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14713,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645338,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":1073003,"data_compressed":50737,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808865,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":482433,"data_compressed":14713,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070502,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":482811,"data_compressed":14713,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332139,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":1075603,"data_compressed":50737,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495666,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":477937,"data_compressed":14713,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1007616,"data_stored":502108,"data_compressed":14713,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":479393,"data_compressed":14713,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":490020,"data_compressed":14713,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182466,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":477645,"data_compressed":14713,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920830,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":482402,"data_compressed":14713,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757303,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":1074890,"data_compressed":50737,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34380,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34380,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34380,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36024,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36024,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36024,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36024,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36024,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":43658,"data_compressed":1826,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":12805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T03:31:12.585 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T03:31:12.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.825+0000 7fc7dec00640 1 -- 172.21.15.117:0/393377276 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7d8100570 msgr2=0x7fc7d8100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:12.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.825+0000 7fc7dec00640 1 --2- 172.21.15.117:0/393377276 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7d8100570 0x7fc7d8100990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc009900 tx=0x7fc7cc02bc30 comp rx=0 tx=0).stop 2024-03-19T03:31:12.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.825+0000 7fc7dec00640 1 -- 172.21.15.117:0/393377276 shutdown_connections 2024-03-19T03:31:12.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.825+0000 7fc7dec00640 1 --2- 172.21.15.117:0/393377276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc7d8102120 0x7fc7d81089b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:12.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.825+0000 7fc7dec00640 1 --2- 172.21.15.117:0/393377276 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc7d8101760 0x7fc7d8101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:12.827 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.825+0000 7fc7dec00640 1 --2- 172.21.15.117:0/393377276 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7d8100570 0x7fc7d8100990 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:12.828 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.825+0000 7fc7dec00640 1 -- 172.21.15.117:0/393377276 >> 172.21.15.117:0/393377276 conn(0x7fc7d80fbb00 msgr2=0x7fc7d80fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:12.828 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.825+0000 7fc7dec00640 1 -- 172.21.15.117:0/393377276 shutdown_connections 2024-03-19T03:31:12.828 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.825+0000 7fc7dec00640 1 -- 172.21.15.117:0/393377276 wait complete. 2024-03-19T03:31:12.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.827+0000 7fc7dec00640 1 Processor -- start 2024-03-19T03:31:12.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.827+0000 7fc7dec00640 1 -- start start 2024-03-19T03:31:12.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.828+0000 7fc7dec00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc7d8100570 0x7fc7d8069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:12.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.828+0000 7fc7dec00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7d8101760 0x7fc7d819c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:12.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.828+0000 7fc7dec00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc7d8102120 0x7fc7d819ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:12.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.828+0000 7fc7dec00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d810c250 con 0x7fc7d8100570 2024-03-19T03:31:12.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.828+0000 7fc7dec00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d810c0d0 con 0x7fc7d8101760 2024-03-19T03:31:12.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.828+0000 7fc7dec00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d810c3d0 con 0x7fc7d8102120 2024-03-19T03:31:12.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.828+0000 7fc7dd800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc7d8100570 0x7fc7d8069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:12.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.828+0000 7fc7dce00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7d8101760 0x7fc7d819c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:12.830 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.828+0000 7fc7dce00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7d8101760 0x7fc7d819c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:51986/0 (socket says 172.21.15.117:51986) 2024-03-19T03:31:12.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.828+0000 7fc7dd800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc7d8100570 0x7fc7d8069550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:60972/0 (socket says 172.21.15.117:60972) 2024-03-19T03:31:12.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.828+0000 7fc7dce00640 1 -- 172.21.15.117:0/4005000896 learned_addr learned my addr 172.21.15.117:0/4005000896 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:12.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.828+0000 7fc7de200640 1 --2- 172.21.15.117:0/4005000896 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc7d8102120 0x7fc7d819ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:12.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.828+0000 7fc7dce00640 1 -- 172.21.15.117:0/4005000896 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc7d8102120 msgr2=0x7fc7d819ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:12.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.828+0000 7fc7dce00640 1 --2- 172.21.15.117:0/4005000896 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc7d8102120 0x7fc7d819ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:12.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.828+0000 7fc7dce00640 1 -- 172.21.15.117:0/4005000896 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc7d8100570 msgr2=0x7fc7d8069550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:12.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.828+0000 7fc7dce00640 1 --2- 172.21.15.117:0/4005000896 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc7d8100570 0x7fc7d8069550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:12.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.828+0000 7fc7dce00640 1 -- 172.21.15.117:0/4005000896 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7d819f5c0 con 0x7fc7d8101760 2024-03-19T03:31:12.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.829+0000 7fc7dce00640 1 --2- 172.21.15.117:0/4005000896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7d8101760 0x7fc7d819c690 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc7c400e9f0 tx=0x7fc7c400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:12.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.829+0000 7fc7d6000640 1 -- 172.21.15.117:0/4005000896 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c400cd80 con 0x7fc7d8101760 2024-03-19T03:31:12.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.829+0000 7fc7d6000640 1 -- 172.21.15.117:0/4005000896 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7c4005500 con 0x7fc7d8101760 2024-03-19T03:31:12.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.829+0000 7fc7dec00640 1 -- 172.21.15.117:0/4005000896 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7d819f8b0 con 0x7fc7d8101760 2024-03-19T03:31:12.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.829+0000 7fc7d6000640 1 -- 172.21.15.117:0/4005000896 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c4010690 con 0x7fc7d8101760 2024-03-19T03:31:12.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.829+0000 7fc7dec00640 1 -- 172.21.15.117:0/4005000896 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7d819fe00 con 0x7fc7d8101760 2024-03-19T03:31:12.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.830+0000 7fc7dec00640 1 -- 172.21.15.117:0/4005000896 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7d8107890 con 0x7fc7d8101760 2024-03-19T03:31:12.833 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.833+0000 7fc7d6000640 1 -- 172.21.15.117:0/4005000896 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fc7c40046c0 con 0x7fc7d8101760 2024-03-19T03:31:12.835 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.834+0000 7fc7d6000640 1 --2- 172.21.15.117:0/4005000896 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc7c00a2910 0x7fc7c00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:12.835 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.834+0000 7fc7d6000640 1 -- 172.21.15.117:0/4005000896 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc7c40cf210 con 0x7fc7d8101760 2024-03-19T03:31:12.835 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.834+0000 7fc7dd800640 1 --2- 172.21.15.117:0/4005000896 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc7c00a2910 0x7fc7c00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:12.836 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.835+0000 7fc7dd800640 1 --2- 172.21.15.117:0/4005000896 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc7c00a2910 0x7fc7c00a4dd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc02c170 tx=0x7fc7cc02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:12.843 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:12.842+0000 7fc7d6000640 1 -- 172.21.15.117:0/4005000896 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc7c409beb0 con 0x7fc7d8101760 2024-03-19T03:31:13.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:13.041+0000 7fc7dec00640 1 -- 172.21.15.117:0/4005000896 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc7d8061c60 con 0x7fc7c00a2910 2024-03-19T03:31:13.053 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:13.052+0000 7fc7d6000640 1 -- 172.21.15.117:0/4005000896 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340705 (secure 0 0 0) 0x7fc7d8061c60 con 0x7fc7c00a2910 2024-03-19T03:31:13.058 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:31:13.059 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-03-19T03:31:13.061 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:13.058+0000 7fc7dec00640 1 -- 172.21.15.117:0/4005000896 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc7c00a2910 msgr2=0x7fc7c00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:13.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:13.058+0000 7fc7dec00640 1 --2- 172.21.15.117:0/4005000896 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc7c00a2910 0x7fc7c00a4dd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc02c170 tx=0x7fc7cc02df90 comp rx=0 tx=0).stop 2024-03-19T03:31:13.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:13.058+0000 7fc7dec00640 1 -- 172.21.15.117:0/4005000896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7d8101760 msgr2=0x7fc7d819c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:13.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:13.058+0000 7fc7dec00640 1 --2- 172.21.15.117:0/4005000896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7d8101760 0x7fc7d819c690 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc7c400e9f0 tx=0x7fc7c400edf0 comp rx=0 tx=0).stop 2024-03-19T03:31:13.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:13.059+0000 7fc7dec00640 1 -- 172.21.15.117:0/4005000896 shutdown_connections 2024-03-19T03:31:13.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:13.059+0000 7fc7dec00640 1 --2- 172.21.15.117:0/4005000896 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc7d8102120 0x7fc7d819ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:13.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:13.059+0000 7fc7dec00640 1 --2- 172.21.15.117:0/4005000896 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc7c00a2910 0x7fc7c00a4dd0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:13.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:13.059+0000 7fc7dec00640 1 --2- 172.21.15.117:0/4005000896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7d8101760 0x7fc7d819c690 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:13.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:13.059+0000 7fc7dec00640 1 --2- 172.21.15.117:0/4005000896 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc7d8100570 0x7fc7d8069550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:13.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:13.059+0000 7fc7dec00640 1 -- 172.21.15.117:0/4005000896 >> 172.21.15.117:0/4005000896 conn(0x7fc7d80fbb00 msgr2=0x7fc7d8104a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:13.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:13.059+0000 7fc7dec00640 1 -- 172.21.15.117:0/4005000896 shutdown_connections 2024-03-19T03:31:13.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:13.059+0000 7fc7dec00640 1 -- 172.21.15.117:0/4005000896 wait complete. 2024-03-19T03:31:13.087 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-03-19T03:31:12.291038+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":601558,"num_objects":97,"num_object_clones":0,"num_object_copies":293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":167,"num_read_kb":143,"num_write":387,"num_write_kb":3010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":62,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":318,"ondisk_log_size":318,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360988,"kb_used_data":11836,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580836,"statfs":{"total":1151940427776,"available":1151570776064,"internally_reserved":0,"allocated":12120064,"data_stored":7576042,"data_compressed":284628,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19138,"internal_metadata":357414206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6550,"num_objects":73,"num_object_clones":0,"num_object_copies":221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":157,"num_write_kb":53,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005702"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800069+0000","last_change":"2024-03-19T03:30:35.676095+0000","last_active":"2024-03-19T03:30:40.800069+0000","last_peered":"2024-03-19T03:30:40.800069+0000","last_clean":"2024-03-19T03:30:40.800069+0000","last_became_active":"2024-03-19T03:30:35.675772+0000","last_became_peered":"2024-03-19T03:30:35.675772+0000","last_unstale":"2024-03-19T03:30:40.800069+0000","last_undegraded":"2024-03-19T03:30:40.800069+0000","last_fullsized":"2024-03-19T03:30:40.800069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:12:31.902482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.729653+0000","last_change":"2024-03-19T03:30:35.672451+0000","last_active":"2024-03-19T03:30:40.729653+0000","last_peered":"2024-03-19T03:30:40.729653+0000","last_clean":"2024-03-19T03:30:40.729653+0000","last_became_active":"2024-03-19T03:30:35.671516+0000","last_became_peered":"2024-03-19T03:30:35.671516+0000","last_unstale":"2024-03-19T03:30:40.729653+0000","last_undegraded":"2024-03-19T03:30:40.729653+0000","last_fullsized":"2024-03-19T03:30:40.729653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:14:07.021704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.729332+0000","last_change":"2024-03-19T03:30:35.674425+0000","last_active":"2024-03-19T03:30:40.729332+0000","last_peered":"2024-03-19T03:30:40.729332+0000","last_clean":"2024-03-19T03:30:40.729332+0000","last_became_active":"2024-03-19T03:30:35.674012+0000","last_became_peered":"2024-03-19T03:30:35.674012+0000","last_unstale":"2024-03-19T03:30:40.729332+0000","last_undegraded":"2024-03-19T03:30:40.729332+0000","last_fullsized":"2024-03-19T03:30:40.729332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:51:44.850847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799603+0000","last_change":"2024-03-19T03:30:35.649218+0000","last_active":"2024-03-19T03:30:40.799603+0000","last_peered":"2024-03-19T03:30:40.799603+0000","last_clean":"2024-03-19T03:30:40.799603+0000","last_became_active":"2024-03-19T03:30:35.648848+0000","last_became_peered":"2024-03-19T03:30:35.648848+0000","last_unstale":"2024-03-19T03:30:40.799603+0000","last_undegraded":"2024-03-19T03:30:40.799603+0000","last_fullsized":"2024-03-19T03:30:40.799603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:24.204126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799449+0000","last_change":"2024-03-19T03:30:35.670208+0000","last_active":"2024-03-19T03:30:40.799449+0000","last_peered":"2024-03-19T03:30:40.799449+0000","last_clean":"2024-03-19T03:30:40.799449+0000","last_became_active":"2024-03-19T03:30:35.669939+0000","last_became_peered":"2024-03-19T03:30:35.669939+0000","last_unstale":"2024-03-19T03:30:40.799449+0000","last_undegraded":"2024-03-19T03:30:40.799449+0000","last_fullsized":"2024-03-19T03:30:40.799449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:59.687262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800396+0000","last_change":"2024-03-19T03:30:35.662329+0000","last_active":"2024-03-19T03:30:40.800396+0000","last_peered":"2024-03-19T03:30:40.800396+0000","last_clean":"2024-03-19T03:30:40.800396+0000","last_became_active":"2024-03-19T03:30:35.661969+0000","last_became_peered":"2024-03-19T03:30:35.661969+0000","last_unstale":"2024-03-19T03:30:40.800396+0000","last_undegraded":"2024-03-19T03:30:40.800396+0000","last_fullsized":"2024-03-19T03:30:40.800396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:29.265017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.723926+0000","last_change":"2024-03-19T03:30:35.662586+0000","last_active":"2024-03-19T03:30:40.723926+0000","last_peered":"2024-03-19T03:30:40.723926+0000","last_clean":"2024-03-19T03:30:40.723926+0000","last_became_active":"2024-03-19T03:30:35.661774+0000","last_became_peered":"2024-03-19T03:30:35.661774+0000","last_unstale":"2024-03-19T03:30:40.723926+0000","last_undegraded":"2024-03-19T03:30:40.723926+0000","last_fullsized":"2024-03-19T03:30:40.723926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:49.258178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800781+0000","last_change":"2024-03-19T03:30:35.638512+0000","last_active":"2024-03-19T03:30:40.800781+0000","last_peered":"2024-03-19T03:30:40.800781+0000","last_clean":"2024-03-19T03:30:40.800781+0000","last_became_active":"2024-03-19T03:30:35.637205+0000","last_became_peered":"2024-03-19T03:30:35.637205+0000","last_unstale":"2024-03-19T03:30:40.800781+0000","last_undegraded":"2024-03-19T03:30:40.800781+0000","last_fullsized":"2024-03-19T03:30:40.800781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:29:34.578520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799770+0000","last_change":"2024-03-19T03:30:35.650681+0000","last_active":"2024-03-19T03:30:40.799770+0000","last_peered":"2024-03-19T03:30:40.799770+0000","last_clean":"2024-03-19T03:30:40.799770+0000","last_became_active":"2024-03-19T03:30:35.650324+0000","last_became_peered":"2024-03-19T03:30:35.650324+0000","last_unstale":"2024-03-19T03:30:40.799770+0000","last_undegraded":"2024-03-19T03:30:40.799770+0000","last_fullsized":"2024-03-19T03:30:40.799770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:48:23.413530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800642+0000","last_change":"2024-03-19T03:30:35.650240+0000","last_active":"2024-03-19T03:30:40.800642+0000","last_peered":"2024-03-19T03:30:40.800642+0000","last_clean":"2024-03-19T03:30:40.800642+0000","last_became_active":"2024-03-19T03:30:35.649888+0000","last_became_peered":"2024-03-19T03:30:35.649888+0000","last_unstale":"2024-03-19T03:30:40.800642+0000","last_undegraded":"2024-03-19T03:30:40.800642+0000","last_fullsized":"2024-03-19T03:30:40.800642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:55:30.110289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.724094+0000","last_change":"2024-03-19T03:30:35.657058+0000","last_active":"2024-03-19T03:30:40.724094+0000","last_peered":"2024-03-19T03:30:40.724094+0000","last_clean":"2024-03-19T03:30:40.724094+0000","last_became_active":"2024-03-19T03:30:35.656678+0000","last_became_peered":"2024-03-19T03:30:35.656678+0000","last_unstale":"2024-03-19T03:30:40.724094+0000","last_undegraded":"2024-03-19T03:30:40.724094+0000","last_fullsized":"2024-03-19T03:30:40.724094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:10.211852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799551+0000","last_change":"2024-03-19T03:30:35.651543+0000","last_active":"2024-03-19T03:30:40.799551+0000","last_peered":"2024-03-19T03:30:40.799551+0000","last_clean":"2024-03-19T03:30:40.799551+0000","last_became_active":"2024-03-19T03:30:35.649514+0000","last_became_peered":"2024-03-19T03:30:35.649514+0000","last_unstale":"2024-03-19T03:30:40.799551+0000","last_undegraded":"2024-03-19T03:30:40.799551+0000","last_fullsized":"2024-03-19T03:30:40.799551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:54:37.607873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.801550+0000","last_change":"2024-03-19T03:30:35.662494+0000","last_active":"2024-03-19T03:30:40.801550+0000","last_peered":"2024-03-19T03:30:40.801550+0000","last_clean":"2024-03-19T03:30:40.801550+0000","last_became_active":"2024-03-19T03:30:35.660596+0000","last_became_peered":"2024-03-19T03:30:35.660596+0000","last_unstale":"2024-03-19T03:30:40.801550+0000","last_undegraded":"2024-03-19T03:30:40.801550+0000","last_fullsized":"2024-03-19T03:30:40.801550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:25:59.184025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.728184+0000","last_change":"2024-03-19T03:30:35.675751+0000","last_active":"2024-03-19T03:30:40.728184+0000","last_peered":"2024-03-19T03:30:40.728184+0000","last_clean":"2024-03-19T03:30:40.728184+0000","last_became_active":"2024-03-19T03:30:35.674313+0000","last_became_peered":"2024-03-19T03:30:35.674313+0000","last_unstale":"2024-03-19T03:30:40.728184+0000","last_undegraded":"2024-03-19T03:30:40.728184+0000","last_fullsized":"2024-03-19T03:30:40.728184+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:26:26.433958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.728403+0000","last_change":"2024-03-19T03:30:35.672628+0000","last_active":"2024-03-19T03:30:40.728403+0000","last_peered":"2024-03-19T03:30:40.728403+0000","last_clean":"2024-03-19T03:30:40.728403+0000","last_became_active":"2024-03-19T03:30:35.672246+0000","last_became_peered":"2024-03-19T03:30:35.672246+0000","last_unstale":"2024-03-19T03:30:40.728403+0000","last_undegraded":"2024-03-19T03:30:40.728403+0000","last_fullsized":"2024-03-19T03:30:40.728403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:07:59.891108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.798739+0000","last_change":"2024-03-19T03:30:35.678718+0000","last_active":"2024-03-19T03:30:40.798739+0000","last_peered":"2024-03-19T03:30:40.798739+0000","last_clean":"2024-03-19T03:30:40.798739+0000","last_became_active":"2024-03-19T03:30:35.678461+0000","last_became_peered":"2024-03-19T03:30:35.678461+0000","last_unstale":"2024-03-19T03:30:40.798739+0000","last_undegraded":"2024-03-19T03:30:40.798739+0000","last_fullsized":"2024-03-19T03:30:40.798739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:19:39.079048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799956+0000","last_change":"2024-03-19T03:30:35.675847+0000","last_active":"2024-03-19T03:30:40.799956+0000","last_peered":"2024-03-19T03:30:40.799956+0000","last_clean":"2024-03-19T03:30:40.799956+0000","last_became_active":"2024-03-19T03:30:35.675204+0000","last_became_peered":"2024-03-19T03:30:35.675204+0000","last_unstale":"2024-03-19T03:30:40.799956+0000","last_undegraded":"2024-03-19T03:30:40.799956+0000","last_fullsized":"2024-03-19T03:30:40.799956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:08.732536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.802017+0000","last_change":"2024-03-19T03:30:35.663110+0000","last_active":"2024-03-19T03:30:40.802017+0000","last_peered":"2024-03-19T03:30:40.802017+0000","last_clean":"2024-03-19T03:30:40.802017+0000","last_became_active":"2024-03-19T03:30:35.658376+0000","last_became_peered":"2024-03-19T03:30:35.658376+0000","last_unstale":"2024-03-19T03:30:40.802017+0000","last_undegraded":"2024-03-19T03:30:40.802017+0000","last_fullsized":"2024-03-19T03:30:40.802017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:29:44.658681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.724720+0000","last_change":"2024-03-19T03:30:35.661154+0000","last_active":"2024-03-19T03:30:40.724720+0000","last_peered":"2024-03-19T03:30:40.724720+0000","last_clean":"2024-03-19T03:30:40.724720+0000","last_became_active":"2024-03-19T03:30:35.660794+0000","last_became_peered":"2024-03-19T03:30:35.660794+0000","last_unstale":"2024-03-19T03:30:40.724720+0000","last_undegraded":"2024-03-19T03:30:40.724720+0000","last_fullsized":"2024-03-19T03:30:40.724720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:26:29.803827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.724176+0000","last_change":"2024-03-19T03:30:39.735971+0000","last_active":"2024-03-19T03:30:40.724176+0000","last_peered":"2024-03-19T03:30:40.724176+0000","last_clean":"2024-03-19T03:30:40.724176+0000","last_became_active":"2024-03-19T03:30:39.735585+0000","last_became_peered":"2024-03-19T03:30:39.735585+0000","last_unstale":"2024-03-19T03:30:40.724176+0000","last_undegraded":"2024-03-19T03:30:40.724176+0000","last_fullsized":"2024-03-19T03:30:40.724176+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:30:15.969148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:10.003637+0000","last_change":"2024-03-19T03:30:34.632177+0000","last_active":"2024-03-19T03:31:10.003637+0000","last_peered":"2024-03-19T03:31:10.003637+0000","last_clean":"2024-03-19T03:31:10.003637+0000","last_became_active":"2024-03-19T03:30:34.631836+0000","last_became_peered":"2024-03-19T03:30:34.631836+0000","last_unstale":"2024-03-19T03:31:10.003637+0000","last_undegraded":"2024-03-19T03:31:10.003637+0000","last_fullsized":"2024-03-19T03:31:10.003637+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:15:42.883089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.803099+0000","last_change":"2024-03-19T03:30:35.654203+0000","last_active":"2024-03-19T03:30:40.803099+0000","last_peered":"2024-03-19T03:30:40.803099+0000","last_clean":"2024-03-19T03:30:40.803099+0000","last_became_active":"2024-03-19T03:30:35.653665+0000","last_became_peered":"2024-03-19T03:30:35.653665+0000","last_unstale":"2024-03-19T03:30:40.803099+0000","last_undegraded":"2024-03-19T03:30:40.803099+0000","last_fullsized":"2024-03-19T03:30:40.803099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:06:49.153811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799196+0000","last_change":"2024-03-19T03:30:39.733744+0000","last_active":"2024-03-19T03:30:40.799196+0000","last_peered":"2024-03-19T03:30:40.799196+0000","last_clean":"2024-03-19T03:30:40.799196+0000","last_became_active":"2024-03-19T03:30:39.733373+0000","last_became_peered":"2024-03-19T03:30:39.733373+0000","last_unstale":"2024-03-19T03:30:40.799196+0000","last_undegraded":"2024-03-19T03:30:40.799196+0000","last_fullsized":"2024-03-19T03:30:40.799196+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:48:48.186995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.803237+0000","last_change":"2024-03-19T03:30:34.632369+0000","last_active":"2024-03-19T03:30:40.803237+0000","last_peered":"2024-03-19T03:30:40.803237+0000","last_clean":"2024-03-19T03:30:40.803237+0000","last_became_active":"2024-03-19T03:30:34.632027+0000","last_became_peered":"2024-03-19T03:30:34.632027+0000","last_unstale":"2024-03-19T03:30:40.803237+0000","last_undegraded":"2024-03-19T03:30:40.803237+0000","last_fullsized":"2024-03-19T03:30:40.803237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:04:22.575864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.729798+0000","last_change":"2024-03-19T03:30:35.677250+0000","last_active":"2024-03-19T03:30:40.729798+0000","last_peered":"2024-03-19T03:30:40.729798+0000","last_clean":"2024-03-19T03:30:40.729798+0000","last_became_active":"2024-03-19T03:30:35.676741+0000","last_became_peered":"2024-03-19T03:30:35.676741+0000","last_unstale":"2024-03-19T03:30:40.729798+0000","last_undegraded":"2024-03-19T03:30:40.729798+0000","last_fullsized":"2024-03-19T03:30:40.729798+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:52:06.095738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799273+0000","last_change":"2024-03-19T03:30:39.733501+0000","last_active":"2024-03-19T03:30:40.799273+0000","last_peered":"2024-03-19T03:30:40.799273+0000","last_clean":"2024-03-19T03:30:40.799273+0000","last_became_active":"2024-03-19T03:30:39.733102+0000","last_became_peered":"2024-03-19T03:30:39.733102+0000","last_unstale":"2024-03-19T03:30:40.799273+0000","last_undegraded":"2024-03-19T03:30:40.799273+0000","last_fullsized":"2024-03-19T03:30:40.799273+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:35:06.004381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:09.003240+0000","last_change":"2024-03-19T03:30:34.627192+0000","last_active":"2024-03-19T03:31:09.003240+0000","last_peered":"2024-03-19T03:31:09.003240+0000","last_clean":"2024-03-19T03:31:09.003240+0000","last_became_active":"2024-03-19T03:30:34.626730+0000","last_became_peered":"2024-03-19T03:30:34.626730+0000","last_unstale":"2024-03-19T03:31:09.003240+0000","last_undegraded":"2024-03-19T03:31:09.003240+0000","last_fullsized":"2024-03-19T03:31:09.003240+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:20:25.816500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799710+0000","last_change":"2024-03-19T03:30:35.663783+0000","last_active":"2024-03-19T03:30:40.799710+0000","last_peered":"2024-03-19T03:30:40.799710+0000","last_clean":"2024-03-19T03:30:40.799710+0000","last_became_active":"2024-03-19T03:30:35.663472+0000","last_became_peered":"2024-03-19T03:30:35.663472+0000","last_unstale":"2024-03-19T03:30:40.799710+0000","last_undegraded":"2024-03-19T03:30:40.799710+0000","last_fullsized":"2024-03-19T03:30:40.799710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:51:15.211113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799308+0000","last_change":"2024-03-19T03:30:39.731492+0000","last_active":"2024-03-19T03:30:40.799308+0000","last_peered":"2024-03-19T03:30:40.799308+0000","last_clean":"2024-03-19T03:30:40.799308+0000","last_became_active":"2024-03-19T03:30:39.731168+0000","last_became_peered":"2024-03-19T03:30:39.731168+0000","last_unstale":"2024-03-19T03:30:40.799308+0000","last_undegraded":"2024-03-19T03:30:40.799308+0000","last_fullsized":"2024-03-19T03:30:40.799308+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:11:48.281607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:09.003327+0000","last_change":"2024-03-19T03:30:34.627589+0000","last_active":"2024-03-19T03:31:09.003327+0000","last_peered":"2024-03-19T03:31:09.003327+0000","last_clean":"2024-03-19T03:31:09.003327+0000","last_became_active":"2024-03-19T03:30:34.627229+0000","last_became_peered":"2024-03-19T03:30:34.627229+0000","last_unstale":"2024-03-19T03:31:09.003327+0000","last_undegraded":"2024-03-19T03:31:09.003327+0000","last_fullsized":"2024-03-19T03:31:09.003327+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:53:23.128932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800936+0000","last_change":"2024-03-19T03:30:35.650954+0000","last_active":"2024-03-19T03:30:40.800936+0000","last_peered":"2024-03-19T03:30:40.800936+0000","last_clean":"2024-03-19T03:30:40.800936+0000","last_became_active":"2024-03-19T03:30:35.650268+0000","last_became_peered":"2024-03-19T03:30:35.650268+0000","last_unstale":"2024-03-19T03:30:40.800936+0000","last_undegraded":"2024-03-19T03:30:40.800936+0000","last_fullsized":"2024-03-19T03:30:40.800936+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T15:22:28.132438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.724455+0000","last_change":"2024-03-19T03:30:39.727351+0000","last_active":"2024-03-19T03:30:40.724455+0000","last_peered":"2024-03-19T03:30:40.724455+0000","last_clean":"2024-03-19T03:30:40.724455+0000","last_became_active":"2024-03-19T03:30:39.726868+0000","last_became_peered":"2024-03-19T03:30:39.726868+0000","last_unstale":"2024-03-19T03:30:40.724455+0000","last_undegraded":"2024-03-19T03:30:40.724455+0000","last_fullsized":"2024-03-19T03:30:40.724455+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:40:59.003064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:08.996425+0000","last_change":"2024-03-19T03:30:34.623911+0000","last_active":"2024-03-19T03:31:08.996425+0000","last_peered":"2024-03-19T03:31:08.996425+0000","last_clean":"2024-03-19T03:31:08.996425+0000","last_became_active":"2024-03-19T03:30:34.623548+0000","last_became_peered":"2024-03-19T03:30:34.623548+0000","last_unstale":"2024-03-19T03:31:08.996425+0000","last_undegraded":"2024-03-19T03:31:08.996425+0000","last_fullsized":"2024-03-19T03:31:08.996425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:29:56.616353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799771+0000","last_change":"2024-03-19T03:30:35.661874+0000","last_active":"2024-03-19T03:30:40.799771+0000","last_peered":"2024-03-19T03:30:40.799771+0000","last_clean":"2024-03-19T03:30:40.799771+0000","last_became_active":"2024-03-19T03:30:35.661521+0000","last_became_peered":"2024-03-19T03:30:35.661521+0000","last_unstale":"2024-03-19T03:30:40.799771+0000","last_undegraded":"2024-03-19T03:30:40.799771+0000","last_fullsized":"2024-03-19T03:30:40.799771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:19:35.017020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.724476+0000","last_change":"2024-03-19T03:30:39.748197+0000","last_active":"2024-03-19T03:30:40.724476+0000","last_peered":"2024-03-19T03:30:40.724476+0000","last_clean":"2024-03-19T03:30:40.724476+0000","last_became_active":"2024-03-19T03:30:39.747833+0000","last_became_peered":"2024-03-19T03:30:39.747833+0000","last_unstale":"2024-03-19T03:30:40.724476+0000","last_undegraded":"2024-03-19T03:30:40.724476+0000","last_fullsized":"2024-03-19T03:30:40.724476+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:29:19.213403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:07.477261+0000","last_change":"2024-03-19T03:30:34.623491+0000","last_active":"2024-03-19T03:31:07.477261+0000","last_peered":"2024-03-19T03:31:07.477261+0000","last_clean":"2024-03-19T03:31:07.477261+0000","last_became_active":"2024-03-19T03:30:34.623152+0000","last_became_peered":"2024-03-19T03:30:34.623152+0000","last_unstale":"2024-03-19T03:31:07.477261+0000","last_undegraded":"2024-03-19T03:31:07.477261+0000","last_fullsized":"2024-03-19T03:31:07.477261+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:00:25.530926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800823+0000","last_change":"2024-03-19T03:30:35.651528+0000","last_active":"2024-03-19T03:30:40.800823+0000","last_peered":"2024-03-19T03:30:40.800823+0000","last_clean":"2024-03-19T03:30:40.800823+0000","last_became_active":"2024-03-19T03:30:35.650240+0000","last_became_peered":"2024-03-19T03:30:35.650240+0000","last_unstale":"2024-03-19T03:30:40.800823+0000","last_undegraded":"2024-03-19T03:30:40.800823+0000","last_fullsized":"2024-03-19T03:30:40.800823+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:25.123570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799874+0000","last_change":"2024-03-19T03:30:39.740630+0000","last_active":"2024-03-19T03:30:40.799874+0000","last_peered":"2024-03-19T03:30:40.799874+0000","last_clean":"2024-03-19T03:30:40.799874+0000","last_became_active":"2024-03-19T03:30:39.740170+0000","last_became_peered":"2024-03-19T03:30:39.740170+0000","last_unstale":"2024-03-19T03:30:40.799874+0000","last_undegraded":"2024-03-19T03:30:40.799874+0000","last_fullsized":"2024-03-19T03:30:40.799874+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:23:49.415718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:07.483948+0000","last_change":"2024-03-19T03:30:34.635173+0000","last_active":"2024-03-19T03:31:07.483948+0000","last_peered":"2024-03-19T03:31:07.483948+0000","last_clean":"2024-03-19T03:31:07.483948+0000","last_became_active":"2024-03-19T03:30:34.633809+0000","last_became_peered":"2024-03-19T03:30:34.633809+0000","last_unstale":"2024-03-19T03:31:07.483948+0000","last_undegraded":"2024-03-19T03:31:07.483948+0000","last_fullsized":"2024-03-19T03:31:07.483948+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:03:57.409707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800659+0000","last_change":"2024-03-19T03:30:35.665660+0000","last_active":"2024-03-19T03:30:40.800659+0000","last_peered":"2024-03-19T03:30:40.800659+0000","last_clean":"2024-03-19T03:30:40.800659+0000","last_became_active":"2024-03-19T03:30:35.665290+0000","last_became_peered":"2024-03-19T03:30:35.665290+0000","last_unstale":"2024-03-19T03:30:40.800659+0000","last_undegraded":"2024-03-19T03:30:40.800659+0000","last_fullsized":"2024-03-19T03:30:40.800659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:20.333009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.729721+0000","last_change":"2024-03-19T03:30:39.744391+0000","last_active":"2024-03-19T03:30:40.729721+0000","last_peered":"2024-03-19T03:30:40.729721+0000","last_clean":"2024-03-19T03:30:40.729721+0000","last_became_active":"2024-03-19T03:30:39.744000+0000","last_became_peered":"2024-03-19T03:30:39.744000+0000","last_unstale":"2024-03-19T03:30:40.729721+0000","last_undegraded":"2024-03-19T03:30:40.729721+0000","last_fullsized":"2024-03-19T03:30:40.729721+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:17:06.124286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:08.994577+0000","last_change":"2024-03-19T03:30:34.621811+0000","last_active":"2024-03-19T03:31:08.994577+0000","last_peered":"2024-03-19T03:31:08.994577+0000","last_clean":"2024-03-19T03:31:08.994577+0000","last_became_active":"2024-03-19T03:30:34.621308+0000","last_became_peered":"2024-03-19T03:30:34.621308+0000","last_unstale":"2024-03-19T03:31:08.994577+0000","last_undegraded":"2024-03-19T03:31:08.994577+0000","last_fullsized":"2024-03-19T03:31:08.994577+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:36:45.895720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.802438+0000","last_change":"2024-03-19T03:30:35.673683+0000","last_active":"2024-03-19T03:30:40.802438+0000","last_peered":"2024-03-19T03:30:40.802438+0000","last_clean":"2024-03-19T03:30:40.802438+0000","last_became_active":"2024-03-19T03:30:35.670489+0000","last_became_peered":"2024-03-19T03:30:35.670489+0000","last_unstale":"2024-03-19T03:30:40.802438+0000","last_undegraded":"2024-03-19T03:30:40.802438+0000","last_fullsized":"2024-03-19T03:30:40.802438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:48:30.521454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.801202+0000","last_change":"2024-03-19T03:30:39.726358+0000","last_active":"2024-03-19T03:30:40.801202+0000","last_peered":"2024-03-19T03:30:40.801202+0000","last_clean":"2024-03-19T03:30:40.801202+0000","last_became_active":"2024-03-19T03:30:39.725720+0000","last_became_peered":"2024-03-19T03:30:39.725720+0000","last_unstale":"2024-03-19T03:30:40.801202+0000","last_undegraded":"2024-03-19T03:30:40.801202+0000","last_fullsized":"2024-03-19T03:30:40.801202+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:01:08.759794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:06.474278+0000","last_change":"2024-03-19T03:30:34.626336+0000","last_active":"2024-03-19T03:31:06.474278+0000","last_peered":"2024-03-19T03:31:06.474278+0000","last_clean":"2024-03-19T03:31:06.474278+0000","last_became_active":"2024-03-19T03:30:34.625991+0000","last_became_peered":"2024-03-19T03:30:34.625991+0000","last_unstale":"2024-03-19T03:31:06.474278+0000","last_undegraded":"2024-03-19T03:31:06.474278+0000","last_fullsized":"2024-03-19T03:31:06.474278+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:40:58.791759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799988+0000","last_change":"2024-03-19T03:30:35.667001+0000","last_active":"2024-03-19T03:30:40.799988+0000","last_peered":"2024-03-19T03:30:40.799988+0000","last_clean":"2024-03-19T03:30:40.799988+0000","last_became_active":"2024-03-19T03:30:35.666586+0000","last_became_peered":"2024-03-19T03:30:35.666586+0000","last_unstale":"2024-03-19T03:30:40.799988+0000","last_undegraded":"2024-03-19T03:30:40.799988+0000","last_fullsized":"2024-03-19T03:30:40.799988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:40:30.100860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.802488+0000","last_change":"2024-03-19T03:30:39.745267+0000","last_active":"2024-03-19T03:30:40.802488+0000","last_peered":"2024-03-19T03:30:40.802488+0000","last_clean":"2024-03-19T03:30:40.802488+0000","last_became_active":"2024-03-19T03:30:39.743434+0000","last_became_peered":"2024-03-19T03:30:39.743434+0000","last_unstale":"2024-03-19T03:30:40.802488+0000","last_undegraded":"2024-03-19T03:30:40.802488+0000","last_fullsized":"2024-03-19T03:30:40.802488+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:53:37.583886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:09.001673+0000","last_change":"2024-03-19T03:30:34.617769+0000","last_active":"2024-03-19T03:31:09.001673+0000","last_peered":"2024-03-19T03:31:09.001673+0000","last_clean":"2024-03-19T03:31:09.001673+0000","last_became_active":"2024-03-19T03:30:34.617428+0000","last_became_peered":"2024-03-19T03:30:34.617428+0000","last_unstale":"2024-03-19T03:31:09.001673+0000","last_undegraded":"2024-03-19T03:31:09.001673+0000","last_fullsized":"2024-03-19T03:31:09.001673+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:58:09.181320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.729801+0000","last_change":"2024-03-19T03:30:35.676278+0000","last_active":"2024-03-19T03:30:40.729801+0000","last_peered":"2024-03-19T03:30:40.729801+0000","last_clean":"2024-03-19T03:30:40.729801+0000","last_became_active":"2024-03-19T03:30:35.673773+0000","last_became_peered":"2024-03-19T03:30:35.673773+0000","last_unstale":"2024-03-19T03:30:40.729801+0000","last_undegraded":"2024-03-19T03:30:40.729801+0000","last_fullsized":"2024-03-19T03:30:40.729801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:39:12.983732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800083+0000","last_change":"2024-03-19T03:30:39.733273+0000","last_active":"2024-03-19T03:30:40.800083+0000","last_peered":"2024-03-19T03:30:40.800083+0000","last_clean":"2024-03-19T03:30:40.800083+0000","last_became_active":"2024-03-19T03:30:39.732885+0000","last_became_peered":"2024-03-19T03:30:39.732885+0000","last_unstale":"2024-03-19T03:30:40.800083+0000","last_undegraded":"2024-03-19T03:30:40.800083+0000","last_fullsized":"2024-03-19T03:30:40.800083+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:48:10.446577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:08.494914+0000","last_change":"2024-03-19T03:30:34.637076+0000","last_active":"2024-03-19T03:31:08.494914+0000","last_peered":"2024-03-19T03:31:08.494914+0000","last_clean":"2024-03-19T03:31:08.494914+0000","last_became_active":"2024-03-19T03:30:34.636642+0000","last_became_peered":"2024-03-19T03:30:34.636642+0000","last_unstale":"2024-03-19T03:31:08.494914+0000","last_undegraded":"2024-03-19T03:31:08.494914+0000","last_fullsized":"2024-03-19T03:31:08.494914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:26:17.494081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799830+0000","last_change":"2024-03-19T03:30:35.651739+0000","last_active":"2024-03-19T03:30:40.799830+0000","last_peered":"2024-03-19T03:30:40.799830+0000","last_clean":"2024-03-19T03:30:40.799830+0000","last_became_active":"2024-03-19T03:30:35.651419+0000","last_became_peered":"2024-03-19T03:30:35.651419+0000","last_unstale":"2024-03-19T03:30:40.799830+0000","last_undegraded":"2024-03-19T03:30:40.799830+0000","last_fullsized":"2024-03-19T03:30:40.799830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T15:17:55.022050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800590+0000","last_change":"2024-03-19T03:30:39.733228+0000","last_active":"2024-03-19T03:30:40.800590+0000","last_peered":"2024-03-19T03:30:40.800590+0000","last_clean":"2024-03-19T03:30:40.800590+0000","last_became_active":"2024-03-19T03:30:39.732819+0000","last_became_peered":"2024-03-19T03:30:39.732819+0000","last_unstale":"2024-03-19T03:30:40.800590+0000","last_undegraded":"2024-03-19T03:30:40.800590+0000","last_fullsized":"2024-03-19T03:30:40.800590+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:03:43.968209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:06.469000+0000","last_change":"2024-03-19T03:30:34.618323+0000","last_active":"2024-03-19T03:31:06.469000+0000","last_peered":"2024-03-19T03:31:06.469000+0000","last_clean":"2024-03-19T03:31:06.469000+0000","last_became_active":"2024-03-19T03:30:34.617975+0000","last_became_peered":"2024-03-19T03:30:34.617975+0000","last_unstale":"2024-03-19T03:31:06.469000+0000","last_undegraded":"2024-03-19T03:31:06.469000+0000","last_fullsized":"2024-03-19T03:31:06.469000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:37:15.261998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800119+0000","last_change":"2024-03-19T03:30:35.652754+0000","last_active":"2024-03-19T03:30:40.800119+0000","last_peered":"2024-03-19T03:30:40.800119+0000","last_clean":"2024-03-19T03:30:40.800119+0000","last_became_active":"2024-03-19T03:30:35.652415+0000","last_became_peered":"2024-03-19T03:30:35.652415+0000","last_unstale":"2024-03-19T03:30:40.800119+0000","last_undegraded":"2024-03-19T03:30:40.800119+0000","last_fullsized":"2024-03-19T03:30:40.800119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:29:31.838302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.801350+0000","last_change":"2024-03-19T03:30:39.732885+0000","last_active":"2024-03-19T03:30:40.801350+0000","last_peered":"2024-03-19T03:30:40.801350+0000","last_clean":"2024-03-19T03:30:40.801350+0000","last_became_active":"2024-03-19T03:30:39.732449+0000","last_became_peered":"2024-03-19T03:30:39.732449+0000","last_unstale":"2024-03-19T03:30:40.801350+0000","last_undegraded":"2024-03-19T03:30:40.801350+0000","last_fullsized":"2024-03-19T03:30:40.801350+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:08:01.304805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:09.007040+0000","last_change":"2024-03-19T03:30:34.617560+0000","last_active":"2024-03-19T03:31:09.007040+0000","last_peered":"2024-03-19T03:31:09.007040+0000","last_clean":"2024-03-19T03:31:09.007040+0000","last_became_active":"2024-03-19T03:30:34.617234+0000","last_became_peered":"2024-03-19T03:30:34.617234+0000","last_unstale":"2024-03-19T03:31:09.007040+0000","last_undegraded":"2024-03-19T03:31:09.007040+0000","last_fullsized":"2024-03-19T03:31:09.007040+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:00:46.206589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800748+0000","last_change":"2024-03-19T03:30:35.672723+0000","last_active":"2024-03-19T03:30:40.800748+0000","last_peered":"2024-03-19T03:30:40.800748+0000","last_clean":"2024-03-19T03:30:40.800748+0000","last_became_active":"2024-03-19T03:30:35.670828+0000","last_became_peered":"2024-03-19T03:30:35.670828+0000","last_unstale":"2024-03-19T03:30:40.800748+0000","last_undegraded":"2024-03-19T03:30:40.800748+0000","last_fullsized":"2024-03-19T03:30:40.800748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:22:19.254329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800833+0000","last_change":"2024-03-19T03:30:39.738887+0000","last_active":"2024-03-19T03:30:40.800833+0000","last_peered":"2024-03-19T03:30:40.800833+0000","last_clean":"2024-03-19T03:30:40.800833+0000","last_became_active":"2024-03-19T03:30:39.738450+0000","last_became_peered":"2024-03-19T03:30:39.738450+0000","last_unstale":"2024-03-19T03:30:40.800833+0000","last_undegraded":"2024-03-19T03:30:40.800833+0000","last_fullsized":"2024-03-19T03:30:40.800833+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:23:57.550464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:09.007078+0000","last_change":"2024-03-19T03:30:34.634810+0000","last_active":"2024-03-19T03:31:09.007078+0000","last_peered":"2024-03-19T03:31:09.007078+0000","last_clean":"2024-03-19T03:31:09.007078+0000","last_became_active":"2024-03-19T03:30:34.634404+0000","last_became_peered":"2024-03-19T03:30:34.634404+0000","last_unstale":"2024-03-19T03:31:09.007078+0000","last_undegraded":"2024-03-19T03:31:09.007078+0000","last_fullsized":"2024-03-19T03:31:09.007078+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:19:09.231175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.723928+0000","last_change":"2024-03-19T03:30:35.664808+0000","last_active":"2024-03-19T03:30:40.723928+0000","last_peered":"2024-03-19T03:30:40.723928+0000","last_clean":"2024-03-19T03:30:40.723928+0000","last_became_active":"2024-03-19T03:30:35.664431+0000","last_became_peered":"2024-03-19T03:30:35.664431+0000","last_unstale":"2024-03-19T03:30:40.723928+0000","last_undegraded":"2024-03-19T03:30:40.723928+0000","last_fullsized":"2024-03-19T03:30:40.723928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:18:47.623528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.801170+0000","last_change":"2024-03-19T03:30:39.736905+0000","last_active":"2024-03-19T03:30:40.801170+0000","last_peered":"2024-03-19T03:30:40.801170+0000","last_clean":"2024-03-19T03:30:40.801170+0000","last_became_active":"2024-03-19T03:30:39.736329+0000","last_became_peered":"2024-03-19T03:30:39.736329+0000","last_unstale":"2024-03-19T03:30:40.801170+0000","last_undegraded":"2024-03-19T03:30:40.801170+0000","last_fullsized":"2024-03-19T03:30:40.801170+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:12:44.745999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:10.001301+0000","last_change":"2024-03-19T03:30:34.619890+0000","last_active":"2024-03-19T03:31:10.001301+0000","last_peered":"2024-03-19T03:31:10.001301+0000","last_clean":"2024-03-19T03:31:10.001301+0000","last_became_active":"2024-03-19T03:30:34.619518+0000","last_became_peered":"2024-03-19T03:30:34.619518+0000","last_unstale":"2024-03-19T03:31:10.001301+0000","last_undegraded":"2024-03-19T03:31:10.001301+0000","last_fullsized":"2024-03-19T03:31:10.001301+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T15:19:41.314321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.723621+0000","last_change":"2024-03-19T03:30:35.649241+0000","last_active":"2024-03-19T03:30:40.723621+0000","last_peered":"2024-03-19T03:30:40.723621+0000","last_clean":"2024-03-19T03:30:40.723621+0000","last_became_active":"2024-03-19T03:30:35.648883+0000","last_became_peered":"2024-03-19T03:30:35.648883+0000","last_unstale":"2024-03-19T03:30:40.723621+0000","last_undegraded":"2024-03-19T03:30:40.723621+0000","last_fullsized":"2024-03-19T03:30:40.723621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:43:44.219322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.801794+0000","last_change":"2024-03-19T03:30:39.741401+0000","last_active":"2024-03-19T03:30:40.801794+0000","last_peered":"2024-03-19T03:30:40.801794+0000","last_clean":"2024-03-19T03:30:40.801794+0000","last_became_active":"2024-03-19T03:30:39.741019+0000","last_became_peered":"2024-03-19T03:30:39.741019+0000","last_unstale":"2024-03-19T03:30:40.801794+0000","last_undegraded":"2024-03-19T03:30:40.801794+0000","last_fullsized":"2024-03-19T03:30:40.801794+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:29:12.824867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:08.990202+0000","last_change":"2024-03-19T03:30:34.635076+0000","last_active":"2024-03-19T03:31:08.990202+0000","last_peered":"2024-03-19T03:31:08.990202+0000","last_clean":"2024-03-19T03:31:08.990202+0000","last_became_active":"2024-03-19T03:30:34.634680+0000","last_became_peered":"2024-03-19T03:30:34.634680+0000","last_unstale":"2024-03-19T03:31:08.990202+0000","last_undegraded":"2024-03-19T03:31:08.990202+0000","last_fullsized":"2024-03-19T03:31:08.990202+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:16:38.874086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799993+0000","last_change":"2024-03-19T03:30:35.651912+0000","last_active":"2024-03-19T03:30:40.799993+0000","last_peered":"2024-03-19T03:30:40.799993+0000","last_clean":"2024-03-19T03:30:40.799993+0000","last_became_active":"2024-03-19T03:30:35.651563+0000","last_became_peered":"2024-03-19T03:30:35.651563+0000","last_unstale":"2024-03-19T03:30:40.799993+0000","last_undegraded":"2024-03-19T03:30:40.799993+0000","last_fullsized":"2024-03-19T03:30:40.799993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:41:05.677230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799949+0000","last_change":"2024-03-19T03:30:39.737466+0000","last_active":"2024-03-19T03:30:40.799949+0000","last_peered":"2024-03-19T03:30:40.799949+0000","last_clean":"2024-03-19T03:30:40.799949+0000","last_became_active":"2024-03-19T03:30:39.737126+0000","last_became_peered":"2024-03-19T03:30:39.737126+0000","last_unstale":"2024-03-19T03:30:40.799949+0000","last_undegraded":"2024-03-19T03:30:40.799949+0000","last_fullsized":"2024-03-19T03:30:40.799949+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:52:25.030107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:08.474804+0000","last_change":"2024-03-19T03:30:34.622729+0000","last_active":"2024-03-19T03:31:08.474804+0000","last_peered":"2024-03-19T03:31:08.474804+0000","last_clean":"2024-03-19T03:31:08.474804+0000","last_became_active":"2024-03-19T03:30:34.622349+0000","last_became_peered":"2024-03-19T03:30:34.622349+0000","last_unstale":"2024-03-19T03:31:08.474804+0000","last_undegraded":"2024-03-19T03:31:08.474804+0000","last_fullsized":"2024-03-19T03:31:08.474804+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:28.893702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.801057+0000","last_change":"2024-03-19T03:30:35.654167+0000","last_active":"2024-03-19T03:30:40.801057+0000","last_peered":"2024-03-19T03:30:40.801057+0000","last_clean":"2024-03-19T03:30:40.801057+0000","last_became_active":"2024-03-19T03:30:35.653782+0000","last_became_peered":"2024-03-19T03:30:35.653782+0000","last_unstale":"2024-03-19T03:30:40.801057+0000","last_undegraded":"2024-03-19T03:30:40.801057+0000","last_fullsized":"2024-03-19T03:30:40.801057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:26:47.584353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799710+0000","last_change":"2024-03-19T03:30:39.728505+0000","last_active":"2024-03-19T03:30:40.799710+0000","last_peered":"2024-03-19T03:30:40.799710+0000","last_clean":"2024-03-19T03:30:40.799710+0000","last_became_active":"2024-03-19T03:30:39.728095+0000","last_became_peered":"2024-03-19T03:30:39.728095+0000","last_unstale":"2024-03-19T03:30:40.799710+0000","last_undegraded":"2024-03-19T03:30:40.799710+0000","last_fullsized":"2024-03-19T03:30:40.799710+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:59:44.189992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.802826+0000","last_change":"2024-03-19T03:30:34.642738+0000","last_active":"2024-03-19T03:30:40.802826+0000","last_peered":"2024-03-19T03:30:40.802826+0000","last_clean":"2024-03-19T03:30:40.802826+0000","last_became_active":"2024-03-19T03:30:34.642173+0000","last_became_peered":"2024-03-19T03:30:34.642173+0000","last_unstale":"2024-03-19T03:30:40.802826+0000","last_undegraded":"2024-03-19T03:30:40.802826+0000","last_fullsized":"2024-03-19T03:30:40.802826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:17:51.633284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800650+0000","last_change":"2024-03-19T03:30:35.663676+0000","last_active":"2024-03-19T03:30:40.800650+0000","last_peered":"2024-03-19T03:30:40.800650+0000","last_clean":"2024-03-19T03:30:40.800650+0000","last_became_active":"2024-03-19T03:30:35.663313+0000","last_became_peered":"2024-03-19T03:30:35.663313+0000","last_unstale":"2024-03-19T03:30:40.800650+0000","last_undegraded":"2024-03-19T03:30:40.800650+0000","last_fullsized":"2024-03-19T03:30:40.800650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:02:25.933784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.801529+0000","last_change":"2024-03-19T03:30:39.744967+0000","last_active":"2024-03-19T03:30:40.801529+0000","last_peered":"2024-03-19T03:30:40.801529+0000","last_clean":"2024-03-19T03:30:40.801529+0000","last_became_active":"2024-03-19T03:30:39.744542+0000","last_became_peered":"2024-03-19T03:30:39.744542+0000","last_unstale":"2024-03-19T03:30:40.801529+0000","last_undegraded":"2024-03-19T03:30:40.801529+0000","last_fullsized":"2024-03-19T03:30:40.801529+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:56:54.699749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:10.004351+0000","last_change":"2024-03-19T03:30:34.622225+0000","last_active":"2024-03-19T03:31:10.004351+0000","last_peered":"2024-03-19T03:31:10.004351+0000","last_clean":"2024-03-19T03:31:10.004351+0000","last_became_active":"2024-03-19T03:30:34.621836+0000","last_became_peered":"2024-03-19T03:30:34.621836+0000","last_unstale":"2024-03-19T03:31:10.004351+0000","last_undegraded":"2024-03-19T03:31:10.004351+0000","last_fullsized":"2024-03-19T03:31:10.004351+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:20:35.854632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800182+0000","last_change":"2024-03-19T03:30:35.675961+0000","last_active":"2024-03-19T03:30:40.800182+0000","last_peered":"2024-03-19T03:30:40.800182+0000","last_clean":"2024-03-19T03:30:40.800182+0000","last_became_active":"2024-03-19T03:30:35.675603+0000","last_became_peered":"2024-03-19T03:30:35.675603+0000","last_unstale":"2024-03-19T03:30:40.800182+0000","last_undegraded":"2024-03-19T03:30:40.800182+0000","last_fullsized":"2024-03-19T03:30:40.800182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:39:47.429027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.801751+0000","last_change":"2024-03-19T03:30:39.746747+0000","last_active":"2024-03-19T03:30:40.801751+0000","last_peered":"2024-03-19T03:30:40.801751+0000","last_clean":"2024-03-19T03:30:40.801751+0000","last_became_active":"2024-03-19T03:30:39.745553+0000","last_became_peered":"2024-03-19T03:30:39.745553+0000","last_unstale":"2024-03-19T03:30:40.801751+0000","last_undegraded":"2024-03-19T03:30:40.801751+0000","last_fullsized":"2024-03-19T03:30:40.801751+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:14:56.211061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800676+0000","last_change":"2024-03-19T03:30:35.649795+0000","last_active":"2024-03-19T03:30:40.800676+0000","last_peered":"2024-03-19T03:30:40.800676+0000","last_clean":"2024-03-19T03:30:40.800676+0000","last_became_active":"2024-03-19T03:30:35.649417+0000","last_became_peered":"2024-03-19T03:30:35.649417+0000","last_unstale":"2024-03-19T03:30:40.800676+0000","last_undegraded":"2024-03-19T03:30:40.800676+0000","last_fullsized":"2024-03-19T03:30:40.800676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:24.065817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:10.001028+0000","last_change":"2024-03-19T03:30:34.637446+0000","last_active":"2024-03-19T03:31:10.001028+0000","last_peered":"2024-03-19T03:31:10.001028+0000","last_clean":"2024-03-19T03:31:10.001028+0000","last_became_active":"2024-03-19T03:30:34.637083+0000","last_became_peered":"2024-03-19T03:30:34.637083+0000","last_unstale":"2024-03-19T03:31:10.001028+0000","last_undegraded":"2024-03-19T03:31:10.001028+0000","last_fullsized":"2024-03-19T03:31:10.001028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:51:02.090890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.987872+0000","last_change":"2024-03-19T03:30:39.726337+0000","last_active":"2024-03-19T03:30:40.987872+0000","last_peered":"2024-03-19T03:30:40.987872+0000","last_clean":"2024-03-19T03:30:40.987872+0000","last_became_active":"2024-03-19T03:30:39.725919+0000","last_became_peered":"2024-03-19T03:30:39.725919+0000","last_unstale":"2024-03-19T03:30:40.987872+0000","last_undegraded":"2024-03-19T03:30:40.987872+0000","last_fullsized":"2024-03-19T03:30:40.987872+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:19:48.993790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.802078+0000","last_change":"2024-03-19T03:30:35.670002+0000","last_active":"2024-03-19T03:30:40.802078+0000","last_peered":"2024-03-19T03:30:40.802078+0000","last_clean":"2024-03-19T03:30:40.802078+0000","last_became_active":"2024-03-19T03:30:35.653208+0000","last_became_peered":"2024-03-19T03:30:35.653208+0000","last_unstale":"2024-03-19T03:30:40.802078+0000","last_undegraded":"2024-03-19T03:30:40.802078+0000","last_fullsized":"2024-03-19T03:30:40.802078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:37:15.246977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:09.005888+0000","last_change":"2024-03-19T03:30:34.625107+0000","last_active":"2024-03-19T03:31:09.005888+0000","last_peered":"2024-03-19T03:31:09.005888+0000","last_clean":"2024-03-19T03:31:09.005888+0000","last_became_active":"2024-03-19T03:30:34.624711+0000","last_became_peered":"2024-03-19T03:30:34.624711+0000","last_unstale":"2024-03-19T03:31:09.005888+0000","last_undegraded":"2024-03-19T03:31:09.005888+0000","last_fullsized":"2024-03-19T03:31:09.005888+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:39:33.880100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":346,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.723512+0000","last_change":"2024-03-19T03:29:11.374136+0000","last_active":"2024-03-19T03:30:40.723512+0000","last_peered":"2024-03-19T03:30:40.723512+0000","last_clean":"2024-03-19T03:30:40.723512+0000","last_became_active":"2024-03-19T03:29:11.373691+0000","last_became_peered":"2024-03-19T03:29:11.373691+0000","last_unstale":"2024-03-19T03:30:40.723512+0000","last_undegraded":"2024-03-19T03:30:40.723512+0000","last_fullsized":"2024-03-19T03:30:40.723512+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:25:19.274380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:25:19.274380+0000","last_clean_scrub_stamp":"2024-03-19T03:25:19.274380+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:29:59.731772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.801172+0000","last_change":"2024-03-19T03:30:39.744391+0000","last_active":"2024-03-19T03:30:40.801172+0000","last_peered":"2024-03-19T03:30:40.801172+0000","last_clean":"2024-03-19T03:30:40.801172+0000","last_became_active":"2024-03-19T03:30:39.743995+0000","last_became_peered":"2024-03-19T03:30:39.743995+0000","last_unstale":"2024-03-19T03:30:40.801172+0000","last_undegraded":"2024-03-19T03:30:40.801172+0000","last_fullsized":"2024-03-19T03:30:40.801172+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:41:02.714132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.801257+0000","last_change":"2024-03-19T03:30:35.662528+0000","last_active":"2024-03-19T03:30:40.801257+0000","last_peered":"2024-03-19T03:30:40.801257+0000","last_clean":"2024-03-19T03:30:40.801257+0000","last_became_active":"2024-03-19T03:30:35.659782+0000","last_became_peered":"2024-03-19T03:30:35.659782+0000","last_unstale":"2024-03-19T03:30:40.801257+0000","last_undegraded":"2024-03-19T03:30:40.801257+0000","last_fullsized":"2024-03-19T03:30:40.801257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:18:12.838234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:07.477796+0000","last_change":"2024-03-19T03:30:34.620988+0000","last_active":"2024-03-19T03:31:07.477796+0000","last_peered":"2024-03-19T03:31:07.477796+0000","last_clean":"2024-03-19T03:31:07.477796+0000","last_became_active":"2024-03-19T03:30:34.620411+0000","last_became_peered":"2024-03-19T03:30:34.620411+0000","last_unstale":"2024-03-19T03:31:07.477796+0000","last_undegraded":"2024-03-19T03:31:07.477796+0000","last_fullsized":"2024-03-19T03:31:07.477796+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:37:13.275121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800830+0000","last_change":"2024-03-19T03:30:39.733325+0000","last_active":"2024-03-19T03:30:40.800830+0000","last_peered":"2024-03-19T03:30:40.800830+0000","last_clean":"2024-03-19T03:30:40.800830+0000","last_became_active":"2024-03-19T03:30:39.732553+0000","last_became_peered":"2024-03-19T03:30:39.732553+0000","last_unstale":"2024-03-19T03:30:40.800830+0000","last_undegraded":"2024-03-19T03:30:40.800830+0000","last_fullsized":"2024-03-19T03:30:40.800830+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:52:47.403673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:08.991819+0000","last_change":"2024-03-19T03:30:34.637100+0000","last_active":"2024-03-19T03:31:08.991819+0000","last_peered":"2024-03-19T03:31:08.991819+0000","last_clean":"2024-03-19T03:31:08.991819+0000","last_became_active":"2024-03-19T03:30:34.636715+0000","last_became_peered":"2024-03-19T03:30:34.636715+0000","last_unstale":"2024-03-19T03:31:08.991819+0000","last_undegraded":"2024-03-19T03:31:08.991819+0000","last_fullsized":"2024-03-19T03:31:08.991819+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:48:16.212158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.802061+0000","last_change":"2024-03-19T03:30:35.668405+0000","last_active":"2024-03-19T03:30:40.802061+0000","last_peered":"2024-03-19T03:30:40.802061+0000","last_clean":"2024-03-19T03:30:40.802061+0000","last_became_active":"2024-03-19T03:30:35.667972+0000","last_became_peered":"2024-03-19T03:30:35.667972+0000","last_unstale":"2024-03-19T03:30:40.802061+0000","last_undegraded":"2024-03-19T03:30:40.802061+0000","last_fullsized":"2024-03-19T03:30:40.802061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:01:52.816110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800510+0000","last_change":"2024-03-19T03:30:39.747535+0000","last_active":"2024-03-19T03:30:40.800510+0000","last_peered":"2024-03-19T03:30:40.800510+0000","last_clean":"2024-03-19T03:30:40.800510+0000","last_became_active":"2024-03-19T03:30:39.747122+0000","last_became_peered":"2024-03-19T03:30:39.747122+0000","last_unstale":"2024-03-19T03:30:40.800510+0000","last_undegraded":"2024-03-19T03:30:40.800510+0000","last_fullsized":"2024-03-19T03:30:40.800510+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:07:35.642809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'9","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:09.675963+0000","last_change":"2024-03-19T03:30:34.637768+0000","last_active":"2024-03-19T03:31:09.675963+0000","last_peered":"2024-03-19T03:31:09.675963+0000","last_clean":"2024-03-19T03:31:09.675963+0000","last_became_active":"2024-03-19T03:30:34.637431+0000","last_became_peered":"2024-03-19T03:30:34.637431+0000","last_unstale":"2024-03-19T03:31:09.675963+0000","last_undegraded":"2024-03-19T03:31:09.675963+0000","last_fullsized":"2024-03-19T03:31:09.675963+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:22:50.487209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.803278+0000","last_change":"2024-03-19T03:30:35.661352+0000","last_active":"2024-03-19T03:30:40.803278+0000","last_peered":"2024-03-19T03:30:40.803278+0000","last_clean":"2024-03-19T03:30:40.803278+0000","last_became_active":"2024-03-19T03:30:35.658656+0000","last_became_peered":"2024-03-19T03:30:35.658656+0000","last_unstale":"2024-03-19T03:30:40.803278+0000","last_undegraded":"2024-03-19T03:30:40.803278+0000","last_fullsized":"2024-03-19T03:30:40.803278+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:36.532748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.723999+0000","last_change":"2024-03-19T03:30:39.724637+0000","last_active":"2024-03-19T03:30:40.723999+0000","last_peered":"2024-03-19T03:30:40.723999+0000","last_clean":"2024-03-19T03:30:40.723999+0000","last_became_active":"2024-03-19T03:30:39.723842+0000","last_became_peered":"2024-03-19T03:30:39.723842+0000","last_unstale":"2024-03-19T03:30:40.723999+0000","last_undegraded":"2024-03-19T03:30:40.723999+0000","last_fullsized":"2024-03-19T03:30:40.723999+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:04:02.446185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:09.003790+0000","last_change":"2024-03-19T03:30:34.620404+0000","last_active":"2024-03-19T03:31:09.003790+0000","last_peered":"2024-03-19T03:31:09.003790+0000","last_clean":"2024-03-19T03:31:09.003790+0000","last_became_active":"2024-03-19T03:30:34.620056+0000","last_became_peered":"2024-03-19T03:30:34.620056+0000","last_unstale":"2024-03-19T03:31:09.003790+0000","last_undegraded":"2024-03-19T03:31:09.003790+0000","last_fullsized":"2024-03-19T03:31:09.003790+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:54:42.203912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.730004+0000","last_change":"2024-03-19T03:30:35.686618+0000","last_active":"2024-03-19T03:30:40.730004+0000","last_peered":"2024-03-19T03:30:40.730004+0000","last_clean":"2024-03-19T03:30:40.730004+0000","last_became_active":"2024-03-19T03:30:35.685814+0000","last_became_peered":"2024-03-19T03:30:35.685814+0000","last_unstale":"2024-03-19T03:30:40.730004+0000","last_undegraded":"2024-03-19T03:30:40.730004+0000","last_fullsized":"2024-03-19T03:30:40.730004+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:08.401307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799486+0000","last_change":"2024-03-19T03:30:39.744039+0000","last_active":"2024-03-19T03:30:40.799486+0000","last_peered":"2024-03-19T03:30:40.799486+0000","last_clean":"2024-03-19T03:30:40.799486+0000","last_became_active":"2024-03-19T03:30:39.743637+0000","last_became_peered":"2024-03-19T03:30:39.743637+0000","last_unstale":"2024-03-19T03:30:40.799486+0000","last_undegraded":"2024-03-19T03:30:40.799486+0000","last_fullsized":"2024-03-19T03:30:40.799486+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:02:22.498333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:06.473944+0000","last_change":"2024-03-19T03:30:34.629059+0000","last_active":"2024-03-19T03:31:06.473944+0000","last_peered":"2024-03-19T03:31:06.473944+0000","last_clean":"2024-03-19T03:31:06.473944+0000","last_became_active":"2024-03-19T03:30:34.628617+0000","last_became_peered":"2024-03-19T03:30:34.628617+0000","last_unstale":"2024-03-19T03:31:06.473944+0000","last_undegraded":"2024-03-19T03:31:06.473944+0000","last_fullsized":"2024-03-19T03:31:06.473944+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:47:09.064502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799393+0000","last_change":"2024-03-19T03:30:35.650256+0000","last_active":"2024-03-19T03:30:40.799393+0000","last_peered":"2024-03-19T03:30:40.799393+0000","last_clean":"2024-03-19T03:30:40.799393+0000","last_became_active":"2024-03-19T03:30:35.649086+0000","last_became_peered":"2024-03-19T03:30:35.649086+0000","last_unstale":"2024-03-19T03:30:40.799393+0000","last_undegraded":"2024-03-19T03:30:40.799393+0000","last_fullsized":"2024-03-19T03:30:40.799393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:00:49.204307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.803484+0000","last_change":"2024-03-19T03:30:39.736253+0000","last_active":"2024-03-19T03:30:40.803484+0000","last_peered":"2024-03-19T03:30:40.803484+0000","last_clean":"2024-03-19T03:30:40.803484+0000","last_became_active":"2024-03-19T03:30:39.735699+0000","last_became_peered":"2024-03-19T03:30:39.735699+0000","last_unstale":"2024-03-19T03:30:40.803484+0000","last_undegraded":"2024-03-19T03:30:40.803484+0000","last_fullsized":"2024-03-19T03:30:40.803484+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:46:01.926210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:09.006187+0000","last_change":"2024-03-19T03:30:34.634935+0000","last_active":"2024-03-19T03:31:09.006187+0000","last_peered":"2024-03-19T03:31:09.006187+0000","last_clean":"2024-03-19T03:31:09.006187+0000","last_became_active":"2024-03-19T03:30:34.634602+0000","last_became_peered":"2024-03-19T03:30:34.634602+0000","last_unstale":"2024-03-19T03:31:09.006187+0000","last_undegraded":"2024-03-19T03:31:09.006187+0000","last_fullsized":"2024-03-19T03:31:09.006187+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:16:57.158105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800551+0000","last_change":"2024-03-19T03:30:35.675650+0000","last_active":"2024-03-19T03:30:40.800551+0000","last_peered":"2024-03-19T03:30:40.800551+0000","last_clean":"2024-03-19T03:30:40.800551+0000","last_became_active":"2024-03-19T03:30:35.675255+0000","last_became_peered":"2024-03-19T03:30:35.675255+0000","last_unstale":"2024-03-19T03:30:40.800551+0000","last_undegraded":"2024-03-19T03:30:40.800551+0000","last_fullsized":"2024-03-19T03:30:40.800551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:55:03.201526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:41.558163+0000","last_change":"2024-03-19T03:30:39.733413+0000","last_active":"2024-03-19T03:30:41.558163+0000","last_peered":"2024-03-19T03:30:41.558163+0000","last_clean":"2024-03-19T03:30:41.558163+0000","last_became_active":"2024-03-19T03:30:39.732998+0000","last_became_peered":"2024-03-19T03:30:39.732998+0000","last_unstale":"2024-03-19T03:30:41.558163+0000","last_undegraded":"2024-03-19T03:30:41.558163+0000","last_fullsized":"2024-03-19T03:30:41.558163+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:07:39.512355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:07.467552+0000","last_change":"2024-03-19T03:30:34.623749+0000","last_active":"2024-03-19T03:31:07.467552+0000","last_peered":"2024-03-19T03:31:07.467552+0000","last_clean":"2024-03-19T03:31:07.467552+0000","last_became_active":"2024-03-19T03:30:34.623378+0000","last_became_peered":"2024-03-19T03:30:34.623378+0000","last_unstale":"2024-03-19T03:31:07.467552+0000","last_undegraded":"2024-03-19T03:31:07.467552+0000","last_fullsized":"2024-03-19T03:31:07.467552+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:06:01.939685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.802593+0000","last_change":"2024-03-19T03:30:35.685702+0000","last_active":"2024-03-19T03:30:40.802593+0000","last_peered":"2024-03-19T03:30:40.802593+0000","last_clean":"2024-03-19T03:30:40.802593+0000","last_became_active":"2024-03-19T03:30:35.683340+0000","last_became_peered":"2024-03-19T03:30:35.683340+0000","last_unstale":"2024-03-19T03:30:40.802593+0000","last_undegraded":"2024-03-19T03:30:40.802593+0000","last_fullsized":"2024-03-19T03:30:40.802593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:12.246255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799661+0000","last_change":"2024-03-19T03:30:39.726747+0000","last_active":"2024-03-19T03:30:40.799661+0000","last_peered":"2024-03-19T03:30:40.799661+0000","last_clean":"2024-03-19T03:30:40.799661+0000","last_became_active":"2024-03-19T03:30:39.726236+0000","last_became_peered":"2024-03-19T03:30:39.726236+0000","last_unstale":"2024-03-19T03:30:40.799661+0000","last_undegraded":"2024-03-19T03:30:40.799661+0000","last_fullsized":"2024-03-19T03:30:40.799661+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:02.333544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:06.471949+0000","last_change":"2024-03-19T03:30:34.627828+0000","last_active":"2024-03-19T03:31:06.471949+0000","last_peered":"2024-03-19T03:31:06.471949+0000","last_clean":"2024-03-19T03:31:06.471949+0000","last_became_active":"2024-03-19T03:30:34.627290+0000","last_became_peered":"2024-03-19T03:30:34.627290+0000","last_unstale":"2024-03-19T03:31:06.471949+0000","last_undegraded":"2024-03-19T03:31:06.471949+0000","last_fullsized":"2024-03-19T03:31:06.471949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:07:50.140087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.723835+0000","last_change":"2024-03-19T03:30:35.661180+0000","last_active":"2024-03-19T03:30:40.723835+0000","last_peered":"2024-03-19T03:30:40.723835+0000","last_clean":"2024-03-19T03:30:40.723835+0000","last_became_active":"2024-03-19T03:30:35.660797+0000","last_became_peered":"2024-03-19T03:30:35.660797+0000","last_unstale":"2024-03-19T03:30:40.723835+0000","last_undegraded":"2024-03-19T03:30:40.723835+0000","last_fullsized":"2024-03-19T03:30:40.723835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:54:36.034281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.798935+0000","last_change":"2024-03-19T03:30:39.732832+0000","last_active":"2024-03-19T03:30:40.798935+0000","last_peered":"2024-03-19T03:30:40.798935+0000","last_clean":"2024-03-19T03:30:40.798935+0000","last_became_active":"2024-03-19T03:30:39.732434+0000","last_became_peered":"2024-03-19T03:30:39.732434+0000","last_unstale":"2024-03-19T03:30:40.798935+0000","last_undegraded":"2024-03-19T03:30:40.798935+0000","last_fullsized":"2024-03-19T03:30:40.798935+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:49.737361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:08.995286+0000","last_change":"2024-03-19T03:30:34.625655+0000","last_active":"2024-03-19T03:31:08.995286+0000","last_peered":"2024-03-19T03:31:08.995286+0000","last_clean":"2024-03-19T03:31:08.995286+0000","last_became_active":"2024-03-19T03:30:34.625251+0000","last_became_peered":"2024-03-19T03:30:34.625251+0000","last_unstale":"2024-03-19T03:31:08.995286+0000","last_undegraded":"2024-03-19T03:31:08.995286+0000","last_fullsized":"2024-03-19T03:31:08.995286+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:25.592670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.724140+0000","last_change":"2024-03-19T03:30:35.651147+0000","last_active":"2024-03-19T03:30:40.724140+0000","last_peered":"2024-03-19T03:30:40.724140+0000","last_clean":"2024-03-19T03:30:40.724140+0000","last_became_active":"2024-03-19T03:30:35.650817+0000","last_became_peered":"2024-03-19T03:30:35.650817+0000","last_unstale":"2024-03-19T03:30:40.724140+0000","last_undegraded":"2024-03-19T03:30:40.724140+0000","last_fullsized":"2024-03-19T03:30:40.724140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:35.529226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.728902+0000","last_change":"2024-03-19T03:30:39.749895+0000","last_active":"2024-03-19T03:30:40.728902+0000","last_peered":"2024-03-19T03:30:40.728902+0000","last_clean":"2024-03-19T03:30:40.728902+0000","last_became_active":"2024-03-19T03:30:39.749555+0000","last_became_peered":"2024-03-19T03:30:39.749555+0000","last_unstale":"2024-03-19T03:30:40.728902+0000","last_undegraded":"2024-03-19T03:30:40.728902+0000","last_fullsized":"2024-03-19T03:30:40.728902+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:10:14.383271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:08.991668+0000","last_change":"2024-03-19T03:30:34.635456+0000","last_active":"2024-03-19T03:31:08.991668+0000","last_peered":"2024-03-19T03:31:08.991668+0000","last_clean":"2024-03-19T03:31:08.991668+0000","last_became_active":"2024-03-19T03:30:34.635099+0000","last_became_peered":"2024-03-19T03:30:34.635099+0000","last_unstale":"2024-03-19T03:31:08.991668+0000","last_undegraded":"2024-03-19T03:31:08.991668+0000","last_fullsized":"2024-03-19T03:31:08.991668+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:39:30.083678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.723709+0000","last_change":"2024-03-19T03:30:35.659490+0000","last_active":"2024-03-19T03:30:40.723709+0000","last_peered":"2024-03-19T03:30:40.723709+0000","last_clean":"2024-03-19T03:30:40.723709+0000","last_became_active":"2024-03-19T03:30:35.659208+0000","last_became_peered":"2024-03-19T03:30:35.659208+0000","last_unstale":"2024-03-19T03:30:40.723709+0000","last_undegraded":"2024-03-19T03:30:40.723709+0000","last_fullsized":"2024-03-19T03:30:40.723709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:18.788981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799669+0000","last_change":"2024-03-19T03:30:39.725193+0000","last_active":"2024-03-19T03:30:40.799669+0000","last_peered":"2024-03-19T03:30:40.799669+0000","last_clean":"2024-03-19T03:30:40.799669+0000","last_became_active":"2024-03-19T03:30:39.724745+0000","last_became_peered":"2024-03-19T03:30:39.724745+0000","last_unstale":"2024-03-19T03:30:40.799669+0000","last_undegraded":"2024-03-19T03:30:40.799669+0000","last_fullsized":"2024-03-19T03:30:40.799669+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:30:07.621852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:31:09.005192+0000","last_change":"2024-03-19T03:30:34.636532+0000","last_active":"2024-03-19T03:31:09.005192+0000","last_peered":"2024-03-19T03:31:09.005192+0000","last_clean":"2024-03-19T03:31:09.005192+0000","last_became_active":"2024-03-19T03:30:34.634104+0000","last_became_peered":"2024-03-19T03:30:34.634104+0000","last_unstale":"2024-03-19T03:31:09.005192+0000","last_undegraded":"2024-03-19T03:31:09.005192+0000","last_fullsized":"2024-03-19T03:31:09.005192+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:41:18.308628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.723909+0000","last_change":"2024-03-19T03:30:35.653036+0000","last_active":"2024-03-19T03:30:40.723909+0000","last_peered":"2024-03-19T03:30:40.723909+0000","last_clean":"2024-03-19T03:30:40.723909+0000","last_became_active":"2024-03-19T03:30:35.652564+0000","last_became_peered":"2024-03-19T03:30:35.652564+0000","last_unstale":"2024-03-19T03:30:40.723909+0000","last_undegraded":"2024-03-19T03:30:40.723909+0000","last_fullsized":"2024-03-19T03:30:40.723909+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:01.812525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800454+0000","last_change":"2024-03-19T03:30:35.650724+0000","last_active":"2024-03-19T03:30:40.800454+0000","last_peered":"2024-03-19T03:30:40.800454+0000","last_clean":"2024-03-19T03:30:40.800454+0000","last_became_active":"2024-03-19T03:30:35.650204+0000","last_became_peered":"2024-03-19T03:30:35.650204+0000","last_unstale":"2024-03-19T03:30:40.800454+0000","last_undegraded":"2024-03-19T03:30:40.800454+0000","last_fullsized":"2024-03-19T03:30:40.800454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:06:56.159133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.724460+0000","last_change":"2024-03-19T03:30:35.650241+0000","last_active":"2024-03-19T03:30:40.724460+0000","last_peered":"2024-03-19T03:30:40.724460+0000","last_clean":"2024-03-19T03:30:40.724460+0000","last_became_active":"2024-03-19T03:30:35.649901+0000","last_became_peered":"2024-03-19T03:30:35.649901+0000","last_unstale":"2024-03-19T03:30:40.724460+0000","last_undegraded":"2024-03-19T03:30:40.724460+0000","last_fullsized":"2024-03-19T03:30:40.724460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:35:53.333403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.724030+0000","last_change":"2024-03-19T03:30:35.679267+0000","last_active":"2024-03-19T03:30:40.724030+0000","last_peered":"2024-03-19T03:30:40.724030+0000","last_clean":"2024-03-19T03:30:40.724030+0000","last_became_active":"2024-03-19T03:30:35.678871+0000","last_became_peered":"2024-03-19T03:30:35.678871+0000","last_unstale":"2024-03-19T03:30:40.724030+0000","last_undegraded":"2024-03-19T03:30:40.724030+0000","last_fullsized":"2024-03-19T03:30:40.724030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:23:17.681854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.729073+0000","last_change":"2024-03-19T03:30:35.683060+0000","last_active":"2024-03-19T03:30:40.729073+0000","last_peered":"2024-03-19T03:30:40.729073+0000","last_clean":"2024-03-19T03:30:40.729073+0000","last_became_active":"2024-03-19T03:30:35.676872+0000","last_became_peered":"2024-03-19T03:30:35.676872+0000","last_unstale":"2024-03-19T03:30:40.729073+0000","last_undegraded":"2024-03-19T03:30:40.729073+0000","last_fullsized":"2024-03-19T03:30:40.729073+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T15:13:51.530914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.802168+0000","last_change":"2024-03-19T03:30:35.682685+0000","last_active":"2024-03-19T03:30:40.802168+0000","last_peered":"2024-03-19T03:30:40.802168+0000","last_clean":"2024-03-19T03:30:40.802168+0000","last_became_active":"2024-03-19T03:30:35.680309+0000","last_became_peered":"2024-03-19T03:30:35.680309+0000","last_unstale":"2024-03-19T03:30:40.802168+0000","last_undegraded":"2024-03-19T03:30:40.802168+0000","last_fullsized":"2024-03-19T03:30:40.802168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:31:59.925521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800307+0000","last_change":"2024-03-19T03:30:35.652656+0000","last_active":"2024-03-19T03:30:40.800307+0000","last_peered":"2024-03-19T03:30:40.800307+0000","last_clean":"2024-03-19T03:30:40.800307+0000","last_became_active":"2024-03-19T03:30:35.650744+0000","last_became_peered":"2024-03-19T03:30:35.650744+0000","last_unstale":"2024-03-19T03:30:40.800307+0000","last_undegraded":"2024-03-19T03:30:40.800307+0000","last_fullsized":"2024-03-19T03:30:40.800307+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:49:01.450719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.724650+0000","last_change":"2024-03-19T03:30:35.650092+0000","last_active":"2024-03-19T03:30:40.724650+0000","last_peered":"2024-03-19T03:30:40.724650+0000","last_clean":"2024-03-19T03:30:40.724650+0000","last_became_active":"2024-03-19T03:30:35.649727+0000","last_became_peered":"2024-03-19T03:30:35.649727+0000","last_unstale":"2024-03-19T03:30:40.724650+0000","last_undegraded":"2024-03-19T03:30:40.724650+0000","last_fullsized":"2024-03-19T03:30:40.724650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:22:13.968226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.803587+0000","last_change":"2024-03-19T03:30:35.664723+0000","last_active":"2024-03-19T03:30:40.803587+0000","last_peered":"2024-03-19T03:30:40.803587+0000","last_clean":"2024-03-19T03:30:40.803587+0000","last_became_active":"2024-03-19T03:30:35.655429+0000","last_became_peered":"2024-03-19T03:30:35.655429+0000","last_unstale":"2024-03-19T03:30:40.803587+0000","last_undegraded":"2024-03-19T03:30:40.803587+0000","last_fullsized":"2024-03-19T03:30:40.803587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:09.230235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800537+0000","last_change":"2024-03-19T03:30:35.660289+0000","last_active":"2024-03-19T03:30:40.800537+0000","last_peered":"2024-03-19T03:30:40.800537+0000","last_clean":"2024-03-19T03:30:40.800537+0000","last_became_active":"2024-03-19T03:30:35.660005+0000","last_became_peered":"2024-03-19T03:30:35.660005+0000","last_unstale":"2024-03-19T03:30:40.800537+0000","last_undegraded":"2024-03-19T03:30:40.800537+0000","last_fullsized":"2024-03-19T03:30:40.800537+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:36:16.033629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.801327+0000","last_change":"2024-03-19T03:30:35.660461+0000","last_active":"2024-03-19T03:30:40.801327+0000","last_peered":"2024-03-19T03:30:40.801327+0000","last_clean":"2024-03-19T03:30:40.801327+0000","last_became_active":"2024-03-19T03:30:35.660058+0000","last_became_peered":"2024-03-19T03:30:35.660058+0000","last_unstale":"2024-03-19T03:30:40.801327+0000","last_undegraded":"2024-03-19T03:30:40.801327+0000","last_fullsized":"2024-03-19T03:30:40.801327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:08:40.326314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.799421+0000","last_change":"2024-03-19T03:30:35.665634+0000","last_active":"2024-03-19T03:30:40.799421+0000","last_peered":"2024-03-19T03:30:40.799421+0000","last_clean":"2024-03-19T03:30:40.799421+0000","last_became_active":"2024-03-19T03:30:35.665264+0000","last_became_peered":"2024-03-19T03:30:35.665264+0000","last_unstale":"2024-03-19T03:30:40.799421+0000","last_undegraded":"2024-03-19T03:30:40.799421+0000","last_fullsized":"2024-03-19T03:30:40.799421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:33:56.931412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.801353+0000","last_change":"2024-03-19T03:30:35.681887+0000","last_active":"2024-03-19T03:30:40.801353+0000","last_peered":"2024-03-19T03:30:40.801353+0000","last_clean":"2024-03-19T03:30:40.801353+0000","last_became_active":"2024-03-19T03:30:35.680332+0000","last_became_peered":"2024-03-19T03:30:35.680332+0000","last_unstale":"2024-03-19T03:30:40.801353+0000","last_undegraded":"2024-03-19T03:30:40.801353+0000","last_fullsized":"2024-03-19T03:30:40.801353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:50:24.243641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T03:30:40.800987+0000","last_change":"2024-03-19T03:30:35.652988+0000","last_active":"2024-03-19T03:30:40.800987+0000","last_peered":"2024-03-19T03:30:40.800987+0000","last_clean":"2024-03-19T03:30:40.800987+0000","last_became_active":"2024-03-19T03:30:35.652608+0000","last_became_peered":"2024-03-19T03:30:35.652608+0000","last_unstale":"2024-03-19T03:30:40.800987+0000","last_undegraded":"2024-03-19T03:30:40.800987+0000","last_fullsized":"2024-03-19T03:30:40.800987+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:44:35.619094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283260,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10956,"num_objects":93,"num_object_clones":0,"num_object_copies":279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":204,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":62,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":76643,"data_compressed":1826,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},"log_size":179,"ondisk_log_size":179,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383702,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14713,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645338,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":1073003,"data_compressed":50737,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808865,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":482433,"data_compressed":14713,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070502,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":482811,"data_compressed":14713,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332139,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":1075603,"data_compressed":50737,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495666,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":477937,"data_compressed":14713,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1007616,"data_stored":502108,"data_compressed":14713,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":479393,"data_compressed":14713,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":490020,"data_compressed":14713,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182466,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":477645,"data_compressed":14713,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920830,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":482402,"data_compressed":14713,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757303,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":1074890,"data_compressed":50737,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34380,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34380,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34380,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36024,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36024,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36024,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36024,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36024,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":43658,"data_compressed":1826,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":12805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T03:31:13.092 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-19T03:31:13.093 INFO:tasks.cephadm:Setting up client nodes... 2024-03-19T03:31:13.093 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-19T03:31:13.291 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:13 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2294578502' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:31:13.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:13 smithi117 ceph-mon[31838]: pgmap v336: 129 pgs: 129 active+clean; 587 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4.4 KiB/s wr, 16 op/s 2024-03-19T03:31:13.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:13 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2294578502' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:31:13.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:13 smithi178 ceph-mon[67578]: pgmap v336: 129 pgs: 129 active+clean; 587 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4.4 KiB/s wr, 16 op/s 2024-03-19T03:31:13.483 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:13 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2294578502' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:31:13.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:13 smithi119 ceph-mon[66861]: pgmap v336: 129 pgs: 129 active+clean; 587 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4.4 KiB/s wr, 16 op/s 2024-03-19T03:31:13.586 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:31:14.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:14 smithi117 ceph-mon[31838]: from='client.24887 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T03:31:14.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:14 smithi117 ceph-mon[31838]: from='client.24893 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T03:31:14.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:14 smithi178 ceph-mon[67578]: from='client.24887 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T03:31:14.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:14 smithi178 ceph-mon[67578]: from='client.24893 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T03:31:14.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:14 smithi119 ceph-mon[66861]: from='client.24887 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T03:31:14.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:14 smithi119 ceph-mon[66861]: from='client.24893 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T03:31:15.291 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:15 smithi117 ceph-mon[31838]: pgmap v337: 129 pgs: 129 active+clean; 625 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 8.1 KiB/s wr, 20 op/s 2024-03-19T03:31:15.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:15 smithi178 ceph-mon[67578]: pgmap v337: 129 pgs: 129 active+clean; 625 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 8.1 KiB/s wr, 20 op/s 2024-03-19T03:31:15.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:15 smithi119 ceph-mon[66861]: pgmap v337: 129 pgs: 129 active+clean; 625 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 8.1 KiB/s wr, 20 op/s 2024-03-19T03:31:15.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.893+0000 7fc2e5400700 1 -- 172.21.15.117:0/3348229151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2e0102390 msgr2=0x7fc2e01027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:15.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.893+0000 7fc2e5400700 1 --2- 172.21.15.117:0/3348229151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2e0102390 0x7fc2e01027b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc2d00099c0 tx=0x7fc2d0009cd0 comp rx=0 tx=0).stop 2024-03-19T03:31:15.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.893+0000 7fc2e5400700 1 -- 172.21.15.117:0/3348229151 shutdown_connections 2024-03-19T03:31:15.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.893+0000 7fc2e5400700 1 --2- 172.21.15.117:0/3348229151 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2e0103f30 0x7fc2e010a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:15.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.893+0000 7fc2e5400700 1 --2- 172.21.15.117:0/3348229151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2e0102390 0x7fc2e01027b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:15.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.893+0000 7fc2e5400700 1 --2- 172.21.15.117:0/3348229151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc2e0103590 0x7fc2e01039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:15.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.893+0000 7fc2e5400700 1 -- 172.21.15.117:0/3348229151 >> 172.21.15.117:0/3348229151 conn(0x7fc2e00fd930 msgr2=0x7fc2e00ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:15.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.893+0000 7fc2e5400700 1 -- 172.21.15.117:0/3348229151 shutdown_connections 2024-03-19T03:31:15.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.893+0000 7fc2e5400700 1 -- 172.21.15.117:0/3348229151 wait complete. 2024-03-19T03:31:15.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.894+0000 7fc2e5400700 1 Processor -- start 2024-03-19T03:31:15.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.894+0000 7fc2e5400700 1 -- start start 2024-03-19T03:31:15.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.895+0000 7fc2e5400700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2e0102390 0x7fc2e019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:15.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.895+0000 7fc2e5400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc2e0103590 0x7fc2e019ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:15.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.895+0000 7fc2e5400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2e0103f30 0x7fc2e01a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:15.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.895+0000 7fc2e5400700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e010dfc0 con 0x7fc2e0103590 2024-03-19T03:31:15.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.895+0000 7fc2e5400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e010de40 con 0x7fc2e0103f30 2024-03-19T03:31:15.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.895+0000 7fc2e5400700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e010e140 con 0x7fc2e0102390 2024-03-19T03:31:15.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.895+0000 7fc2df200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2e0103f30 0x7fc2e01a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:15.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.895+0000 7fc2dde00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc2e0103590 0x7fc2e019ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:15.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.895+0000 7fc2de800700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2e0102390 0x7fc2e019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:15.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.895+0000 7fc2df200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2e0103f30 0x7fc2e01a1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:43226/0 (socket says 172.21.15.117:43226) 2024-03-19T03:31:15.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.895+0000 7fc2df200700 1 -- 172.21.15.117:0/1910098531 learned_addr learned my addr 172.21.15.117:0/1910098531 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:15.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.896+0000 7fc2df200700 1 -- 172.21.15.117:0/1910098531 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2e0102390 msgr2=0x7fc2e019a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:15.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.896+0000 7fc2df200700 1 --2- 172.21.15.117:0/1910098531 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2e0102390 0x7fc2e019a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:15.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.896+0000 7fc2df200700 1 -- 172.21.15.117:0/1910098531 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc2e0103590 msgr2=0x7fc2e019ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:15.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.896+0000 7fc2df200700 1 --2- 172.21.15.117:0/1910098531 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc2e0103590 0x7fc2e019ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:15.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.896+0000 7fc2df200700 1 -- 172.21.15.117:0/1910098531 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2e01a2390 con 0x7fc2e0103f30 2024-03-19T03:31:15.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.896+0000 7fc2df200700 1 --2- 172.21.15.117:0/1910098531 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2e0103f30 0x7fc2e01a1cf0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc2d800ef30 tx=0x7fc2d800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:15.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.896+0000 7fc2d7400700 1 -- 172.21.15.117:0/1910098531 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d80092e0 con 0x7fc2e0103f30 2024-03-19T03:31:15.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.896+0000 7fc2d7400700 1 -- 172.21.15.117:0/1910098531 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2d8004bb0 con 0x7fc2e0103f30 2024-03-19T03:31:15.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.896+0000 7fc2e5400700 1 -- 172.21.15.117:0/1910098531 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2e01a2620 con 0x7fc2e0103f30 2024-03-19T03:31:15.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.896+0000 7fc2d7400700 1 -- 172.21.15.117:0/1910098531 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8004280 con 0x7fc2e0103f30 2024-03-19T03:31:15.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.897+0000 7fc2e5400700 1 -- 172.21.15.117:0/1910098531 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2e01a2bb0 con 0x7fc2e0103f30 2024-03-19T03:31:15.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.898+0000 7fc2e5400700 1 -- 172.21.15.117:0/1910098531 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2e0050820 con 0x7fc2e0103f30 2024-03-19T03:31:15.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.900+0000 7fc2d7400700 1 -- 172.21.15.117:0/1910098531 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fc2d801b030 con 0x7fc2e0103f30 2024-03-19T03:31:15.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.902+0000 7fc2d7400700 1 --2- 172.21.15.117:0/1910098531 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc2c40a2a50 0x7fc2c40a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:15.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.902+0000 7fc2d7400700 1 -- 172.21.15.117:0/1910098531 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc2d80cea20 con 0x7fc2e0103f30 2024-03-19T03:31:15.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.902+0000 7fc2de800700 1 --2- 172.21.15.117:0/1910098531 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc2c40a2a50 0x7fc2c40a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:15.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.903+0000 7fc2d7400700 1 -- 172.21.15.117:0/1910098531 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc2d8022020 con 0x7fc2e0103f30 2024-03-19T03:31:15.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:15.903+0000 7fc2de800700 1 --2- 172.21.15.117:0/1910098531 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc2c40a2a50 0x7fc2c40a4f10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc2d000b580 tx=0x7fc2d0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:16.212 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:16.211+0000 7fc2e5400700 1 -- 172.21.15.117:0/1910098531 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fc2e01a3090 con 0x7fc2e0103f30 2024-03-19T03:31:16.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:16.217+0000 7fc2d7400700 1 -- 172.21.15.117:0/1910098531 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fc2d8010040 con 0x7fc2e0103f30 2024-03-19T03:31:16.218 INFO:teuthology.orchestra.run.smithi117.stdout:[client.0] 2024-03-19T03:31:16.219 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAEB/lll+S4DBAAH0C224iR+uIacvFHePldyQ== 2024-03-19T03:31:16.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:16.220+0000 7fc2e5400700 1 -- 172.21.15.117:0/1910098531 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc2c40a2a50 msgr2=0x7fc2c40a4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:16.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:16.220+0000 7fc2e5400700 1 --2- 172.21.15.117:0/1910098531 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc2c40a2a50 0x7fc2c40a4f10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc2d000b580 tx=0x7fc2d0005fb0 comp rx=0 tx=0).stop 2024-03-19T03:31:16.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:16.220+0000 7fc2e5400700 1 -- 172.21.15.117:0/1910098531 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2e0103f30 msgr2=0x7fc2e01a1cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:16.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:16.220+0000 7fc2e5400700 1 --2- 172.21.15.117:0/1910098531 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2e0103f30 0x7fc2e01a1cf0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc2d800ef30 tx=0x7fc2d800c3a0 comp rx=0 tx=0).stop 2024-03-19T03:31:16.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:16.220+0000 7fc2e5400700 1 -- 172.21.15.117:0/1910098531 shutdown_connections 2024-03-19T03:31:16.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:16.220+0000 7fc2e5400700 1 --2- 172.21.15.117:0/1910098531 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc2c40a2a50 0x7fc2c40a4f10 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:16.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:16.220+0000 7fc2e5400700 1 --2- 172.21.15.117:0/1910098531 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2e0103f30 0x7fc2e01a1cf0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:16.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:16.220+0000 7fc2e5400700 1 --2- 172.21.15.117:0/1910098531 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2e0102390 0x7fc2e019a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:16.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:16.220+0000 7fc2e5400700 1 --2- 172.21.15.117:0/1910098531 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc2e0103590 0x7fc2e019ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:16.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:16.220+0000 7fc2e5400700 1 -- 172.21.15.117:0/1910098531 >> 172.21.15.117:0/1910098531 conn(0x7fc2e00fd930 msgr2=0x7fc2e0074540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:16.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:16.220+0000 7fc2e5400700 1 -- 172.21.15.117:0/1910098531 shutdown_connections 2024-03-19T03:31:16.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:16.220+0000 7fc2e5400700 1 -- 172.21.15.117:0/1910098531 wait complete. 2024-03-19T03:31:16.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:16 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1910098531' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-19T03:31:16.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:16 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-19T03:31:16.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:16 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-19T03:31:16.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:16 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1910098531' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-19T03:31:16.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:16 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-19T03:31:16.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:16 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-19T03:31:16.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:16 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1910098531' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-19T03:31:16.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:16 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-19T03:31:16.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:16 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-19T03:31:17.032 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-19T03:31:17.032 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-19T03:31:17.032 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-19T03:31:17.163 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-19T03:31:17.164 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-19T03:31:17.164 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-19T03:31:17.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:17 smithi117 ceph-mon[31838]: pgmap v338: 129 pgs: 129 active+clean; 625 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 8.2 KiB/s wr, 22 op/s 2024-03-19T03:31:17.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:17 smithi178 ceph-mon[67578]: pgmap v338: 129 pgs: 129 active+clean; 625 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 8.2 KiB/s wr, 22 op/s 2024-03-19T03:31:17.667 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:31:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:17 smithi119 ceph-mon[66861]: pgmap v338: 129 pgs: 129 active+clean; 625 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 8.2 KiB/s wr, 22 op/s 2024-03-19T03:31:19.678 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:19 smithi117 ceph-mon[31838]: pgmap v339: 129 pgs: 129 active+clean; 625 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 8.2 KiB/s wr, 23 op/s 2024-03-19T03:31:19.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:19 smithi119 ceph-mon[66861]: pgmap v339: 129 pgs: 129 active+clean; 625 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 8.2 KiB/s wr, 23 op/s 2024-03-19T03:31:19.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:19 smithi178 ceph-mon[67578]: pgmap v339: 129 pgs: 129 active+clean; 625 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 8.2 KiB/s wr, 23 op/s 2024-03-19T03:31:19.846 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.844+0000 7f827fa00700 1 -- 172.21.15.117:0/1116335586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8278108360 msgr2=0x7f8278108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:19.846 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.844+0000 7f827fa00700 1 --2- 172.21.15.117:0/1116335586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8278108360 0x7f8278108740 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f826c009990 tx=0x7f826c009ca0 comp rx=0 tx=0).stop 2024-03-19T03:31:19.846 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.844+0000 7f827fa00700 1 -- 172.21.15.117:0/1116335586 shutdown_connections 2024-03-19T03:31:19.846 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.844+0000 7f827fa00700 1 --2- 172.21.15.117:0/1116335586 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f827806a960 0x7f827810f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:19.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.844+0000 7f827fa00700 1 --2- 172.21.15.117:0/1116335586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8278108360 0x7f8278108740 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:19.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.845+0000 7f827fa00700 1 --2- 172.21.15.117:0/1116335586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8278069fc0 0x7f827806a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:19.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.845+0000 7f827fa00700 1 -- 172.21.15.117:0/1116335586 >> 172.21.15.117:0/1116335586 conn(0x7f8278078140 msgr2=0x7f8278078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:19.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.845+0000 7f827fa00700 1 -- 172.21.15.117:0/1116335586 shutdown_connections 2024-03-19T03:31:19.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.845+0000 7f827fa00700 1 -- 172.21.15.117:0/1116335586 wait complete. 2024-03-19T03:31:19.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.846+0000 7f827fa00700 1 Processor -- start 2024-03-19T03:31:19.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.846+0000 7f827fa00700 1 -- start start 2024-03-19T03:31:19.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.847+0000 7f827fa00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8278069fc0 0x7f82781012e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:19.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.847+0000 7f827fa00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f827806a960 0x7f8278101820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:19.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.847+0000 7f827fa00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8278108360 0x7f8278101d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:19.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.847+0000 7f827fa00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f82781125f0 con 0x7f8278108360 2024-03-19T03:31:19.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.847+0000 7f827fa00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8278112470 con 0x7f827806a960 2024-03-19T03:31:19.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.847+0000 7f827fa00700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f8278112770 con 0x7f8278069fc0 2024-03-19T03:31:19.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.847+0000 7f8277e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f827806a960 0x7f8278101820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:19.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.847+0000 7f8277e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f827806a960 0x7f8278101820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:43234/0 (socket says 172.21.15.117:43234) 2024-03-19T03:31:19.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.847+0000 7f8277e00700 1 -- 172.21.15.117:0/3973795403 learned_addr learned my addr 172.21.15.117:0/3973795403 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:19.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.847+0000 7f827d000700 1 --2- 172.21.15.117:0/3973795403 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8278069fc0 0x7f82781012e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:19.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.847+0000 7f827da00700 1 --2- 172.21.15.117:0/3973795403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8278108360 0x7f8278101d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:19.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.847+0000 7f827d000700 1 -- 172.21.15.117:0/3973795403 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f827806a960 msgr2=0x7f8278101820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:19.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.847+0000 7f827d000700 1 --2- 172.21.15.117:0/3973795403 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f827806a960 0x7f8278101820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:19.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.847+0000 7f827d000700 1 -- 172.21.15.117:0/3973795403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8278108360 msgr2=0x7f8278101d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:19.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.847+0000 7f827d000700 1 --2- 172.21.15.117:0/3973795403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8278108360 0x7f8278101d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:19.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.847+0000 7f827d000700 1 -- 172.21.15.117:0/3973795403 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82781a7000 con 0x7f8278069fc0 2024-03-19T03:31:19.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.848+0000 7f8277e00700 1 --2- 172.21.15.117:0/3973795403 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f827806a960 0x7f8278101820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:31:19.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.848+0000 7f827d000700 1 --2- 172.21.15.117:0/3973795403 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8278069fc0 0x7f82781012e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f826c0098e0 tx=0x7f826c00b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:19.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.848+0000 7f8275600700 1 -- 172.21.15.117:0/3973795403 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c027070 con 0x7f8278069fc0 2024-03-19T03:31:19.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.848+0000 7f8275600700 1 -- 172.21.15.117:0/3973795403 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f826c00fd10 con 0x7f8278069fc0 2024-03-19T03:31:19.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.848+0000 7f827fa00700 1 -- 172.21.15.117:0/3973795403 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82781a7290 con 0x7f8278069fc0 2024-03-19T03:31:19.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.848+0000 7f8275600700 1 -- 172.21.15.117:0/3973795403 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c0173f0 con 0x7f8278069fc0 2024-03-19T03:31:19.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.849+0000 7f827fa00700 1 -- 172.21.15.117:0/3973795403 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82781a7740 con 0x7f8278069fc0 2024-03-19T03:31:19.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.850+0000 7f827fa00700 1 -- 172.21.15.117:0/3973795403 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8278068fd0 con 0x7f8278069fc0 2024-03-19T03:31:19.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.852+0000 7f8275600700 1 -- 172.21.15.117:0/3973795403 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f826c02a070 con 0x7f8278069fc0 2024-03-19T03:31:19.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.853+0000 7f8275600700 1 --2- 172.21.15.117:0/3973795403 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f82600a29a0 0x7f82600a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:19.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.853+0000 7f8275600700 1 -- 172.21.15.117:0/3973795403 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f826c0cfc50 con 0x7f8278069fc0 2024-03-19T03:31:19.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.853+0000 7f8277e00700 1 --2- 172.21.15.117:0/3973795403 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f82600a29a0 0x7f82600a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:19.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.854+0000 7f8277e00700 1 --2- 172.21.15.117:0/3973795403 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f82600a29a0 0x7f82600a4e60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8278107cd0 tx=0x7f8264009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:19.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:19.857+0000 7f8275600700 1 -- 172.21.15.117:0/3973795403 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f826c05e740 con 0x7f8278069fc0 2024-03-19T03:31:20.164 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:20.162+0000 7f827fa00700 1 -- 172.21.15.117:0/3973795403 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f827802dbb0 con 0x7f8278069fc0 2024-03-19T03:31:20.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:20.168+0000 7f8275600700 1 -- 172.21.15.117:0/3973795403 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+262669 (secure 0 0 0) 0x7f826c013070 con 0x7f8278069fc0 2024-03-19T03:31:20.171 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:31:20.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:20.174+0000 7f827fa00700 1 -- 172.21.15.117:0/3973795403 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f82600a29a0 msgr2=0x7f82600a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:20.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:20.174+0000 7f827fa00700 1 --2- 172.21.15.117:0/3973795403 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f82600a29a0 0x7f82600a4e60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8278107cd0 tx=0x7f8264009260 comp rx=0 tx=0).stop 2024-03-19T03:31:20.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:20.174+0000 7f827fa00700 1 -- 172.21.15.117:0/3973795403 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8278069fc0 msgr2=0x7f82781012e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:20.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:20.174+0000 7f827fa00700 1 --2- 172.21.15.117:0/3973795403 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8278069fc0 0x7f82781012e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f826c0098e0 tx=0x7f826c00b840 comp rx=0 tx=0).stop 2024-03-19T03:31:20.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:20.174+0000 7f827fa00700 1 -- 172.21.15.117:0/3973795403 shutdown_connections 2024-03-19T03:31:20.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:20.174+0000 7f827fa00700 1 --2- 172.21.15.117:0/3973795403 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f82600a29a0 0x7f82600a4e60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:20.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:20.174+0000 7f827fa00700 1 --2- 172.21.15.117:0/3973795403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8278108360 0x7f8278101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:20.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:20.174+0000 7f827fa00700 1 --2- 172.21.15.117:0/3973795403 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8278069fc0 0x7f82781012e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:20.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:20.174+0000 7f827fa00700 1 --2- 172.21.15.117:0/3973795403 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f827806a960 0x7f8278101820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:20.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:20.174+0000 7f827fa00700 1 -- 172.21.15.117:0/3973795403 >> 172.21.15.117:0/3973795403 conn(0x7f8278078140 msgr2=0x7f82780ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:20.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:20.175+0000 7f827fa00700 1 -- 172.21.15.117:0/3973795403 shutdown_connections 2024-03-19T03:31:20.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:20.175+0000 7f827fa00700 1 -- 172.21.15.117:0/3973795403 wait complete. 2024-03-19T03:31:20.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:20 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3973795403' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-19T03:31:20.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:20 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3973795403' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-19T03:31:20.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:20 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3973795403' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-19T03:31:21.043 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3144614841},{"type":"v1","addr":"172.21.15.117:6801","nonce":3144614841}]},"active_addr":"172.21.15.117:6801/3144614841","active_change":"2024-03-19T03:22:38.134762+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.117:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":622505007}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":2721065064}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":1074989880}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":1939214638}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":1158984511}]}]} 2024-03-19T03:31:21.048 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-19T03:31:21.048 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-19T03:31:21.048 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-19T03:31:21.511 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:31:21.658 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:21 smithi117 ceph-mon[31838]: pgmap v340: 129 pgs: 129 active+clean; 629 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.1 KiB/s wr, 24 op/s 2024-03-19T03:31:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:21 smithi119 ceph-mon[66861]: pgmap v340: 129 pgs: 129 active+clean; 629 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.1 KiB/s wr, 24 op/s 2024-03-19T03:31:21.747 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:21 smithi178 ceph-mon[67578]: pgmap v340: 129 pgs: 129 active+clean; 629 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.1 KiB/s wr, 24 op/s 2024-03-19T03:31:23.515 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.513+0000 7f506a400700 1 -- 172.21.15.117:0/2242260167 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5064107670 msgr2=0x7f506410f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:23.515 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.513+0000 7f506a400700 1 --2- 172.21.15.117:0/2242260167 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5064107670 0x7f506410f240 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f505c00b0a0 tx=0x7f505c00b3b0 comp rx=0 tx=0).stop 2024-03-19T03:31:23.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.513+0000 7f506a400700 1 -- 172.21.15.117:0/2242260167 shutdown_connections 2024-03-19T03:31:23.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.513+0000 7f506a400700 1 --2- 172.21.15.117:0/2242260167 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5064107670 0x7f506410f240 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:23.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.513+0000 7f506a400700 1 --2- 172.21.15.117:0/2242260167 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5064102590 0x7f5064102970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:23.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.513+0000 7f506a400700 1 --2- 172.21.15.117:0/2242260167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5064102f40 0x7f5064106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:23.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.513+0000 7f506a400700 1 -- 172.21.15.117:0/2242260167 >> 172.21.15.117:0/2242260167 conn(0x7f50640fdde0 msgr2=0x7f5064100200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:23.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.513+0000 7f506a400700 1 -- 172.21.15.117:0/2242260167 shutdown_connections 2024-03-19T03:31:23.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.513+0000 7f506a400700 1 -- 172.21.15.117:0/2242260167 wait complete. 2024-03-19T03:31:23.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.514+0000 7f506a400700 1 Processor -- start 2024-03-19T03:31:23.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.515+0000 7f506a400700 1 -- start start 2024-03-19T03:31:23.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.515+0000 7f506a400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5064102590 0x7f50641a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:23.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.515+0000 7f506a400700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5064102f40 0x7f50641a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:23.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.515+0000 7f506a400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5064107670 0x7f506419aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:23.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.515+0000 7f506a400700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f50641126b0 con 0x7f5064102590 2024-03-19T03:31:23.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.515+0000 7f506a400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5064112530 con 0x7f5064107670 2024-03-19T03:31:23.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.515+0000 7f506a400700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f5064112830 con 0x7f5064102f40 2024-03-19T03:31:23.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.515+0000 7f5063e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5064107670 0x7f506419aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:23.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.515+0000 7f5063e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5064107670 0x7f506419aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:43242/0 (socket says 172.21.15.117:43242) 2024-03-19T03:31:23.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.515+0000 7f5062a00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5064102f40 0x7f50641a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:23.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.515+0000 7f5063e00700 1 -- 172.21.15.117:0/3297108491 learned_addr learned my addr 172.21.15.117:0/3297108491 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:23.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.515+0000 7f5063400700 1 --2- 172.21.15.117:0/3297108491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5064102590 0x7f50641a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:23.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.516+0000 7f5063e00700 1 -- 172.21.15.117:0/3297108491 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5064102f40 msgr2=0x7f50641a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:23.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.516+0000 7f5063e00700 1 --2- 172.21.15.117:0/3297108491 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5064102f40 0x7f50641a0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:23.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.516+0000 7f5063e00700 1 -- 172.21.15.117:0/3297108491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5064102590 msgr2=0x7f50641a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:23.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.516+0000 7f5063e00700 1 --2- 172.21.15.117:0/3297108491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5064102590 0x7f50641a08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:23.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.516+0000 7f5063e00700 1 -- 172.21.15.117:0/3297108491 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f506419b1f0 con 0x7f5064107670 2024-03-19T03:31:23.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.516+0000 7f5063400700 1 --2- 172.21.15.117:0/3297108491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5064102590 0x7f50641a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:31:23.518 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.516+0000 7f5063e00700 1 --2- 172.21.15.117:0/3297108491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5064107670 0x7f506419aa90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f505c0079a0 tx=0x7f505c0079d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:23.519 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.516+0000 7f505be00700 1 -- 172.21.15.117:0/3297108491 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f505c021070 con 0x7f5064107670 2024-03-19T03:31:23.519 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.516+0000 7f505be00700 1 -- 172.21.15.117:0/3297108491 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f505c01db30 con 0x7f5064107670 2024-03-19T03:31:23.519 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.517+0000 7f506a400700 1 -- 172.21.15.117:0/3297108491 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f506419b480 con 0x7f5064107670 2024-03-19T03:31:23.519 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.517+0000 7f505be00700 1 -- 172.21.15.117:0/3297108491 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f505c01bc50 con 0x7f5064107670 2024-03-19T03:31:23.519 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.517+0000 7f506a400700 1 -- 172.21.15.117:0/3297108491 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50641a77a0 con 0x7f5064107670 2024-03-19T03:31:23.519 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.518+0000 7f506a400700 1 -- 172.21.15.117:0/3297108491 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5064068fd0 con 0x7f5064107670 2024-03-19T03:31:23.520 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.519+0000 7f505be00700 1 -- 172.21.15.117:0/3297108491 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f505c024070 con 0x7f5064107670 2024-03-19T03:31:23.521 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.521+0000 7f505be00700 1 --2- 172.21.15.117:0/3297108491 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f50480a29a0 0x7f50480a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:23.522 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.521+0000 7f505be00700 1 -- 172.21.15.117:0/3297108491 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f505c0cfe40 con 0x7f5064107670 2024-03-19T03:31:23.522 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.521+0000 7f5063400700 1 --2- 172.21.15.117:0/3297108491 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f50480a29a0 0x7f50480a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:23.522 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.522+0000 7f5063400700 1 --2- 172.21.15.117:0/3297108491 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f50480a29a0 0x7f50480a4e60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5054005950 tx=0x7f50540058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:23.530 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.529+0000 7f505be00700 1 -- 172.21.15.117:0/3297108491 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f505c09cb70 con 0x7f5064107670 2024-03-19T03:31:23.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:23 smithi119 ceph-mon[66861]: pgmap v341: 129 pgs: 129 active+clean; 629 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.7 KiB/s wr, 23 op/s 2024-03-19T03:31:23.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.772+0000 7f506a400700 1 -- 172.21.15.117:0/3297108491 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f50641a7970 con 0x7f5064107670 2024-03-19T03:31:23.773 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:23 smithi117 ceph-mon[31838]: pgmap v341: 129 pgs: 129 active+clean; 629 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.7 KiB/s wr, 23 op/s 2024-03-19T03:31:23.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.773+0000 7f505be00700 1 -- 172.21.15.117:0/3297108491 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24210 (secure 0 0 0) 0x7f505c009cb0 con 0x7f5064107670 2024-03-19T03:31:23.775 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:31:23.775 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":89,"fsid":"a3a82456-e59f-11ee-95c9-87774f69a715","created":"2024-03-19T03:21:30.427136+0000","modified":"2024-03-19T03:30:40.180745+0000","last_up_change":"2024-03-19T03:29:32.967505+0000","last_in_change":"2024-03-19T03:29:12.848740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:25:18.824270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T03:30:33.137170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T03:30:34.598838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T03:30:38.230023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b32eb0ee-9d4b-4748-87ec-eda02c907415","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6803","nonce":3014934900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6805","nonce":3014934900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6809","nonce":3014934900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6807","nonce":3014934900}]},"public_addr":"172.21.15.117:6803/3014934900","cluster_addr":"172.21.15.117:6805/3014934900","heartbeat_back_addr":"172.21.15.117:6809/3014934900","heartbeat_front_addr":"172.21.15.117:6807/3014934900","state":["exists","up"]},{"osd":1,"uuid":"e07585be-c016-4466-b07c-abbc22a58de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6801","nonce":2425927341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6803","nonce":2425927341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6807","nonce":2425927341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6805","nonce":2425927341}]},"public_addr":"172.21.15.119:6801/2425927341","cluster_addr":"172.21.15.119:6803/2425927341","heartbeat_back_addr":"172.21.15.119:6807/2425927341","heartbeat_front_addr":"172.21.15.119:6805/2425927341","state":["exists","up"]},{"osd":2,"uuid":"ede9edda-3b9f-41ca-ad04-5dcb738ab757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6801","nonce":3733812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6803","nonce":3733812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6807","nonce":3733812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6805","nonce":3733812989}]},"public_addr":"172.21.15.178:6801/3733812989","cluster_addr":"172.21.15.178:6803/3733812989","heartbeat_back_addr":"172.21.15.178:6807/3733812989","heartbeat_front_addr":"172.21.15.178:6805/3733812989","state":["exists","up"]},{"osd":3,"uuid":"f81c41e1-3a69-40db-a90f-58151286b081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6811","nonce":2178310135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6813","nonce":2178310135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6817","nonce":2178310135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6815","nonce":2178310135}]},"public_addr":"172.21.15.117:6811/2178310135","cluster_addr":"172.21.15.117:6813/2178310135","heartbeat_back_addr":"172.21.15.117:6817/2178310135","heartbeat_front_addr":"172.21.15.117:6815/2178310135","state":["exists","up"]},{"osd":4,"uuid":"caf7ce86-caf0-4eaf-bbab-bd2d93e63c33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6809","nonce":3455398086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6811","nonce":3455398086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6815","nonce":3455398086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6813","nonce":3455398086}]},"public_addr":"172.21.15.119:6809/3455398086","cluster_addr":"172.21.15.119:6811/3455398086","heartbeat_back_addr":"172.21.15.119:6815/3455398086","heartbeat_front_addr":"172.21.15.119:6813/3455398086","state":["exists","up"]},{"osd":5,"uuid":"20ba0b8a-2e0f-4ea7-b435-430e44766200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6809","nonce":1259631912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6811","nonce":1259631912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6815","nonce":1259631912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6813","nonce":1259631912}]},"public_addr":"172.21.15.178:6809/1259631912","cluster_addr":"172.21.15.178:6811/1259631912","heartbeat_back_addr":"172.21.15.178:6815/1259631912","heartbeat_front_addr":"172.21.15.178:6813/1259631912","state":["exists","up"]},{"osd":6,"uuid":"75c2ddd7-f837-42e0-ac47-8cfbcf97a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6819","nonce":864146757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6821","nonce":864146757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6825","nonce":864146757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6823","nonce":864146757}]},"public_addr":"172.21.15.117:6819/864146757","cluster_addr":"172.21.15.117:6821/864146757","heartbeat_back_addr":"172.21.15.117:6825/864146757","heartbeat_front_addr":"172.21.15.117:6823/864146757","state":["exists","up"]},{"osd":7,"uuid":"929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6817","nonce":1273365545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6819","nonce":1273365545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6823","nonce":1273365545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6821","nonce":1273365545}]},"public_addr":"172.21.15.119:6817/1273365545","cluster_addr":"172.21.15.119:6819/1273365545","heartbeat_back_addr":"172.21.15.119:6823/1273365545","heartbeat_front_addr":"172.21.15.119:6821/1273365545","state":["exists","up"]},{"osd":8,"uuid":"fd371461-a538-4781-92e1-5eead4ca3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6817","nonce":1059587501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6819","nonce":1059587501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6823","nonce":1059587501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6821","nonce":1059587501}]},"public_addr":"172.21.15.178:6817/1059587501","cluster_addr":"172.21.15.178:6819/1059587501","heartbeat_back_addr":"172.21.15.178:6823/1059587501","heartbeat_front_addr":"172.21.15.178:6821/1059587501","state":["exists","up"]},{"osd":9,"uuid":"0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6827","nonce":3653875600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6829","nonce":3653875600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6833","nonce":3653875600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6831","nonce":3653875600}]},"public_addr":"172.21.15.117:6827/3653875600","cluster_addr":"172.21.15.117:6829/3653875600","heartbeat_back_addr":"172.21.15.117:6833/3653875600","heartbeat_front_addr":"172.21.15.117:6831/3653875600","state":["exists","up"]},{"osd":10,"uuid":"de2049da-d19e-4386-afbf-fd6b5fddbcba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6825","nonce":1050976194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6827","nonce":1050976194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6831","nonce":1050976194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6829","nonce":1050976194}]},"public_addr":"172.21.15.119:6825/1050976194","cluster_addr":"172.21.15.119:6827/1050976194","heartbeat_back_addr":"172.21.15.119:6831/1050976194","heartbeat_front_addr":"172.21.15.119:6829/1050976194","state":["exists","up"]},{"osd":11,"uuid":"3cccc0f9-d273-4b6b-9b8e-c926371472dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6825","nonce":2915080368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6827","nonce":2915080368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6831","nonce":2915080368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6829","nonce":2915080368}]},"public_addr":"172.21.15.178:6825/2915080368","cluster_addr":"172.21.15.178:6827/2915080368","heartbeat_back_addr":"172.21.15.178:6831/2915080368","heartbeat_front_addr":"172.21.15.178:6829/2915080368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:27.876274+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:53.094706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:16.598813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:48.160419+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:14.048925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:38.814304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:12.542328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:38.942747+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:04.488963+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:38.196053+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:05.254728+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:30.874915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.117:0/185684447":"2024-03-20T03:22:38.134578+0000","172.21.15.117:0/361367101":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/2788067603":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3506200675":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6800/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3283471541":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3750920169":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/2780813298":"2024-03-20T03:22:03.905461+0000","172.21.15.117:6800/2788067603":"2024-03-20T03:22:38.134578+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:31:23.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.776+0000 7f506a400700 1 -- 172.21.15.117:0/3297108491 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f50480a29a0 msgr2=0x7f50480a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:23.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.776+0000 7f506a400700 1 --2- 172.21.15.117:0/3297108491 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f50480a29a0 0x7f50480a4e60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5054005950 tx=0x7f50540058e0 comp rx=0 tx=0).stop 2024-03-19T03:31:23.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.777+0000 7f506a400700 1 -- 172.21.15.117:0/3297108491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5064107670 msgr2=0x7f506419aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:23.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.777+0000 7f506a400700 1 --2- 172.21.15.117:0/3297108491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5064107670 0x7f506419aa90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f505c0079a0 tx=0x7f505c0079d0 comp rx=0 tx=0).stop 2024-03-19T03:31:23.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.777+0000 7f506a400700 1 -- 172.21.15.117:0/3297108491 shutdown_connections 2024-03-19T03:31:23.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.777+0000 7f506a400700 1 --2- 172.21.15.117:0/3297108491 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f50480a29a0 0x7f50480a4e60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:23.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.777+0000 7f506a400700 1 --2- 172.21.15.117:0/3297108491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5064107670 0x7f506419aa90 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:23.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.777+0000 7f506a400700 1 --2- 172.21.15.117:0/3297108491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5064102590 0x7f50641a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:23.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.777+0000 7f506a400700 1 --2- 172.21.15.117:0/3297108491 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5064102f40 0x7f50641a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:23.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.777+0000 7f506a400700 1 -- 172.21.15.117:0/3297108491 >> 172.21.15.117:0/3297108491 conn(0x7f50640fdde0 msgr2=0x7f50640fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:23.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.777+0000 7f506a400700 1 -- 172.21.15.117:0/3297108491 shutdown_connections 2024-03-19T03:31:23.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:23.777+0000 7f506a400700 1 -- 172.21.15.117:0/3297108491 wait complete. 2024-03-19T03:31:23.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:23 smithi178 ceph-mon[67578]: pgmap v341: 129 pgs: 129 active+clean; 629 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.7 KiB/s wr, 23 op/s 2024-03-19T03:31:24.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:24 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3297108491' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:31:24.634 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-19T03:31:24.634 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-19T03:31:24.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:24 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3297108491' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:31:24.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:24 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3297108491' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:31:25.125 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:31:25.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:25 smithi117 ceph-mon[31838]: pgmap v342: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.4 KiB/s wr, 13 op/s 2024-03-19T03:31:25.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:25 smithi119 ceph-mon[66861]: pgmap v342: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.4 KiB/s wr, 13 op/s 2024-03-19T03:31:25.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:25 smithi178 ceph-mon[67578]: pgmap v342: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.4 KiB/s wr, 13 op/s 2024-03-19T03:31:27.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.402+0000 7f2a3c600700 1 -- 172.21.15.117:0/2562966840 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a341025c0 msgr2=0x7f2a341029a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:27.404 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.402+0000 7f2a3c600700 1 --2- 172.21.15.117:0/2562966840 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a341025c0 0x7f2a341029a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2a28009990 tx=0x7f2a28009ca0 comp rx=0 tx=0).stop 2024-03-19T03:31:27.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.402+0000 7f2a3c600700 1 -- 172.21.15.117:0/2562966840 shutdown_connections 2024-03-19T03:31:27.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.402+0000 7f2a3c600700 1 --2- 172.21.15.117:0/2562966840 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2a34107690 0x7f2a3410f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:27.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.402+0000 7f2a3c600700 1 --2- 172.21.15.117:0/2562966840 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a341025c0 0x7f2a341029a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:27.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.402+0000 7f2a3c600700 1 --2- 172.21.15.117:0/2562966840 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a34102f70 0x7f2a34106fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:27.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.403+0000 7f2a3c600700 1 -- 172.21.15.117:0/2562966840 >> 172.21.15.117:0/2562966840 conn(0x7f2a34079470 msgr2=0x7f2a34079880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:27.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.403+0000 7f2a3c600700 1 -- 172.21.15.117:0/2562966840 shutdown_connections 2024-03-19T03:31:27.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.403+0000 7f2a3c600700 1 -- 172.21.15.117:0/2562966840 wait complete. 2024-03-19T03:31:27.405 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.404+0000 7f2a3c600700 1 Processor -- start 2024-03-19T03:31:27.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.404+0000 7f2a3c600700 1 -- start start 2024-03-19T03:31:27.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.404+0000 7f2a3c600700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a341025c0 0x7f2a3419e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:27.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.404+0000 7f2a3c600700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2a34102f70 0x7f2a3419ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:27.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.404+0000 7f2a3c600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a34107690 0x7f2a34198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:27.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.404+0000 7f2a3c600700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2a341126c0 con 0x7f2a341025c0 2024-03-19T03:31:27.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.404+0000 7f2a3c600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2a34112540 con 0x7f2a34107690 2024-03-19T03:31:27.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.404+0000 7f2a3c600700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f2a34112840 con 0x7f2a34102f70 2024-03-19T03:31:27.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.405+0000 7f2a39c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a341025c0 0x7f2a3419e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:27.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.405+0000 7f2a39c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a341025c0 0x7f2a3419e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39860/0 (socket says 172.21.15.117:39860) 2024-03-19T03:31:27.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.405+0000 7f2a39c00700 1 -- 172.21.15.117:0/863665374 learned_addr learned my addr 172.21.15.117:0/863665374 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:27.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.405+0000 7f2a3a600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a34107690 0x7f2a34198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:27.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.405+0000 7f2a39200700 1 --2- 172.21.15.117:0/863665374 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2a34102f70 0x7f2a3419ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:27.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.405+0000 7f2a3a600700 1 -- 172.21.15.117:0/863665374 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2a34102f70 msgr2=0x7f2a3419ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:27.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.405+0000 7f2a3a600700 1 --2- 172.21.15.117:0/863665374 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2a34102f70 0x7f2a3419ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:27.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.405+0000 7f2a3a600700 1 -- 172.21.15.117:0/863665374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a341025c0 msgr2=0x7f2a3419e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:27.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.405+0000 7f2a3a600700 1 --2- 172.21.15.117:0/863665374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a341025c0 0x7f2a3419e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:27.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.405+0000 7f2a3a600700 1 -- 172.21.15.117:0/863665374 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a34199180 con 0x7f2a34107690 2024-03-19T03:31:27.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.406+0000 7f2a3a600700 1 --2- 172.21.15.117:0/863665374 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a34107690 0x7f2a34198a20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c007f10 tx=0x7f2a2c00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:27.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.406+0000 7f2a32000700 1 -- 172.21.15.117:0/863665374 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c004db0 con 0x7f2a34107690 2024-03-19T03:31:27.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.406+0000 7f2a32000700 1 -- 172.21.15.117:0/863665374 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a2c009e20 con 0x7f2a34107690 2024-03-19T03:31:27.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.406+0000 7f2a32000700 1 -- 172.21.15.117:0/863665374 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c015400 con 0x7f2a34107690 2024-03-19T03:31:27.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.406+0000 7f2a3c600700 1 -- 172.21.15.117:0/863665374 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a34199470 con 0x7f2a34107690 2024-03-19T03:31:27.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.406+0000 7f2a3c600700 1 -- 172.21.15.117:0/863665374 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a341a56e0 con 0x7f2a34107690 2024-03-19T03:31:27.408 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.407+0000 7f2a3c600700 1 -- 172.21.15.117:0/863665374 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a34068fd0 con 0x7f2a34107690 2024-03-19T03:31:27.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.409+0000 7f2a32000700 1 -- 172.21.15.117:0/863665374 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f2a2c00db20 con 0x7f2a34107690 2024-03-19T03:31:27.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.411+0000 7f2a32000700 1 --2- 172.21.15.117:0/863665374 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f2a1c0a29a0 0x7f2a1c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:27.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.411+0000 7f2a32000700 1 -- 172.21.15.117:0/863665374 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2a2c0cee90 con 0x7f2a34107690 2024-03-19T03:31:27.412 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.411+0000 7f2a39c00700 1 --2- 172.21.15.117:0/863665374 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f2a1c0a29a0 0x7f2a1c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:27.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.412+0000 7f2a39c00700 1 --2- 172.21.15.117:0/863665374 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f2a1c0a29a0 0x7f2a1c0a4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2a28009750 tx=0x7f2a2801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:27.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.417+0000 7f2a32000700 1 -- 172.21.15.117:0/863665374 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2a2c010070 con 0x7f2a34107690 2024-03-19T03:31:27.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.656+0000 7f2a3c600700 1 -- 172.21.15.117:0/863665374 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2a3402dbb0 con 0x7f2a34107690 2024-03-19T03:31:27.658 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:27 smithi117 ceph-mon[31838]: pgmap v343: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-19T03:31:27.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.659+0000 7f2a32000700 1 -- 172.21.15.117:0/863665374 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24210 (secure 0 0 0) 0x7f2a2c0a3e20 con 0x7f2a34107690 2024-03-19T03:31:27.661 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:31:27.661 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":89,"fsid":"a3a82456-e59f-11ee-95c9-87774f69a715","created":"2024-03-19T03:21:30.427136+0000","modified":"2024-03-19T03:30:40.180745+0000","last_up_change":"2024-03-19T03:29:32.967505+0000","last_in_change":"2024-03-19T03:29:12.848740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:25:18.824270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T03:30:33.137170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T03:30:34.598838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T03:30:38.230023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b32eb0ee-9d4b-4748-87ec-eda02c907415","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6803","nonce":3014934900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6805","nonce":3014934900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6809","nonce":3014934900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6807","nonce":3014934900}]},"public_addr":"172.21.15.117:6803/3014934900","cluster_addr":"172.21.15.117:6805/3014934900","heartbeat_back_addr":"172.21.15.117:6809/3014934900","heartbeat_front_addr":"172.21.15.117:6807/3014934900","state":["exists","up"]},{"osd":1,"uuid":"e07585be-c016-4466-b07c-abbc22a58de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6801","nonce":2425927341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6803","nonce":2425927341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6807","nonce":2425927341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6805","nonce":2425927341}]},"public_addr":"172.21.15.119:6801/2425927341","cluster_addr":"172.21.15.119:6803/2425927341","heartbeat_back_addr":"172.21.15.119:6807/2425927341","heartbeat_front_addr":"172.21.15.119:6805/2425927341","state":["exists","up"]},{"osd":2,"uuid":"ede9edda-3b9f-41ca-ad04-5dcb738ab757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6801","nonce":3733812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6803","nonce":3733812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6807","nonce":3733812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6805","nonce":3733812989}]},"public_addr":"172.21.15.178:6801/3733812989","cluster_addr":"172.21.15.178:6803/3733812989","heartbeat_back_addr":"172.21.15.178:6807/3733812989","heartbeat_front_addr":"172.21.15.178:6805/3733812989","state":["exists","up"]},{"osd":3,"uuid":"f81c41e1-3a69-40db-a90f-58151286b081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6811","nonce":2178310135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6813","nonce":2178310135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6817","nonce":2178310135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6815","nonce":2178310135}]},"public_addr":"172.21.15.117:6811/2178310135","cluster_addr":"172.21.15.117:6813/2178310135","heartbeat_back_addr":"172.21.15.117:6817/2178310135","heartbeat_front_addr":"172.21.15.117:6815/2178310135","state":["exists","up"]},{"osd":4,"uuid":"caf7ce86-caf0-4eaf-bbab-bd2d93e63c33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6809","nonce":3455398086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6811","nonce":3455398086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6815","nonce":3455398086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6813","nonce":3455398086}]},"public_addr":"172.21.15.119:6809/3455398086","cluster_addr":"172.21.15.119:6811/3455398086","heartbeat_back_addr":"172.21.15.119:6815/3455398086","heartbeat_front_addr":"172.21.15.119:6813/3455398086","state":["exists","up"]},{"osd":5,"uuid":"20ba0b8a-2e0f-4ea7-b435-430e44766200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6809","nonce":1259631912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6811","nonce":1259631912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6815","nonce":1259631912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6813","nonce":1259631912}]},"public_addr":"172.21.15.178:6809/1259631912","cluster_addr":"172.21.15.178:6811/1259631912","heartbeat_back_addr":"172.21.15.178:6815/1259631912","heartbeat_front_addr":"172.21.15.178:6813/1259631912","state":["exists","up"]},{"osd":6,"uuid":"75c2ddd7-f837-42e0-ac47-8cfbcf97a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6819","nonce":864146757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6821","nonce":864146757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6825","nonce":864146757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6823","nonce":864146757}]},"public_addr":"172.21.15.117:6819/864146757","cluster_addr":"172.21.15.117:6821/864146757","heartbeat_back_addr":"172.21.15.117:6825/864146757","heartbeat_front_addr":"172.21.15.117:6823/864146757","state":["exists","up"]},{"osd":7,"uuid":"929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6817","nonce":1273365545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6819","nonce":1273365545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6823","nonce":1273365545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6821","nonce":1273365545}]},"public_addr":"172.21.15.119:6817/1273365545","cluster_addr":"172.21.15.119:6819/1273365545","heartbeat_back_addr":"172.21.15.119:6823/1273365545","heartbeat_front_addr":"172.21.15.119:6821/1273365545","state":["exists","up"]},{"osd":8,"uuid":"fd371461-a538-4781-92e1-5eead4ca3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6817","nonce":1059587501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6819","nonce":1059587501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6823","nonce":1059587501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6821","nonce":1059587501}]},"public_addr":"172.21.15.178:6817/1059587501","cluster_addr":"172.21.15.178:6819/1059587501","heartbeat_back_addr":"172.21.15.178:6823/1059587501","heartbeat_front_addr":"172.21.15.178:6821/1059587501","state":["exists","up"]},{"osd":9,"uuid":"0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6827","nonce":3653875600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6829","nonce":3653875600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6833","nonce":3653875600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6831","nonce":3653875600}]},"public_addr":"172.21.15.117:6827/3653875600","cluster_addr":"172.21.15.117:6829/3653875600","heartbeat_back_addr":"172.21.15.117:6833/3653875600","heartbeat_front_addr":"172.21.15.117:6831/3653875600","state":["exists","up"]},{"osd":10,"uuid":"de2049da-d19e-4386-afbf-fd6b5fddbcba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6825","nonce":1050976194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6827","nonce":1050976194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6831","nonce":1050976194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6829","nonce":1050976194}]},"public_addr":"172.21.15.119:6825/1050976194","cluster_addr":"172.21.15.119:6827/1050976194","heartbeat_back_addr":"172.21.15.119:6831/1050976194","heartbeat_front_addr":"172.21.15.119:6829/1050976194","state":["exists","up"]},{"osd":11,"uuid":"3cccc0f9-d273-4b6b-9b8e-c926371472dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6825","nonce":2915080368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6827","nonce":2915080368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6831","nonce":2915080368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6829","nonce":2915080368}]},"public_addr":"172.21.15.178:6825/2915080368","cluster_addr":"172.21.15.178:6827/2915080368","heartbeat_back_addr":"172.21.15.178:6831/2915080368","heartbeat_front_addr":"172.21.15.178:6829/2915080368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:27.876274+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:53.094706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:16.598813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:48.160419+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:14.048925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:38.814304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:12.542328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:38.942747+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:04.488963+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:38.196053+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:05.254728+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:30.874915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.117:0/185684447":"2024-03-20T03:22:38.134578+0000","172.21.15.117:0/361367101":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/2788067603":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3506200675":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6800/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3283471541":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3750920169":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/2780813298":"2024-03-20T03:22:03.905461+0000","172.21.15.117:6800/2788067603":"2024-03-20T03:22:38.134578+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:31:27.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.662+0000 7f2a3c600700 1 -- 172.21.15.117:0/863665374 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f2a1c0a29a0 msgr2=0x7f2a1c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:27.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.662+0000 7f2a3c600700 1 --2- 172.21.15.117:0/863665374 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f2a1c0a29a0 0x7f2a1c0a4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2a28009750 tx=0x7f2a2801a040 comp rx=0 tx=0).stop 2024-03-19T03:31:27.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.663+0000 7f2a3c600700 1 -- 172.21.15.117:0/863665374 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a34107690 msgr2=0x7f2a34198a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:27.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.663+0000 7f2a3c600700 1 --2- 172.21.15.117:0/863665374 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a34107690 0x7f2a34198a20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c007f10 tx=0x7f2a2c00d3b0 comp rx=0 tx=0).stop 2024-03-19T03:31:27.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.663+0000 7f2a3c600700 1 -- 172.21.15.117:0/863665374 shutdown_connections 2024-03-19T03:31:27.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.663+0000 7f2a3c600700 1 --2- 172.21.15.117:0/863665374 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f2a1c0a29a0 0x7f2a1c0a4e60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:27.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.663+0000 7f2a3c600700 1 --2- 172.21.15.117:0/863665374 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a34107690 0x7f2a34198a20 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:27.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.663+0000 7f2a3c600700 1 --2- 172.21.15.117:0/863665374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a341025c0 0x7f2a3419e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:27.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.663+0000 7f2a3c600700 1 --2- 172.21.15.117:0/863665374 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2a34102f70 0x7f2a3419ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:27.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.663+0000 7f2a3c600700 1 -- 172.21.15.117:0/863665374 >> 172.21.15.117:0/863665374 conn(0x7f2a34079470 msgr2=0x7f2a34106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:27.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.663+0000 7f2a3c600700 1 -- 172.21.15.117:0/863665374 shutdown_connections 2024-03-19T03:31:27.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:27.663+0000 7f2a3c600700 1 -- 172.21.15.117:0/863665374 wait complete. 2024-03-19T03:31:27.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:27 smithi119 ceph-mon[66861]: pgmap v343: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-19T03:31:27.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:27 smithi178 ceph-mon[67578]: pgmap v343: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-19T03:31:28.444 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-19T03:31:28.445 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-19T03:31:28.445 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-19T03:31:28.445 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-19T03:31:28.446 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-19T03:31:28.446 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-19T03:31:28.446 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-19T03:31:28.447 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-19T03:31:28.448 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-19T03:31:28.448 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-19T03:31:28.448 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-19T03:31:28.449 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-19T03:31:28.661 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:28 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/863665374' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:31:28.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:28 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/863665374' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:31:28.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:28 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/863665374' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:31:29.672 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:31:29.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:29 smithi119 ceph-mon[66861]: pgmap v344: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-19T03:31:29.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:29 smithi117 ceph-mon[31838]: pgmap v344: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-19T03:31:29.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:29 smithi178 ceph-mon[67578]: pgmap v344: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-19T03:31:29.887 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:31:29.897 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:31:29.902 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:31:29.902 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:31:29.905 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:31:29.908 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:31:31.442 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:31:31.442 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:31:31.444 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:31:31.447 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:31:31.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:31 smithi119 ceph-mon[66861]: pgmap v345: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-19T03:31:31.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:31 smithi117 ceph-mon[31838]: pgmap v345: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-19T03:31:31.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:31 smithi178 ceph-mon[67578]: pgmap v345: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-19T03:31:32.283 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:31:33.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:33 smithi119 ceph-mon[66861]: pgmap v346: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-19T03:31:33.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:33 smithi117 ceph-mon[31838]: pgmap v346: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-19T03:31:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:33 smithi178 ceph-mon[67578]: pgmap v346: 129 pgs: 129 active+clean; 641 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-19T03:31:35.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:35 smithi119 ceph-mon[66861]: pgmap v347: 129 pgs: 129 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-19T03:31:35.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:35 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:31:35.762 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:35 smithi178 ceph-mon[67578]: pgmap v347: 129 pgs: 129 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-19T03:31:35.762 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:35 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:31:35.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:35 smithi117 ceph-mon[31838]: pgmap v347: 129 pgs: 129 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-19T03:31:35.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:35 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:31:37.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.450+0000 7f3787c00700 1 -- 172.21.15.117:0/1554123198 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3780107250 msgr2=0x7f3780107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:37.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.450+0000 7f3787c00700 1 --2- 172.21.15.117:0/1554123198 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3780107250 0x7f3780107630 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f37740098f0 tx=0x7f3774009c00 comp rx=0 tx=0).stop 2024-03-19T03:31:37.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.450+0000 7f3787c00700 1 -- 172.21.15.117:0/1554123198 shutdown_connections 2024-03-19T03:31:37.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.450+0000 7f3787c00700 1 --2- 172.21.15.117:0/1554123198 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3780101bb0 0x7f378010e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:37.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.450+0000 7f3787c00700 1 --2- 172.21.15.117:0/1554123198 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3780107250 0x7f3780107630 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:37.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.450+0000 7f3787c00700 1 --2- 172.21.15.117:0/1554123198 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3780101210 0x7f3780101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:37.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.450+0000 7f3787c00700 1 -- 172.21.15.117:0/1554123198 >> 172.21.15.117:0/1554123198 conn(0x7f3780078140 msgr2=0x7f3780078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:37.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.450+0000 7f3787c00700 1 -- 172.21.15.117:0/1554123198 shutdown_connections 2024-03-19T03:31:37.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.450+0000 7f3787c00700 1 -- 172.21.15.117:0/1554123198 wait complete. 2024-03-19T03:31:37.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.451+0000 7f3787c00700 1 Processor -- start 2024-03-19T03:31:37.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.452+0000 7f3787c00700 1 -- start start 2024-03-19T03:31:37.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.452+0000 7f3787c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3780101210 0x7f3780102ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:37.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.452+0000 7f3787c00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3780101bb0 0x7f3780103400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:37.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.452+0000 7f3787c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3780107250 0x7f3780103940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:37.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.452+0000 7f3787c00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f37801114e0 con 0x7f3780101210 2024-03-19T03:31:37.453 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.452+0000 7f3787c00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3780111360 con 0x7f3780107250 2024-03-19T03:31:37.453 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.452+0000 7f3787c00700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f3780111660 con 0x7f3780101bb0 2024-03-19T03:31:37.453 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.452+0000 7f377fe00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3780101bb0 0x7f3780103400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:37.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.452+0000 7f3785c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3780107250 0x7f3780103940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:37.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.452+0000 7f3785200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3780101210 0x7f3780102ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:37.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.452+0000 7f377fe00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3780101bb0 0x7f3780103400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:52786/0 (socket says 172.21.15.117:52786) 2024-03-19T03:31:37.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.452+0000 7f3785c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3780107250 0x7f3780103940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:44730/0 (socket says 172.21.15.117:44730) 2024-03-19T03:31:37.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.452+0000 7f377fe00700 1 -- 172.21.15.117:0/3148915754 learned_addr learned my addr 172.21.15.117:0/3148915754 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:37.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.453+0000 7f377fe00700 1 -- 172.21.15.117:0/3148915754 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3780107250 msgr2=0x7f3780103940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:37.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.453+0000 7f377fe00700 1 --2- 172.21.15.117:0/3148915754 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3780107250 0x7f3780103940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:37.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.453+0000 7f377fe00700 1 -- 172.21.15.117:0/3148915754 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3780101210 msgr2=0x7f3780102ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:37.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.453+0000 7f377fe00700 1 --2- 172.21.15.117:0/3148915754 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3780101210 0x7f3780102ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:37.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.453+0000 7f377fe00700 1 -- 172.21.15.117:0/3148915754 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37801a6ff0 con 0x7f3780101bb0 2024-03-19T03:31:37.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.453+0000 7f377fe00700 1 --2- 172.21.15.117:0/3148915754 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3780101bb0 0x7f3780103400 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f376c00e9c0 tx=0x7f376c00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:37.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.453+0000 7f377d600700 1 -- 172.21.15.117:0/3148915754 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f376c004a30 con 0x7f3780101bb0 2024-03-19T03:31:37.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.454+0000 7f3787c00700 1 -- 172.21.15.117:0/3148915754 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37801a72e0 con 0x7f3780101bb0 2024-03-19T03:31:37.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.454+0000 7f377d600700 1 -- 172.21.15.117:0/3148915754 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f376c00cca0 con 0x7f3780101bb0 2024-03-19T03:31:37.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.454+0000 7f377d600700 1 -- 172.21.15.117:0/3148915754 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f376c005610 con 0x7f3780101bb0 2024-03-19T03:31:37.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.454+0000 7f3787c00700 1 -- 172.21.15.117:0/3148915754 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37801a7870 con 0x7f3780101bb0 2024-03-19T03:31:37.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.455+0000 7f3787c00700 1 -- 172.21.15.117:0/3148915754 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f37801052d0 con 0x7f3780101bb0 2024-03-19T03:31:37.456 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.457+0000 7f377d600700 1 -- 172.21.15.117:0/3148915754 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f376c004cf0 con 0x7f3780101bb0 2024-03-19T03:31:37.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.459+0000 7f377d600700 1 --2- 172.21.15.117:0/3148915754 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f37680a29a0 0x7f37680a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:37.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.459+0000 7f377d600700 1 -- 172.21.15.117:0/3148915754 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f376c0ceff0 con 0x7f3780101bb0 2024-03-19T03:31:37.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.459+0000 7f3785200700 1 --2- 172.21.15.117:0/3148915754 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f37680a29a0 0x7f37680a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:37.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.460+0000 7f377d600700 1 --2- 172.21.15.117:0/3148915754 >> [v2:172.21.15.117:6818/864146757,v1:172.21.15.117:6819/864146757] conn(0x7f37680ad660 0x7f37680afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:37.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.460+0000 7f377d600700 1 -- 172.21.15.117:0/3148915754 --> [v2:172.21.15.117:6818/864146757,v1:172.21.15.117:6819/864146757] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f376c00ce50 con 0x7f37680ad660 2024-03-19T03:31:37.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.460+0000 7f377d600700 1 -- 172.21.15.117:0/3148915754 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f376c0cf400 con 0x7f3780101bb0 2024-03-19T03:31:37.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.460+0000 7f3785c00700 1 --2- 172.21.15.117:0/3148915754 >> [v2:172.21.15.117:6818/864146757,v1:172.21.15.117:6819/864146757] conn(0x7f37680ad660 0x7f37680afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:37.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.460+0000 7f3785200700 1 --2- 172.21.15.117:0/3148915754 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f37680a29a0 0x7f37680a4e60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3774009810 tx=0x7f3774019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:37.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.461+0000 7f3785c00700 1 --2- 172.21.15.117:0/3148915754 >> [v2:172.21.15.117:6818/864146757,v1:172.21.15.117:6819/864146757] conn(0x7f37680ad660 0x7f37680afaa0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:37.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.462+0000 7f377d600700 1 -- 172.21.15.117:0/3148915754 <== osd.6 v2:172.21.15.117:6818/864146757 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f376c00ce50 con 0x7f37680ad660 2024-03-19T03:31:37.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.480+0000 7f3787c00700 1 -- 172.21.15.117:0/3148915754 --> [v2:172.21.15.117:6818/864146757,v1:172.21.15.117:6819/864146757] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3780102900 con 0x7f37680ad660 2024-03-19T03:31:37.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.480+0000 7f377d600700 1 -- 172.21.15.117:0/3148915754 <== osd.6 v2:172.21.15.117:6818/864146757 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3780102900 con 0x7f37680ad660 2024-03-19T03:31:37.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.481+0000 7f3787c00700 1 -- 172.21.15.117:0/3148915754 >> [v2:172.21.15.117:6818/864146757,v1:172.21.15.117:6819/864146757] conn(0x7f37680ad660 msgr2=0x7f37680afaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:37.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.481+0000 7f3787c00700 1 --2- 172.21.15.117:0/3148915754 >> [v2:172.21.15.117:6818/864146757,v1:172.21.15.117:6819/864146757] conn(0x7f37680ad660 0x7f37680afaa0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:37.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.481+0000 7f3787c00700 1 -- 172.21.15.117:0/3148915754 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f37680a29a0 msgr2=0x7f37680a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:37.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.481+0000 7f3787c00700 1 --2- 172.21.15.117:0/3148915754 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f37680a29a0 0x7f37680a4e60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3774009810 tx=0x7f3774019040 comp rx=0 tx=0).stop 2024-03-19T03:31:37.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.481+0000 7f3787c00700 1 -- 172.21.15.117:0/3148915754 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3780101bb0 msgr2=0x7f3780103400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:37.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.481+0000 7f3787c00700 1 --2- 172.21.15.117:0/3148915754 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3780101bb0 0x7f3780103400 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f376c00e9c0 tx=0x7f376c00ecd0 comp rx=0 tx=0).stop 2024-03-19T03:31:37.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.481+0000 7f3785c00700 1 -- 172.21.15.117:0/3148915754 reap_dead start 2024-03-19T03:31:37.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.481+0000 7f3787c00700 1 -- 172.21.15.117:0/3148915754 shutdown_connections 2024-03-19T03:31:37.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.481+0000 7f3787c00700 1 -- 172.21.15.117:0/3148915754 >> 172.21.15.117:0/3148915754 conn(0x7f3780078140 msgr2=0x7f3780100570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:37.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.481+0000 7f3787c00700 1 -- 172.21.15.117:0/3148915754 shutdown_connections 2024-03-19T03:31:37.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:37.481+0000 7f3787c00700 1 -- 172.21.15.117:0/3148915754 wait complete. 2024-03-19T03:31:37.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:37 smithi117 ceph-mon[31838]: pgmap v348: 129 pgs: 129 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-19T03:31:37.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:31:37.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:31:37.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:31:37.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:31:37.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:37 smithi119 ceph-mon[66861]: pgmap v348: 129 pgs: 129 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-19T03:31:37.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:37 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:31:37.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:37 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:31:37.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:37 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:31:37.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:37 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:31:37.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:37 smithi178 ceph-mon[67578]: pgmap v348: 129 pgs: 129 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-19T03:31:37.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:31:37.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:31:37.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:31:37.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:31:38.517 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:38 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]: dispatch 2024-03-19T03:31:38.518 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:38 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7, 5, 8]}]: dispatch 2024-03-19T03:31:38.518 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:38 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-03-19T03:31:38.518 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:38 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-03-19T03:31:38.518 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:38 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2024-03-19T03:31:38.518 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:38 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]: dispatch 2024-03-19T03:31:38.518 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:38 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]: dispatch 2024-03-19T03:31:38.518 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:38 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-03-19T03:31:38.518 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:38 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-03-19T03:31:38.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:38 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]: dispatch 2024-03-19T03:31:38.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:38 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7, 5, 8]}]: dispatch 2024-03-19T03:31:38.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:38 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-03-19T03:31:38.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:38 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-03-19T03:31:38.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:38 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2024-03-19T03:31:38.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:38 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]: dispatch 2024-03-19T03:31:38.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:38 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]: dispatch 2024-03-19T03:31:38.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:38 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-03-19T03:31:38.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:38 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-03-19T03:31:38.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:38 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]: dispatch 2024-03-19T03:31:38.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:38 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7, 5, 8]}]: dispatch 2024-03-19T03:31:38.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:38 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-03-19T03:31:38.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:38 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-03-19T03:31:38.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:38 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2024-03-19T03:31:38.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:38 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]: dispatch 2024-03-19T03:31:38.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:38 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]: dispatch 2024-03-19T03:31:38.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:38 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-03-19T03:31:38.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:38 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-03-19T03:31:39.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.427+0000 7f7c57e00700 1 -- 172.21.15.117:0/231980905 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c58100fa0 msgr2=0x7f7c58101380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:39.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.427+0000 7f7c57e00700 1 --2- 172.21.15.117:0/231980905 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c58100fa0 0x7f7c58101380 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7c48009990 tx=0x7f7c48009ca0 comp rx=0 tx=0).stop 2024-03-19T03:31:39.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.427+0000 7f7c57e00700 1 -- 172.21.15.117:0/231980905 shutdown_connections 2024-03-19T03:31:39.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.427+0000 7f7c57e00700 1 --2- 172.21.15.117:0/231980905 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c5810eb50 0x7f7c58110f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:39.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.427+0000 7f7c57e00700 1 --2- 172.21.15.117:0/231980905 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c58100fa0 0x7f7c58101380 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:39.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.427+0000 7f7c57e00700 1 --2- 172.21.15.117:0/231980905 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c58101950 0x7f7c5810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:39.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.427+0000 7f7c57e00700 1 -- 172.21.15.117:0/231980905 >> 172.21.15.117:0/231980905 conn(0x7f7c580fcbe0 msgr2=0x7f7c580ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:39.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.427+0000 7f7c57e00700 1 -- 172.21.15.117:0/231980905 shutdown_connections 2024-03-19T03:31:39.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.427+0000 7f7c57e00700 1 -- 172.21.15.117:0/231980905 wait complete. 2024-03-19T03:31:39.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.428+0000 7f7c57e00700 1 Processor -- start 2024-03-19T03:31:39.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.429+0000 7f7c57e00700 1 -- start start 2024-03-19T03:31:39.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.429+0000 7f7c57e00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c58100fa0 0x7f7c581a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:39.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.429+0000 7f7c57e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c58101950 0x7f7c581a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:39.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.429+0000 7f7c57e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c5810eb50 0x7f7c5819aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:39.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.429+0000 7f7c57e00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7c58114780 con 0x7f7c58101950 2024-03-19T03:31:39.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.429+0000 7f7c57e00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7c58114600 con 0x7f7c5810eb50 2024-03-19T03:31:39.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.429+0000 7f7c57e00700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f7c58114900 con 0x7f7c58100fa0 2024-03-19T03:31:39.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.429+0000 7f7c56a00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c58100fa0 0x7f7c581a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:39.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.429+0000 7f7c56000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c58101950 0x7f7c581a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:39.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.429+0000 7f7c56a00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c58100fa0 0x7f7c581a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:52782/0 (socket says 172.21.15.117:52782) 2024-03-19T03:31:39.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.429+0000 7f7c57400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c5810eb50 0x7f7c5819aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:39.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.429+0000 7f7c56a00700 1 -- 172.21.15.117:0/1476596408 learned_addr learned my addr 172.21.15.117:0/1476596408 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:39.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.430+0000 7f7c56a00700 1 -- 172.21.15.117:0/1476596408 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c5810eb50 msgr2=0x7f7c5819aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:39.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.430+0000 7f7c56a00700 1 --2- 172.21.15.117:0/1476596408 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c5810eb50 0x7f7c5819aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:39.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.430+0000 7f7c56a00700 1 -- 172.21.15.117:0/1476596408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c58101950 msgr2=0x7f7c581a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:39.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.430+0000 7f7c56a00700 1 --2- 172.21.15.117:0/1476596408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c58101950 0x7f7c581a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:39.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.430+0000 7f7c56a00700 1 -- 172.21.15.117:0/1476596408 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c5819b160 con 0x7f7c58100fa0 2024-03-19T03:31:39.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.430+0000 7f7c56a00700 1 --2- 172.21.15.117:0/1476596408 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c58100fa0 0x7f7c581a0840 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7c48005910 tx=0x7f7c4800bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:39.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.430+0000 7f7c4f400700 1 -- 172.21.15.117:0/1476596408 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c48027070 con 0x7f7c58100fa0 2024-03-19T03:31:39.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.430+0000 7f7c4f400700 1 -- 172.21.15.117:0/1476596408 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c48004650 con 0x7f7c58100fa0 2024-03-19T03:31:39.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.430+0000 7f7c57e00700 1 -- 172.21.15.117:0/1476596408 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c5819b3f0 con 0x7f7c58100fa0 2024-03-19T03:31:39.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.430+0000 7f7c4f400700 1 -- 172.21.15.117:0/1476596408 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4800f460 con 0x7f7c58100fa0 2024-03-19T03:31:39.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.431+0000 7f7c57e00700 1 -- 172.21.15.117:0/1476596408 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c581a7710 con 0x7f7c58100fa0 2024-03-19T03:31:39.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.432+0000 7f7c57e00700 1 -- 172.21.15.117:0/1476596408 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7c58062aa0 con 0x7f7c58100fa0 2024-03-19T03:31:39.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.434+0000 7f7c4f400700 1 -- 172.21.15.117:0/1476596408 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f7c4802a070 con 0x7f7c58100fa0 2024-03-19T03:31:39.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.436+0000 7f7c4f400700 1 --2- 172.21.15.117:0/1476596408 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7c3c0a29a0 0x7f7c3c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:39.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.437+0000 7f7c4f400700 1 -- 172.21.15.117:0/1476596408 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7c480cfcb0 con 0x7f7c58100fa0 2024-03-19T03:31:39.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.437+0000 7f7c56000700 1 --2- 172.21.15.117:0/1476596408 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7c3c0a29a0 0x7f7c3c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:39.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.437+0000 7f7c4f400700 1 --2- 172.21.15.117:0/1476596408 >> [v2:172.21.15.117:6802/3014934900,v1:172.21.15.117:6803/3014934900] conn(0x7f7c3c0ad660 0x7f7c3c0afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:39.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.437+0000 7f7c4f400700 1 -- 172.21.15.117:0/1476596408 --> [v2:172.21.15.117:6802/3014934900,v1:172.21.15.117:6803/3014934900] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7c4802be50 con 0x7f7c3c0ad660 2024-03-19T03:31:39.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.437+0000 7f7c4f400700 1 -- 172.21.15.117:0/1476596408 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f7c480d2250 con 0x7f7c58100fa0 2024-03-19T03:31:39.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.437+0000 7f7c57400700 1 --2- 172.21.15.117:0/1476596408 >> [v2:172.21.15.117:6802/3014934900,v1:172.21.15.117:6803/3014934900] conn(0x7f7c3c0ad660 0x7f7c3c0afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:39.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.437+0000 7f7c56000700 1 --2- 172.21.15.117:0/1476596408 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7c3c0a29a0 0x7f7c3c0a4e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7c580ff020 tx=0x7f7c44009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:39.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.438+0000 7f7c57400700 1 --2- 172.21.15.117:0/1476596408 >> [v2:172.21.15.117:6802/3014934900,v1:172.21.15.117:6803/3014934900] conn(0x7f7c3c0ad660 0x7f7c3c0afaa0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:39.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.439+0000 7f7c4f400700 1 -- 172.21.15.117:0/1476596408 <== osd.0 v2:172.21.15.117:6802/3014934900 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7c4802be50 con 0x7f7c3c0ad660 2024-03-19T03:31:39.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.458+0000 7f7c57e00700 1 -- 172.21.15.117:0/1476596408 --> [v2:172.21.15.117:6802/3014934900,v1:172.21.15.117:6803/3014934900] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7c58062ea0 con 0x7f7c3c0ad660 2024-03-19T03:31:39.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.458+0000 7f7c4f400700 1 -- 172.21.15.117:0/1476596408 <== osd.0 v2:172.21.15.117:6802/3014934900 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7c58062ea0 con 0x7f7c3c0ad660 2024-03-19T03:31:39.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.458+0000 7f7c57e00700 1 -- 172.21.15.117:0/1476596408 >> [v2:172.21.15.117:6802/3014934900,v1:172.21.15.117:6803/3014934900] conn(0x7f7c3c0ad660 msgr2=0x7f7c3c0afaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:39.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.458+0000 7f7c57e00700 1 --2- 172.21.15.117:0/1476596408 >> [v2:172.21.15.117:6802/3014934900,v1:172.21.15.117:6803/3014934900] conn(0x7f7c3c0ad660 0x7f7c3c0afaa0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:39.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.459+0000 7f7c57e00700 1 -- 172.21.15.117:0/1476596408 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7c3c0a29a0 msgr2=0x7f7c3c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:39.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.459+0000 7f7c57e00700 1 --2- 172.21.15.117:0/1476596408 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7c3c0a29a0 0x7f7c3c0a4e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7c580ff020 tx=0x7f7c44009260 comp rx=0 tx=0).stop 2024-03-19T03:31:39.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.459+0000 7f7c57e00700 1 -- 172.21.15.117:0/1476596408 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c58100fa0 msgr2=0x7f7c581a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:39.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.459+0000 7f7c57e00700 1 --2- 172.21.15.117:0/1476596408 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c58100fa0 0x7f7c581a0840 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7c48005910 tx=0x7f7c4800bab0 comp rx=0 tx=0).stop 2024-03-19T03:31:39.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.459+0000 7f7c57400700 1 -- 172.21.15.117:0/1476596408 reap_dead start 2024-03-19T03:31:39.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.459+0000 7f7c57e00700 1 -- 172.21.15.117:0/1476596408 shutdown_connections 2024-03-19T03:31:39.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.459+0000 7f7c57e00700 1 -- 172.21.15.117:0/1476596408 >> 172.21.15.117:0/1476596408 conn(0x7f7c580fcbe0 msgr2=0x7f7c58102870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:39.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.459+0000 7f7c57e00700 1 -- 172.21.15.117:0/1476596408 shutdown_connections 2024-03-19T03:31:39.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:36.459+0000 7f7c57e00700 1 -- 172.21.15.117:0/1476596408 wait complete. 2024-03-19T03:31:39.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:39 smithi119 ceph-mon[66861]: pgmap v349: 129 pgs: 129 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-19T03:31:39.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]': finished 2024-03-19T03:31:39.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7, 5, 8]}]': finished 2024-03-19T03:31:39.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-03-19T03:31:39.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-03-19T03:31:39.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-03-19T03:31:39.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]': finished 2024-03-19T03:31:39.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]': finished 2024-03-19T03:31:39.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-03-19T03:31:39.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-03-19T03:31:39.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:39 smithi119 ceph-mon[66861]: osdmap e90: 12 total, 12 up, 12 in 2024-03-19T03:31:39.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:39 smithi117 ceph-mon[31838]: pgmap v349: 129 pgs: 129 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-19T03:31:39.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]': finished 2024-03-19T03:31:39.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7, 5, 8]}]': finished 2024-03-19T03:31:39.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-03-19T03:31:39.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-03-19T03:31:39.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-03-19T03:31:39.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]': finished 2024-03-19T03:31:39.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]': finished 2024-03-19T03:31:39.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-03-19T03:31:39.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-03-19T03:31:39.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:39 smithi117 ceph-mon[31838]: osdmap e90: 12 total, 12 up, 12 in 2024-03-19T03:31:39.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:39 smithi178 ceph-mon[67578]: pgmap v349: 129 pgs: 129 active+clean; 645 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-19T03:31:39.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]': finished 2024-03-19T03:31:39.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7, 5, 8]}]': finished 2024-03-19T03:31:39.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-03-19T03:31:39.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-03-19T03:31:39.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-03-19T03:31:39.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]': finished 2024-03-19T03:31:39.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]': finished 2024-03-19T03:31:39.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-03-19T03:31:39.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-03-19T03:31:39.847 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:39 smithi178 ceph-mon[67578]: osdmap e90: 12 total, 12 up, 12 in 2024-03-19T03:31:40.523 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:40 smithi117 ceph-mon[31838]: osdmap e91: 12 total, 12 up, 12 in 2024-03-19T03:31:40.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:40 smithi119 ceph-mon[66861]: osdmap e91: 12 total, 12 up, 12 in 2024-03-19T03:31:40.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:40 smithi178 ceph-mon[67578]: osdmap e91: 12 total, 12 up, 12 in 2024-03-19T03:31:41.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.692+0000 7f6651e00700 1 -- 172.21.15.117:0/616465001 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f664c074fd0 msgr2=0x7f664c112dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:41.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.692+0000 7f6651e00700 1 --2- 172.21.15.117:0/616465001 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f664c074fd0 0x7f664c112dc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6638009990 tx=0x7f6638009ca0 comp rx=0 tx=0).stop 2024-03-19T03:31:41.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.692+0000 7f6651e00700 1 -- 172.21.15.117:0/616465001 shutdown_connections 2024-03-19T03:31:41.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.692+0000 7f6651e00700 1 --2- 172.21.15.117:0/616465001 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f664c113300 0x7f664c1156f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:41.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.692+0000 7f6651e00700 1 --2- 172.21.15.117:0/616465001 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f664c0746b0 0x7f664c074a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:41.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.692+0000 7f6651e00700 1 --2- 172.21.15.117:0/616465001 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f664c074fd0 0x7f664c112dc0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:41.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.692+0000 7f6651e00700 1 -- 172.21.15.117:0/616465001 >> 172.21.15.117:0/616465001 conn(0x7f664c0fddc0 msgr2=0x7f664c1001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:41.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.693+0000 7f6651e00700 1 -- 172.21.15.117:0/616465001 shutdown_connections 2024-03-19T03:31:41.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.693+0000 7f6651e00700 1 -- 172.21.15.117:0/616465001 wait complete. 2024-03-19T03:31:41.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.694+0000 7f6651e00700 1 Processor -- start 2024-03-19T03:31:41.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.694+0000 7f6651e00700 1 -- start start 2024-03-19T03:31:41.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.695+0000 7f6651e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f664c0746b0 0x7f664c19f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:41.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.695+0000 7f6651e00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f664c074fd0 0x7f664c19f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:41.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.695+0000 7f6651e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f664c113300 0x7f664c1a39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:41.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.695+0000 7f6651e00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f664c118c40 con 0x7f664c0746b0 2024-03-19T03:31:41.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.695+0000 7f6651e00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f664c118ac0 con 0x7f664c113300 2024-03-19T03:31:41.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.695+0000 7f6651e00700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f664c118dc0 con 0x7f664c074fd0 2024-03-19T03:31:41.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.695+0000 7f664aa00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f664c0746b0 0x7f664c19f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:41.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.695+0000 7f664aa00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f664c0746b0 0x7f664c19f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:53266/0 (socket says 172.21.15.117:53266) 2024-03-19T03:31:41.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.695+0000 7f664aa00700 1 -- 172.21.15.117:0/4132917906 learned_addr learned my addr 172.21.15.117:0/4132917906 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:41.696 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:41 smithi117 ceph-mon[31838]: pgmap v352: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 895 B/s wr, 10 op/s 2024-03-19T03:31:41.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.695+0000 7f664b400700 1 --2- 172.21.15.117:0/4132917906 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f664c113300 0x7f664c1a39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:41.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.695+0000 7f664a000700 1 --2- 172.21.15.117:0/4132917906 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f664c074fd0 0x7f664c19f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:41.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.696+0000 7f664aa00700 1 -- 172.21.15.117:0/4132917906 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f664c074fd0 msgr2=0x7f664c19f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:41.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.696+0000 7f664aa00700 1 --2- 172.21.15.117:0/4132917906 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f664c074fd0 0x7f664c19f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:41.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.696+0000 7f664aa00700 1 -- 172.21.15.117:0/4132917906 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f664c113300 msgr2=0x7f664c1a39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:41.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.696+0000 7f664aa00700 1 --2- 172.21.15.117:0/4132917906 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f664c113300 0x7f664c1a39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:41.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.696+0000 7f664aa00700 1 -- 172.21.15.117:0/4132917906 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f664c1a40e0 con 0x7f664c0746b0 2024-03-19T03:31:41.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.696+0000 7f664aa00700 1 --2- 172.21.15.117:0/4132917906 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f664c0746b0 0x7f664c19f0d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f663c00e9a0 tx=0x7f663c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:41.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.697+0000 7f6650a00700 1 -- 172.21.15.117:0/4132917906 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663c00ca10 con 0x7f664c0746b0 2024-03-19T03:31:41.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.697+0000 7f6650a00700 1 -- 172.21.15.117:0/4132917906 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f663c00cbb0 con 0x7f664c0746b0 2024-03-19T03:31:41.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.697+0000 7f6651e00700 1 -- 172.21.15.117:0/4132917906 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f664c1a43d0 con 0x7f664c0746b0 2024-03-19T03:31:41.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.697+0000 7f6650a00700 1 -- 172.21.15.117:0/4132917906 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663c018680 con 0x7f664c0746b0 2024-03-19T03:31:41.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.697+0000 7f6651e00700 1 -- 172.21.15.117:0/4132917906 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f664c1abc50 con 0x7f664c0746b0 2024-03-19T03:31:41.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.698+0000 7f6651e00700 1 -- 172.21.15.117:0/4132917906 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f664c062aa0 con 0x7f664c0746b0 2024-03-19T03:31:41.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.701+0000 7f6650a00700 1 -- 172.21.15.117:0/4132917906 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f663c010430 con 0x7f664c0746b0 2024-03-19T03:31:41.702 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.702+0000 7f6650a00700 1 --2- 172.21.15.117:0/4132917906 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f66300a29a0 0x7f66300a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:41.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.702+0000 7f6650a00700 1 -- 172.21.15.117:0/4132917906 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f663c0cf940 con 0x7f664c0746b0 2024-03-19T03:31:41.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.702+0000 7f664a000700 1 --2- 172.21.15.117:0/4132917906 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f66300a29a0 0x7f66300a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:41.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.703+0000 7f6650a00700 1 --2- 172.21.15.117:0/4132917906 >> [v2:172.21.15.119:6816/1273365545,v1:172.21.15.119:6817/1273365545] conn(0x7f66300ada80 0x7f66300aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:41.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.703+0000 7f6650a00700 1 -- 172.21.15.117:0/4132917906 --> [v2:172.21.15.119:6816/1273365545,v1:172.21.15.119:6817/1273365545] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f663c01bec0 con 0x7f66300ada80 2024-03-19T03:31:41.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.703+0000 7f6650a00700 1 -- 172.21.15.117:0/4132917906 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f663c0cfcf0 con 0x7f664c0746b0 2024-03-19T03:31:41.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.703+0000 7f664b400700 1 --2- 172.21.15.117:0/4132917906 >> [v2:172.21.15.119:6816/1273365545,v1:172.21.15.119:6817/1273365545] conn(0x7f66300ada80 0x7f66300aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:41.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.703+0000 7f664a000700 1 --2- 172.21.15.117:0/4132917906 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f66300a29a0 0x7f66300a4e60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f66380098e0 tx=0x7f6638005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:41.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.704+0000 7f664b400700 1 --2- 172.21.15.117:0/4132917906 >> [v2:172.21.15.119:6816/1273365545,v1:172.21.15.119:6817/1273365545] conn(0x7f66300ada80 0x7f66300aff20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:41.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.705+0000 7f6650a00700 1 -- 172.21.15.117:0/4132917906 <== osd.7 v2:172.21.15.119:6816/1273365545 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f663c01bec0 con 0x7f66300ada80 2024-03-19T03:31:41.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.723+0000 7f6651e00700 1 -- 172.21.15.117:0/4132917906 --> [v2:172.21.15.119:6816/1273365545,v1:172.21.15.119:6817/1273365545] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f664c062f20 con 0x7f66300ada80 2024-03-19T03:31:41.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.724+0000 7f6650a00700 1 -- 172.21.15.117:0/4132917906 <== osd.7 v2:172.21.15.119:6816/1273365545 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f664c062f20 con 0x7f66300ada80 2024-03-19T03:31:41.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.724+0000 7f6651e00700 1 -- 172.21.15.117:0/4132917906 >> [v2:172.21.15.119:6816/1273365545,v1:172.21.15.119:6817/1273365545] conn(0x7f66300ada80 msgr2=0x7f66300aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:41.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.724+0000 7f6651e00700 1 --2- 172.21.15.117:0/4132917906 >> [v2:172.21.15.119:6816/1273365545,v1:172.21.15.119:6817/1273365545] conn(0x7f66300ada80 0x7f66300aff20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:41.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.724+0000 7f6651e00700 1 -- 172.21.15.117:0/4132917906 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f66300a29a0 msgr2=0x7f66300a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:41.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.724+0000 7f6651e00700 1 --2- 172.21.15.117:0/4132917906 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f66300a29a0 0x7f66300a4e60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f66380098e0 tx=0x7f6638005960 comp rx=0 tx=0).stop 2024-03-19T03:31:41.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.724+0000 7f6651e00700 1 -- 172.21.15.117:0/4132917906 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f664c0746b0 msgr2=0x7f664c19f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:41.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.724+0000 7f6651e00700 1 --2- 172.21.15.117:0/4132917906 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f664c0746b0 0x7f664c19f0d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f663c00e9a0 tx=0x7f663c00ecb0 comp rx=0 tx=0).stop 2024-03-19T03:31:41.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.724+0000 7f664b400700 1 -- 172.21.15.117:0/4132917906 reap_dead start 2024-03-19T03:31:41.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.724+0000 7f6651e00700 1 -- 172.21.15.117:0/4132917906 shutdown_connections 2024-03-19T03:31:41.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.724+0000 7f6651e00700 1 -- 172.21.15.117:0/4132917906 >> 172.21.15.117:0/4132917906 conn(0x7f664c0fddc0 msgr2=0x7f664c0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:41.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.725+0000 7f6651e00700 1 -- 172.21.15.117:0/4132917906 shutdown_connections 2024-03-19T03:31:41.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:41.725+0000 7f6651e00700 1 -- 172.21.15.117:0/4132917906 wait complete. 2024-03-19T03:31:41.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:41 smithi119 ceph-mon[66861]: pgmap v352: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 895 B/s wr, 10 op/s 2024-03-19T03:31:41.820 INFO:teuthology.orchestra.run.smithi117.stdout:197568495672 2024-03-19T03:31:41.820 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-19T03:31:41.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:41 smithi178 ceph-mon[67578]: pgmap v352: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 895 B/s wr, 10 op/s 2024-03-19T03:31:42.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:42 smithi117 ceph-mon[31838]: pgmap v353: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 9 op/s; 14 B/s, 0 objects/s recovering 2024-03-19T03:31:42.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:42 smithi117 ceph-mon[31838]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-19T03:31:42.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:42 smithi178 ceph-mon[67578]: pgmap v353: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 9 op/s; 14 B/s, 0 objects/s recovering 2024-03-19T03:31:42.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:42 smithi178 ceph-mon[67578]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-19T03:31:42.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:42 smithi119 ceph-mon[66861]: pgmap v353: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 9 op/s; 14 B/s, 0 objects/s recovering 2024-03-19T03:31:42.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:42 smithi119 ceph-mon[66861]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-19T03:31:44.636 INFO:teuthology.orchestra.run.smithi117.stdout:219043332146 2024-03-19T03:31:44.637 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-19T03:31:44.715 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:44 smithi117 ceph-mon[31838]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-19T03:31:44.715 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:44 smithi117 ceph-mon[31838]: Cluster is now healthy 2024-03-19T03:31:44.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:44 smithi119 ceph-mon[66861]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-19T03:31:44.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:44 smithi119 ceph-mon[66861]: Cluster is now healthy 2024-03-19T03:31:44.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:44 smithi178 ceph-mon[67578]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-19T03:31:44.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:44 smithi178 ceph-mon[67578]: Cluster is now healthy 2024-03-19T03:31:45.626 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:45 smithi117 ceph-mon[31838]: pgmap v354: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 10 op/s; 14 B/s, 0 objects/s recovering 2024-03-19T03:31:45.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.691+0000 7f5374e00700 1 -- 172.21.15.117:0/146029297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5370107530 msgr2=0x7f537010f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:45.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.691+0000 7f5374e00700 1 --2- 172.21.15.117:0/146029297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5370107530 0x7f537010f100 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f536800b0a0 tx=0x7f536800b3b0 comp rx=0 tx=0).stop 2024-03-19T03:31:45.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.692+0000 7f5374e00700 1 -- 172.21.15.117:0/146029297 shutdown_connections 2024-03-19T03:31:45.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.692+0000 7f5374e00700 1 --2- 172.21.15.117:0/146029297 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5370107530 0x7f537010f100 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:45.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.692+0000 7f5374e00700 1 --2- 172.21.15.117:0/146029297 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5370102450 0x7f5370102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:45.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.692+0000 7f5374e00700 1 --2- 172.21.15.117:0/146029297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5370102e00 0x7f5370106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:45.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.692+0000 7f5374e00700 1 -- 172.21.15.117:0/146029297 >> 172.21.15.117:0/146029297 conn(0x7f53700fdce0 msgr2=0x7f5370100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:45.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.692+0000 7f5374e00700 1 -- 172.21.15.117:0/146029297 shutdown_connections 2024-03-19T03:31:45.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.692+0000 7f5374e00700 1 -- 172.21.15.117:0/146029297 wait complete. 2024-03-19T03:31:45.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.693+0000 7f5374e00700 1 Processor -- start 2024-03-19T03:31:45.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.694+0000 7f5374e00700 1 -- start start 2024-03-19T03:31:45.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.694+0000 7f5374e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5370102450 0x7f5370074610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:45.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.694+0000 7f5374e00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5370102e00 0x7f5370074b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:45.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.694+0000 7f5374e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5370107530 0x7f5370075090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:45.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.694+0000 7f5374e00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5370112590 con 0x7f5370102450 2024-03-19T03:31:45.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.694+0000 7f5374e00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f5370112410 con 0x7f5370107530 2024-03-19T03:31:45.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.694+0000 7f5374e00700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f5370112710 con 0x7f5370102e00 2024-03-19T03:31:45.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.694+0000 7f536f400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5370102450 0x7f5370074610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:45.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.695+0000 7f536f400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5370102450 0x7f5370074610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:53288/0 (socket says 172.21.15.117:53288) 2024-03-19T03:31:45.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.695+0000 7f536f400700 1 -- 172.21.15.117:0/2596632648 learned_addr learned my addr 172.21.15.117:0/2596632648 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:45.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.695+0000 7f536fe00700 1 --2- 172.21.15.117:0/2596632648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5370107530 0x7f5370075090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:45.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.695+0000 7f536ea00700 1 --2- 172.21.15.117:0/2596632648 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5370102e00 0x7f5370074b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:45.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.695+0000 7f536f400700 1 -- 172.21.15.117:0/2596632648 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5370102e00 msgr2=0x7f5370074b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:45.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.695+0000 7f536f400700 1 --2- 172.21.15.117:0/2596632648 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5370102e00 0x7f5370074b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:45.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.695+0000 7f536f400700 1 -- 172.21.15.117:0/2596632648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5370107530 msgr2=0x7f5370075090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:45.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.695+0000 7f536f400700 1 --2- 172.21.15.117:0/2596632648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5370107530 0x7f5370075090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:45.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.695+0000 7f536f400700 1 -- 172.21.15.117:0/2596632648 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53701a6ee0 con 0x7f5370102450 2024-03-19T03:31:45.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.695+0000 7f536ea00700 1 --2- 172.21.15.117:0/2596632648 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5370102e00 0x7f5370074b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:31:45.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.695+0000 7f536f400700 1 --2- 172.21.15.117:0/2596632648 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5370102450 0x7f5370074610 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f536000eb30 tx=0x7f536000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:45.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.696+0000 7f5367e00700 1 -- 172.21.15.117:0/2596632648 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f536000cbd0 con 0x7f5370102450 2024-03-19T03:31:45.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.696+0000 7f5374e00700 1 -- 172.21.15.117:0/2596632648 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53701a71d0 con 0x7f5370102450 2024-03-19T03:31:45.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.696+0000 7f5367e00700 1 -- 172.21.15.117:0/2596632648 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5360004500 con 0x7f5370102450 2024-03-19T03:31:45.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.696+0000 7f5367e00700 1 -- 172.21.15.117:0/2596632648 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5360010430 con 0x7f5370102450 2024-03-19T03:31:45.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.696+0000 7f5374e00700 1 -- 172.21.15.117:0/2596632648 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53701a7760 con 0x7f5370102450 2024-03-19T03:31:45.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.697+0000 7f5374e00700 1 -- 172.21.15.117:0/2596632648 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5370062a20 con 0x7f5370102450 2024-03-19T03:31:45.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.700+0000 7f5367e00700 1 -- 172.21.15.117:0/2596632648 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f5360003f00 con 0x7f5370102450 2024-03-19T03:31:45.701 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.701+0000 7f5367e00700 1 --2- 172.21.15.117:0/2596632648 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f53540a2a20 0x7f53540a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:45.701 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.701+0000 7f5367e00700 1 -- 172.21.15.117:0/2596632648 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f53600cf4d0 con 0x7f5370102450 2024-03-19T03:31:45.702 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.701+0000 7f536ea00700 1 --2- 172.21.15.117:0/2596632648 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f53540a2a20 0x7f53540a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:45.702 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.702+0000 7f5367e00700 1 --2- 172.21.15.117:0/2596632648 >> [v2:172.21.15.117:6826/3653875600,v1:172.21.15.117:6827/3653875600] conn(0x7f53540adb00 0x7f53540affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:45.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.702+0000 7f5367e00700 1 -- 172.21.15.117:0/2596632648 --> [v2:172.21.15.117:6826/3653875600,v1:172.21.15.117:6827/3653875600] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5360012ec0 con 0x7f53540adb00 2024-03-19T03:31:45.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.702+0000 7f5367e00700 1 -- 172.21.15.117:0/2596632648 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f53600cf8c0 con 0x7f5370102450 2024-03-19T03:31:45.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.702+0000 7f536fe00700 1 --2- 172.21.15.117:0/2596632648 >> [v2:172.21.15.117:6826/3653875600,v1:172.21.15.117:6827/3653875600] conn(0x7f53540adb00 0x7f53540affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:45.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.702+0000 7f536ea00700 1 --2- 172.21.15.117:0/2596632648 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f53540a2a20 0x7f53540a4ee0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5370108ae0 tx=0x7f535c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:45.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.703+0000 7f536fe00700 1 --2- 172.21.15.117:0/2596632648 >> [v2:172.21.15.117:6826/3653875600,v1:172.21.15.117:6827/3653875600] conn(0x7f53540adb00 0x7f53540affa0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:45.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.704+0000 7f5367e00700 1 -- 172.21.15.117:0/2596632648 <== osd.9 v2:172.21.15.117:6826/3653875600 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5360012ec0 con 0x7f53540adb00 2024-03-19T03:31:45.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.722+0000 7f5374e00700 1 -- 172.21.15.117:0/2596632648 --> [v2:172.21.15.117:6826/3653875600,v1:172.21.15.117:6827/3653875600] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5370062e20 con 0x7f53540adb00 2024-03-19T03:31:45.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.723+0000 7f5367e00700 1 -- 172.21.15.117:0/2596632648 <== osd.9 v2:172.21.15.117:6826/3653875600 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5370062e20 con 0x7f53540adb00 2024-03-19T03:31:45.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.723+0000 7f5374e00700 1 -- 172.21.15.117:0/2596632648 >> [v2:172.21.15.117:6826/3653875600,v1:172.21.15.117:6827/3653875600] conn(0x7f53540adb00 msgr2=0x7f53540affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:45.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.723+0000 7f5374e00700 1 --2- 172.21.15.117:0/2596632648 >> [v2:172.21.15.117:6826/3653875600,v1:172.21.15.117:6827/3653875600] conn(0x7f53540adb00 0x7f53540affa0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:45.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.723+0000 7f5374e00700 1 -- 172.21.15.117:0/2596632648 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f53540a2a20 msgr2=0x7f53540a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:45.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.723+0000 7f5374e00700 1 --2- 172.21.15.117:0/2596632648 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f53540a2a20 0x7f53540a4ee0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5370108ae0 tx=0x7f535c009260 comp rx=0 tx=0).stop 2024-03-19T03:31:45.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.723+0000 7f5374e00700 1 -- 172.21.15.117:0/2596632648 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5370102450 msgr2=0x7f5370074610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:45.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.723+0000 7f5374e00700 1 --2- 172.21.15.117:0/2596632648 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5370102450 0x7f5370074610 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f536000eb30 tx=0x7f536000ee40 comp rx=0 tx=0).stop 2024-03-19T03:31:45.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.724+0000 7f536fe00700 1 -- 172.21.15.117:0/2596632648 reap_dead start 2024-03-19T03:31:45.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.724+0000 7f5374e00700 1 -- 172.21.15.117:0/2596632648 shutdown_connections 2024-03-19T03:31:45.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.724+0000 7f5374e00700 1 -- 172.21.15.117:0/2596632648 >> 172.21.15.117:0/2596632648 conn(0x7f53700fdce0 msgr2=0x7f53700fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:45.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.724+0000 7f5374e00700 1 -- 172.21.15.117:0/2596632648 shutdown_connections 2024-03-19T03:31:45.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:45.724+0000 7f5374e00700 1 -- 172.21.15.117:0/2596632648 wait complete. 2024-03-19T03:31:45.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:45 smithi119 ceph-mon[66861]: pgmap v354: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 10 op/s; 14 B/s, 0 objects/s recovering 2024-03-19T03:31:45.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:45 smithi178 ceph-mon[67578]: pgmap v354: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 10 op/s; 14 B/s, 0 objects/s recovering 2024-03-19T03:31:46.667 INFO:teuthology.orchestra.run.smithi117.stdout:34359738456 2024-03-19T03:31:46.668 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-19T03:31:47.671 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:47 smithi117 ceph-mon[31838]: pgmap v355: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 11 op/s; 14 B/s, 0 objects/s recovering 2024-03-19T03:31:47.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:47 smithi119 ceph-mon[66861]: pgmap v355: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 11 op/s; 14 B/s, 0 objects/s recovering 2024-03-19T03:31:47.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:47 smithi178 ceph-mon[67578]: pgmap v355: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 11 op/s; 14 B/s, 0 objects/s recovering 2024-03-19T03:31:47.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.619+0000 7fccff000700 1 -- 172.21.15.117:0/3494147342 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccf8102f40 msgr2=0x7fccf8106f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:47.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.619+0000 7fccff000700 1 --2- 172.21.15.117:0/3494147342 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccf8102f40 0x7fccf8106f90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcce40099c0 tx=0x7fcce4009cd0 comp rx=0 tx=0).stop 2024-03-19T03:31:47.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.619+0000 7fccff000700 1 -- 172.21.15.117:0/3494147342 shutdown_connections 2024-03-19T03:31:47.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.619+0000 7fccff000700 1 --2- 172.21.15.117:0/3494147342 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fccf8107670 0x7fccf810f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:47.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.619+0000 7fccff000700 1 --2- 172.21.15.117:0/3494147342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccf8102590 0x7fccf8102970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:47.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.619+0000 7fccff000700 1 --2- 172.21.15.117:0/3494147342 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccf8102f40 0x7fccf8106f90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:47.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.619+0000 7fccff000700 1 -- 172.21.15.117:0/3494147342 >> 172.21.15.117:0/3494147342 conn(0x7fccf80fdde0 msgr2=0x7fccf8100200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:47.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.619+0000 7fccff000700 1 -- 172.21.15.117:0/3494147342 shutdown_connections 2024-03-19T03:31:47.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.619+0000 7fccff000700 1 -- 172.21.15.117:0/3494147342 wait complete. 2024-03-19T03:31:47.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.621+0000 7fccff000700 1 Processor -- start 2024-03-19T03:31:47.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.621+0000 7fccff000700 1 -- start start 2024-03-19T03:31:47.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.621+0000 7fccff000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccf8102590 0x7fccf819e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:47.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.621+0000 7fccff000700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fccf8102f40 0x7fccf819eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:47.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.622+0000 7fccff000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccf8107670 0x7fccf8198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:47.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.622+0000 7fccff000700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fccf81126a0 con 0x7fccf8102590 2024-03-19T03:31:47.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.622+0000 7fccff000700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fccf8112520 con 0x7fccf8107670 2024-03-19T03:31:47.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.622+0000 7fccff000700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fccf8112820 con 0x7fccf8102f40 2024-03-19T03:31:47.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.622+0000 7fccfd000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccf8107670 0x7fccf8198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:47.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.622+0000 7fccfd000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccf8107670 0x7fccf8198920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57796/0 (socket says 172.21.15.117:57796) 2024-03-19T03:31:47.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.622+0000 7fccfd000700 1 -- 172.21.15.117:0/223309054 learned_addr learned my addr 172.21.15.117:0/223309054 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:47.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.622+0000 7fccf7400700 1 --2- 172.21.15.117:0/223309054 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fccf8102f40 0x7fccf819eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:47.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.622+0000 7fccfd000700 1 -- 172.21.15.117:0/223309054 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fccf8102f40 msgr2=0x7fccf819eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:47.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.622+0000 7fccfd000700 1 --2- 172.21.15.117:0/223309054 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fccf8102f40 0x7fccf819eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:47.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.622+0000 7fccfd000700 1 -- 172.21.15.117:0/223309054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccf8102590 msgr2=0x7fccf819e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:47.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.622+0000 7fccf7e00700 1 --2- 172.21.15.117:0/223309054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccf8102590 0x7fccf819e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:47.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.622+0000 7fccfd000700 1 --2- 172.21.15.117:0/223309054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccf8102590 0x7fccf819e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:47.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.622+0000 7fccfd000700 1 -- 172.21.15.117:0/223309054 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccf8199080 con 0x7fccf8107670 2024-03-19T03:31:47.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.623+0000 7fccf7e00700 1 --2- 172.21.15.117:0/223309054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccf8102590 0x7fccf819e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T03:31:47.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.623+0000 7fccfd000700 1 --2- 172.21.15.117:0/223309054 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccf8107670 0x7fccf8198920 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fccf000ef30 tx=0x7fccf000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:47.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.623+0000 7fccf4c00700 1 -- 172.21.15.117:0/223309054 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf00092e0 con 0x7fccf8107670 2024-03-19T03:31:47.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.623+0000 7fccf4c00700 1 -- 172.21.15.117:0/223309054 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccf0004bb0 con 0x7fccf8107670 2024-03-19T03:31:47.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.623+0000 7fccff000700 1 -- 172.21.15.117:0/223309054 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccf8199310 con 0x7fccf8107670 2024-03-19T03:31:47.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.623+0000 7fccf4c00700 1 -- 172.21.15.117:0/223309054 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf0004280 con 0x7fccf8107670 2024-03-19T03:31:47.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.623+0000 7fccff000700 1 -- 172.21.15.117:0/223309054 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccf81a5630 con 0x7fccf8107670 2024-03-19T03:31:47.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.624+0000 7fccff000700 1 -- 172.21.15.117:0/223309054 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fccf8062a20 con 0x7fccf8107670 2024-03-19T03:31:47.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.628+0000 7fccf4c00700 1 -- 172.21.15.117:0/223309054 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fccf001b030 con 0x7fccf8107670 2024-03-19T03:31:47.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.630+0000 7fccf4c00700 1 --2- 172.21.15.117:0/223309054 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fccdc0a2920 0x7fccdc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:47.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.631+0000 7fccf4c00700 1 -- 172.21.15.117:0/223309054 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fccf00cf380 con 0x7fccf8107670 2024-03-19T03:31:47.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.631+0000 7fccf7e00700 1 --2- 172.21.15.117:0/223309054 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fccdc0a2920 0x7fccdc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:47.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.631+0000 7fccf4c00700 1 --2- 172.21.15.117:0/223309054 >> [v2:172.21.15.117:6810/2178310135,v1:172.21.15.117:6811/2178310135] conn(0x7fccdc0ada00 0x7fccdc0afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:47.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.631+0000 7fccf4c00700 1 -- 172.21.15.117:0/223309054 --> [v2:172.21.15.117:6810/2178310135,v1:172.21.15.117:6811/2178310135] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fccdc0b0570 con 0x7fccdc0ada00 2024-03-19T03:31:47.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.631+0000 7fccf4c00700 1 -- 172.21.15.117:0/223309054 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fccf00cf7d0 con 0x7fccf8107670 2024-03-19T03:31:47.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.631+0000 7fccf7400700 1 --2- 172.21.15.117:0/223309054 >> [v2:172.21.15.117:6810/2178310135,v1:172.21.15.117:6811/2178310135] conn(0x7fccdc0ada00 0x7fccdc0afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:47.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.631+0000 7fccf7e00700 1 --2- 172.21.15.117:0/223309054 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fccdc0a2920 0x7fccdc0a4de0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fccf8199a60 tx=0x7fcce8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:47.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.632+0000 7fccf7400700 1 --2- 172.21.15.117:0/223309054 >> [v2:172.21.15.117:6810/2178310135,v1:172.21.15.117:6811/2178310135] conn(0x7fccdc0ada00 0x7fccdc0afea0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:47.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.633+0000 7fccf4c00700 1 -- 172.21.15.117:0/223309054 <== osd.3 v2:172.21.15.117:6810/2178310135 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fccdc0b0570 con 0x7fccdc0ada00 2024-03-19T03:31:47.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.652+0000 7fccff000700 1 -- 172.21.15.117:0/223309054 --> [v2:172.21.15.117:6810/2178310135,v1:172.21.15.117:6811/2178310135] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fccf8062ea0 con 0x7fccdc0ada00 2024-03-19T03:31:47.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.653+0000 7fccf4c00700 1 -- 172.21.15.117:0/223309054 <== osd.3 v2:172.21.15.117:6810/2178310135 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fccf8062ea0 con 0x7fccdc0ada00 2024-03-19T03:31:47.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.653+0000 7fccff000700 1 -- 172.21.15.117:0/223309054 >> [v2:172.21.15.117:6810/2178310135,v1:172.21.15.117:6811/2178310135] conn(0x7fccdc0ada00 msgr2=0x7fccdc0afea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:47.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.653+0000 7fccff000700 1 --2- 172.21.15.117:0/223309054 >> [v2:172.21.15.117:6810/2178310135,v1:172.21.15.117:6811/2178310135] conn(0x7fccdc0ada00 0x7fccdc0afea0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:47.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.654+0000 7fccff000700 1 -- 172.21.15.117:0/223309054 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fccdc0a2920 msgr2=0x7fccdc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:47.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.654+0000 7fccff000700 1 --2- 172.21.15.117:0/223309054 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fccdc0a2920 0x7fccdc0a4de0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fccf8199a60 tx=0x7fcce8009260 comp rx=0 tx=0).stop 2024-03-19T03:31:47.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.654+0000 7fccff000700 1 -- 172.21.15.117:0/223309054 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccf8107670 msgr2=0x7fccf8198920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:47.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.654+0000 7fccff000700 1 --2- 172.21.15.117:0/223309054 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccf8107670 0x7fccf8198920 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fccf000ef30 tx=0x7fccf000c3a0 comp rx=0 tx=0).stop 2024-03-19T03:31:47.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.654+0000 7fccfd000700 1 -- 172.21.15.117:0/223309054 reap_dead start 2024-03-19T03:31:47.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.654+0000 7fccff000700 1 -- 172.21.15.117:0/223309054 shutdown_connections 2024-03-19T03:31:47.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.654+0000 7fccff000700 1 -- 172.21.15.117:0/223309054 >> 172.21.15.117:0/223309054 conn(0x7fccf80fdde0 msgr2=0x7fccf80fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:47.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.654+0000 7fccff000700 1 -- 172.21.15.117:0/223309054 shutdown_connections 2024-03-19T03:31:47.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.654+0000 7fccff000700 1 -- 172.21.15.117:0/223309054 wait complete. 2024-03-19T03:31:49.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.840+0000 7f6035400700 1 -- 172.21.15.117:0/1033631819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f603006a960 msgr2=0x7f603010f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:49.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.840+0000 7f6035400700 1 --2- 172.21.15.117:0/1033631819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f603006a960 0x7f603010f1a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f602800b0a0 tx=0x7f602800b3b0 comp rx=0 tx=0).stop 2024-03-19T03:31:49.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.840+0000 7f6035400700 1 -- 172.21.15.117:0/1033631819 shutdown_connections 2024-03-19T03:31:49.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.840+0000 7f6035400700 1 --2- 172.21.15.117:0/1033631819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f603006a960 0x7f603010f1a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:49.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.840+0000 7f6035400700 1 --2- 172.21.15.117:0/1033631819 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6030108360 0x7f6030108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:49.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.840+0000 7f6035400700 1 --2- 172.21.15.117:0/1033631819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6030069fc0 0x7f603006a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:49.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.840+0000 7f6035400700 1 -- 172.21.15.117:0/1033631819 >> 172.21.15.117:0/1033631819 conn(0x7f6030078140 msgr2=0x7f6030078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:49.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.840+0000 7f6035400700 1 -- 172.21.15.117:0/1033631819 shutdown_connections 2024-03-19T03:31:49.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.840+0000 7f6035400700 1 -- 172.21.15.117:0/1033631819 wait complete. 2024-03-19T03:31:49.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.841+0000 7f6035400700 1 Processor -- start 2024-03-19T03:31:49.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.841+0000 7f6035400700 1 -- start start 2024-03-19T03:31:49.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.842+0000 7f6035400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6030069fc0 0x7f603019e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:49.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.842+0000 7f6035400700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f603006a960 0x7f603019ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:49.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.842+0000 7f6035400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6030108360 0x7f6030198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:49.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.842+0000 7f6035400700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6030112600 con 0x7f6030069fc0 2024-03-19T03:31:49.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.842+0000 7f6035400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6030112480 con 0x7f6030108360 2024-03-19T03:31:49.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.842+0000 7f6035400700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f6030112780 con 0x7f603006a960 2024-03-19T03:31:49.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.842+0000 7f602e800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6030069fc0 0x7f603019e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:49.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.842+0000 7f602de00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f603006a960 0x7f603019ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:49.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.842+0000 7f602f200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6030108360 0x7f6030198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:49.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.842+0000 7f602de00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f603006a960 0x7f603019ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:34328/0 (socket says 172.21.15.117:34328) 2024-03-19T03:31:49.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.842+0000 7f602de00700 1 -- 172.21.15.117:0/103205396 learned_addr learned my addr 172.21.15.117:0/103205396 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:49.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.843+0000 7f602de00700 1 -- 172.21.15.117:0/103205396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6030108360 msgr2=0x7f6030198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:49.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.843+0000 7f602de00700 1 --2- 172.21.15.117:0/103205396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6030108360 0x7f6030198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:49.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.843+0000 7f602de00700 1 -- 172.21.15.117:0/103205396 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6030069fc0 msgr2=0x7f603019e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:49.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.843+0000 7f602de00700 1 --2- 172.21.15.117:0/103205396 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6030069fc0 0x7f603019e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:49.364 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.843+0000 7f602de00700 1 -- 172.21.15.117:0/103205396 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6030199020 con 0x7f603006a960 2024-03-19T03:31:49.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.843+0000 7f602f200700 1 --2- 172.21.15.117:0/103205396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6030108360 0x7f6030198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:31:49.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.843+0000 7f602de00700 1 --2- 172.21.15.117:0/103205396 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f603006a960 0x7f603019ec00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f601c00e9a0 tx=0x7f601c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:49.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.843+0000 7f6027400700 1 -- 172.21.15.117:0/103205396 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601c00ca10 con 0x7f603006a960 2024-03-19T03:31:49.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.843+0000 7f6027400700 1 -- 172.21.15.117:0/103205396 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f601c00cbb0 con 0x7f603006a960 2024-03-19T03:31:49.365 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.843+0000 7f6027400700 1 -- 172.21.15.117:0/103205396 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601c018680 con 0x7f603006a960 2024-03-19T03:31:49.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.843+0000 7f6035400700 1 -- 172.21.15.117:0/103205396 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6030199310 con 0x7f603006a960 2024-03-19T03:31:49.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.843+0000 7f6035400700 1 -- 172.21.15.117:0/103205396 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60301a5520 con 0x7f603006a960 2024-03-19T03:31:49.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.844+0000 7f6035400700 1 -- 172.21.15.117:0/103205396 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6030062a60 con 0x7f603006a960 2024-03-19T03:31:49.366 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.847+0000 7f6027400700 1 -- 172.21.15.117:0/103205396 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f601c010430 con 0x7f603006a960 2024-03-19T03:31:49.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.848+0000 7f6027400700 1 --2- 172.21.15.117:0/103205396 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f60140a29a0 0x7f60140a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:49.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.848+0000 7f6027400700 1 -- 172.21.15.117:0/103205396 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f601c0cf9a0 con 0x7f603006a960 2024-03-19T03:31:49.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.848+0000 7f602e800700 1 --2- 172.21.15.117:0/103205396 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f60140a29a0 0x7f60140a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:49.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.848+0000 7f6027400700 1 --2- 172.21.15.117:0/103205396 >> [v2:172.21.15.178:6800/3733812989,v1:172.21.15.178:6801/3733812989] conn(0x7f60140ada80 0x7f60140aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:49.367 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.849+0000 7f6027400700 1 -- 172.21.15.117:0/103205396 --> [v2:172.21.15.178:6800/3733812989,v1:172.21.15.178:6801/3733812989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f601c01bec0 con 0x7f60140ada80 2024-03-19T03:31:49.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.849+0000 7f6027400700 1 -- 172.21.15.117:0/103205396 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f601c0cfd50 con 0x7f603006a960 2024-03-19T03:31:49.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.849+0000 7f602f200700 1 --2- 172.21.15.117:0/103205396 >> [v2:172.21.15.178:6800/3733812989,v1:172.21.15.178:6801/3733812989] conn(0x7f60140ada80 0x7f60140aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:49.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.849+0000 7f602e800700 1 --2- 172.21.15.117:0/103205396 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f60140a29a0 0x7f60140a4e60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6020007b60 tx=0x7f60200073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:49.368 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.850+0000 7f602f200700 1 --2- 172.21.15.117:0/103205396 >> [v2:172.21.15.178:6800/3733812989,v1:172.21.15.178:6801/3733812989] conn(0x7f60140ada80 0x7f60140aff20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:49.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.850+0000 7f6027400700 1 -- 172.21.15.117:0/103205396 <== osd.2 v2:172.21.15.178:6800/3733812989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f601c01bec0 con 0x7f60140ada80 2024-03-19T03:31:49.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.868+0000 7f6035400700 1 -- 172.21.15.117:0/103205396 --> [v2:172.21.15.178:6800/3733812989,v1:172.21.15.178:6801/3733812989] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6030062e90 con 0x7f60140ada80 2024-03-19T03:31:49.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.869+0000 7f6027400700 1 -- 172.21.15.117:0/103205396 <== osd.2 v2:172.21.15.178:6800/3733812989 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6030062e90 con 0x7f60140ada80 2024-03-19T03:31:49.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.870+0000 7f6035400700 1 -- 172.21.15.117:0/103205396 >> [v2:172.21.15.178:6800/3733812989,v1:172.21.15.178:6801/3733812989] conn(0x7f60140ada80 msgr2=0x7f60140aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:49.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.870+0000 7f6035400700 1 --2- 172.21.15.117:0/103205396 >> [v2:172.21.15.178:6800/3733812989,v1:172.21.15.178:6801/3733812989] conn(0x7f60140ada80 0x7f60140aff20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:49.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.870+0000 7f6035400700 1 -- 172.21.15.117:0/103205396 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f60140a29a0 msgr2=0x7f60140a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:49.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.870+0000 7f6035400700 1 --2- 172.21.15.117:0/103205396 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f60140a29a0 0x7f60140a4e60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6020007b60 tx=0x7f60200073d0 comp rx=0 tx=0).stop 2024-03-19T03:31:49.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.870+0000 7f6035400700 1 -- 172.21.15.117:0/103205396 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f603006a960 msgr2=0x7f603019ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:49.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.870+0000 7f6035400700 1 --2- 172.21.15.117:0/103205396 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f603006a960 0x7f603019ec00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f601c00e9a0 tx=0x7f601c00ecb0 comp rx=0 tx=0).stop 2024-03-19T03:31:49.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.870+0000 7f602f200700 1 -- 172.21.15.117:0/103205396 reap_dead start 2024-03-19T03:31:49.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.870+0000 7f6035400700 1 -- 172.21.15.117:0/103205396 shutdown_connections 2024-03-19T03:31:49.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.870+0000 7f6035400700 1 -- 172.21.15.117:0/103205396 >> 172.21.15.117:0/103205396 conn(0x7f6030078140 msgr2=0x7f60300ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:49.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.870+0000 7f6035400700 1 -- 172.21.15.117:0/103205396 shutdown_connections 2024-03-19T03:31:49.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:47.870+0000 7f6035400700 1 -- 172.21.15.117:0/103205396 wait complete. 2024-03-19T03:31:49.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:49 smithi117 ceph-mon[31838]: pgmap v356: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s; 11 B/s, 0 objects/s recovering 2024-03-19T03:31:49.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:49 smithi119 ceph-mon[66861]: pgmap v356: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s; 11 B/s, 0 objects/s recovering 2024-03-19T03:31:49.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:49 smithi178 ceph-mon[67578]: pgmap v356: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s; 11 B/s, 0 objects/s recovering 2024-03-19T03:31:50.211 INFO:teuthology.orchestra.run.smithi117.stdout:77309411408 2024-03-19T03:31:50.211 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-19T03:31:50.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.981+0000 7fb9c0800700 1 -- 172.21.15.117:0/4025042006 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9b81076c0 msgr2=0x7fb9b810f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:50.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.981+0000 7fb9c0800700 1 --2- 172.21.15.117:0/4025042006 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9b81076c0 0x7fb9b810f240 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb9b400b0a0 tx=0x7fb9b400b3b0 comp rx=0 tx=0).stop 2024-03-19T03:31:50.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.982+0000 7fb9c0800700 1 -- 172.21.15.117:0/4025042006 shutdown_connections 2024-03-19T03:31:50.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.982+0000 7fb9c0800700 1 --2- 172.21.15.117:0/4025042006 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9b81076c0 0x7fb9b810f240 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:50.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.982+0000 7fb9c0800700 1 --2- 172.21.15.117:0/4025042006 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb9b806ac50 0x7fb9b806b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:50.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.982+0000 7fb9c0800700 1 --2- 172.21.15.117:0/4025042006 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9b806b600 0x7fb9b8106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:50.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.982+0000 7fb9c0800700 1 -- 172.21.15.117:0/4025042006 >> 172.21.15.117:0/4025042006 conn(0x7fb9b8077260 msgr2=0x7fb9b8077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:50.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.982+0000 7fb9c0800700 1 -- 172.21.15.117:0/4025042006 shutdown_connections 2024-03-19T03:31:50.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.982+0000 7fb9c0800700 1 -- 172.21.15.117:0/4025042006 wait complete. 2024-03-19T03:31:50.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.983+0000 7fb9c0800700 1 Processor -- start 2024-03-19T03:31:50.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.983+0000 7fb9c0800700 1 -- start start 2024-03-19T03:31:50.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.983+0000 7fb9c0800700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb9b806ac50 0x7fb9b819e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:50.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.983+0000 7fb9c0800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9b806b600 0x7fb9b819ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:50.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.984+0000 7fb9c0800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9b81076c0 0x7fb9b8198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:50.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.984+0000 7fb9c0800700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b81126d0 con 0x7fb9b81076c0 2024-03-19T03:31:50.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.984+0000 7fb9c0800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b8112550 con 0x7fb9b806b600 2024-03-19T03:31:50.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.984+0000 7fb9c0800700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b8112850 con 0x7fb9b806ac50 2024-03-19T03:31:50.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.984+0000 7fb9bd400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9b806b600 0x7fb9b819ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:50.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.984+0000 7fb9bd400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9b806b600 0x7fb9b819ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57852/0 (socket says 172.21.15.117:57852) 2024-03-19T03:31:50.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.984+0000 7fb9be800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9b81076c0 0x7fb9b8198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:50.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.984+0000 7fb9bd400700 1 -- 172.21.15.117:0/3752363228 learned_addr learned my addr 172.21.15.117:0/3752363228 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:50.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.984+0000 7fb9bde00700 1 --2- 172.21.15.117:0/3752363228 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb9b806ac50 0x7fb9b819e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:50.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.984+0000 7fb9bd400700 1 -- 172.21.15.117:0/3752363228 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb9b806ac50 msgr2=0x7fb9b819e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:50.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.984+0000 7fb9bd400700 1 --2- 172.21.15.117:0/3752363228 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb9b806ac50 0x7fb9b819e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:50.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.984+0000 7fb9bd400700 1 -- 172.21.15.117:0/3752363228 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9b81076c0 msgr2=0x7fb9b8198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:50.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.984+0000 7fb9bd400700 1 --2- 172.21.15.117:0/3752363228 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9b81076c0 0x7fb9b8198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:50.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.984+0000 7fb9bd400700 1 -- 172.21.15.117:0/3752363228 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9b8199070 con 0x7fb9b806b600 2024-03-19T03:31:50.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.985+0000 7fb9be800700 1 --2- 172.21.15.117:0/3752363228 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9b81076c0 0x7fb9b8198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:31:50.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.985+0000 7fb9bd400700 1 --2- 172.21.15.117:0/3752363228 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9b806b600 0x7fb9b819ec40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb9a800e940 tx=0x7fb9a800ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:50.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.985+0000 7fb9b2a00700 1 -- 172.21.15.117:0/3752363228 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a800c9e0 con 0x7fb9b806b600 2024-03-19T03:31:50.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.985+0000 7fb9b2a00700 1 -- 172.21.15.117:0/3752363228 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9a800cb80 con 0x7fb9b806b600 2024-03-19T03:31:50.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.985+0000 7fb9b2a00700 1 -- 172.21.15.117:0/3752363228 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a8018680 con 0x7fb9b806b600 2024-03-19T03:31:50.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.985+0000 7fb9c0800700 1 -- 172.21.15.117:0/3752363228 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9b8199360 con 0x7fb9b806b600 2024-03-19T03:31:50.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.985+0000 7fb9c0800700 1 -- 172.21.15.117:0/3752363228 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9b81a55d0 con 0x7fb9b806b600 2024-03-19T03:31:50.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.986+0000 7fb9c0800700 1 -- 172.21.15.117:0/3752363228 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb9b8062a60 con 0x7fb9b806b600 2024-03-19T03:31:50.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.990+0000 7fb9b2a00700 1 -- 172.21.15.117:0/3752363228 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fb9a80188e0 con 0x7fb9b806b600 2024-03-19T03:31:50.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.991+0000 7fb9b2a00700 1 --2- 172.21.15.117:0/3752363228 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fb9a00a29a0 0x7fb9a00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:50.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.991+0000 7fb9b2a00700 1 -- 172.21.15.117:0/3752363228 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fb9a80cf4a0 con 0x7fb9b806b600 2024-03-19T03:31:50.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.991+0000 7fb9bde00700 1 --2- 172.21.15.117:0/3752363228 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fb9a00a29a0 0x7fb9a00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:50.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.991+0000 7fb9b2a00700 1 --2- 172.21.15.117:0/3752363228 >> [v2:172.21.15.119:6824/1050976194,v1:172.21.15.119:6825/1050976194] conn(0x7fb9a00ada80 0x7fb9a00aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:50.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.991+0000 7fb9b2a00700 1 -- 172.21.15.117:0/3752363228 --> [v2:172.21.15.119:6824/1050976194,v1:172.21.15.119:6825/1050976194] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb9a801bec0 con 0x7fb9a00ada80 2024-03-19T03:31:50.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.992+0000 7fb9b2a00700 1 -- 172.21.15.117:0/3752363228 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fb9a80cf850 con 0x7fb9b806b600 2024-03-19T03:31:50.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.992+0000 7fb9be800700 1 --2- 172.21.15.117:0/3752363228 >> [v2:172.21.15.119:6824/1050976194,v1:172.21.15.119:6825/1050976194] conn(0x7fb9a00ada80 0x7fb9a00aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:50.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.992+0000 7fb9bde00700 1 --2- 172.21.15.117:0/3752363228 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fb9a00a29a0 0x7fb9a00a4e60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb9b81072e0 tx=0x7fb9ac00b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:50.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.992+0000 7fb9be800700 1 --2- 172.21.15.117:0/3752363228 >> [v2:172.21.15.119:6824/1050976194,v1:172.21.15.119:6825/1050976194] conn(0x7fb9a00ada80 0x7fb9a00aff20 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:50.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:49.993+0000 7fb9b2a00700 1 -- 172.21.15.117:0/3752363228 <== osd.10 v2:172.21.15.119:6824/1050976194 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb9a801bec0 con 0x7fb9a00ada80 2024-03-19T03:31:50.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:50.014+0000 7fb9c0800700 1 -- 172.21.15.117:0/3752363228 --> [v2:172.21.15.119:6824/1050976194,v1:172.21.15.119:6825/1050976194] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb9b8062f30 con 0x7fb9a00ada80 2024-03-19T03:31:50.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:50.014+0000 7fb9b2a00700 1 -- 172.21.15.117:0/3752363228 <== osd.10 v2:172.21.15.119:6824/1050976194 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb9b8062f30 con 0x7fb9a00ada80 2024-03-19T03:31:50.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:50.015+0000 7fb9c0800700 1 -- 172.21.15.117:0/3752363228 >> [v2:172.21.15.119:6824/1050976194,v1:172.21.15.119:6825/1050976194] conn(0x7fb9a00ada80 msgr2=0x7fb9a00aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:50.437 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:50.015+0000 7fb9c0800700 1 --2- 172.21.15.117:0/3752363228 >> [v2:172.21.15.119:6824/1050976194,v1:172.21.15.119:6825/1050976194] conn(0x7fb9a00ada80 0x7fb9a00aff20 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:50.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:50.015+0000 7fb9c0800700 1 -- 172.21.15.117:0/3752363228 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fb9a00a29a0 msgr2=0x7fb9a00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:50.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:50.015+0000 7fb9c0800700 1 --2- 172.21.15.117:0/3752363228 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fb9a00a29a0 0x7fb9a00a4e60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb9b81072e0 tx=0x7fb9ac00b3c0 comp rx=0 tx=0).stop 2024-03-19T03:31:50.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:50.015+0000 7fb9c0800700 1 -- 172.21.15.117:0/3752363228 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9b806b600 msgr2=0x7fb9b819ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:50.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:50.015+0000 7fb9c0800700 1 --2- 172.21.15.117:0/3752363228 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9b806b600 0x7fb9b819ec40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb9a800e940 tx=0x7fb9a800ec50 comp rx=0 tx=0).stop 2024-03-19T03:31:50.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:50.015+0000 7fb9be800700 1 -- 172.21.15.117:0/3752363228 reap_dead start 2024-03-19T03:31:50.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:50.015+0000 7fb9c0800700 1 -- 172.21.15.117:0/3752363228 shutdown_connections 2024-03-19T03:31:50.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:50.015+0000 7fb9c0800700 1 -- 172.21.15.117:0/3752363228 >> 172.21.15.117:0/3752363228 conn(0x7fb9b8077260 msgr2=0x7fb9b80ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:50.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:50.015+0000 7fb9c0800700 1 -- 172.21.15.117:0/3752363228 shutdown_connections 2024-03-19T03:31:50.438 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:50.015+0000 7fb9c0800700 1 -- 172.21.15.117:0/3752363228 wait complete. 2024-03-19T03:31:51.242 INFO:teuthology.orchestra.run.smithi117.stdout:309237645347 2024-03-19T03:31:51.243 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-19T03:31:51.503 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:51 smithi117 ceph-mon[31838]: pgmap v357: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s; 10 B/s, 0 objects/s recovering 2024-03-19T03:31:51.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.693+0000 7f4c4b400700 1 -- 172.21.15.117:0/4042959666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c44102590 msgr2=0x7f4c44102970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:51.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.693+0000 7f4c4b400700 1 --2- 172.21.15.117:0/4042959666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c44102590 0x7f4c44102970 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4c38009990 tx=0x7f4c38009ca0 comp rx=0 tx=0).stop 2024-03-19T03:31:51.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.694+0000 7f4c4b400700 1 -- 172.21.15.117:0/4042959666 shutdown_connections 2024-03-19T03:31:51.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.694+0000 7f4c4b400700 1 --2- 172.21.15.117:0/4042959666 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4c44107670 0x7f4c4410f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:51.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.694+0000 7f4c4b400700 1 --2- 172.21.15.117:0/4042959666 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c44102590 0x7f4c44102970 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:51.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.694+0000 7f4c4b400700 1 --2- 172.21.15.117:0/4042959666 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c44102f40 0x7f4c44106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:51.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.694+0000 7f4c4b400700 1 -- 172.21.15.117:0/4042959666 >> 172.21.15.117:0/4042959666 conn(0x7f4c440fdde0 msgr2=0x7f4c44100200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:51.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.694+0000 7f4c4b400700 1 -- 172.21.15.117:0/4042959666 shutdown_connections 2024-03-19T03:31:51.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.694+0000 7f4c4b400700 1 -- 172.21.15.117:0/4042959666 wait complete. 2024-03-19T03:31:51.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.695+0000 7f4c4b400700 1 Processor -- start 2024-03-19T03:31:51.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.695+0000 7f4c4b400700 1 -- start start 2024-03-19T03:31:51.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.696+0000 7f4c4b400700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4c44102590 0x7f4c44074610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:51.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.696+0000 7f4c4b400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c44102f40 0x7f4c44074b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:51.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.696+0000 7f4c4b400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c44107670 0x7f4c44075090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:51.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.696+0000 7f4c4b400700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4c44112680 con 0x7f4c44107670 2024-03-19T03:31:51.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.696+0000 7f4c4b400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4c44112500 con 0x7f4c44102f40 2024-03-19T03:31:51.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.696+0000 7f4c4b400700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f4c44112800 con 0x7f4c44102590 2024-03-19T03:31:51.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.696+0000 7f4c48a00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4c44102590 0x7f4c44074610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:51.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.696+0000 7f4c49400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c44107670 0x7f4c44075090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:51.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.696+0000 7f4c43e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c44102f40 0x7f4c44074b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:51.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.696+0000 7f4c48a00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4c44102590 0x7f4c44074610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:34356/0 (socket says 172.21.15.117:34356) 2024-03-19T03:31:51.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.696+0000 7f4c49400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c44107670 0x7f4c44075090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:58294/0 (socket says 172.21.15.117:58294) 2024-03-19T03:31:51.698 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.697+0000 7f4c48a00700 1 -- 172.21.15.117:0/2157386446 learned_addr learned my addr 172.21.15.117:0/2157386446 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:51.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.697+0000 7f4c49400700 1 -- 172.21.15.117:0/2157386446 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4c44102590 msgr2=0x7f4c44074610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:51.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.697+0000 7f4c49400700 1 --2- 172.21.15.117:0/2157386446 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4c44102590 0x7f4c44074610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:51.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.697+0000 7f4c49400700 1 -- 172.21.15.117:0/2157386446 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c44102f40 msgr2=0x7f4c44074b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:51.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.697+0000 7f4c49400700 1 --2- 172.21.15.117:0/2157386446 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c44102f40 0x7f4c44074b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:51.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.697+0000 7f4c49400700 1 -- 172.21.15.117:0/2157386446 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c441a7040 con 0x7f4c44107670 2024-03-19T03:31:51.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.698+0000 7f4c49400700 1 --2- 172.21.15.117:0/2157386446 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c44107670 0x7f4c44075090 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c00efc0 tx=0x7f4c3c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:51.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.698+0000 7f4c41600700 1 -- 172.21.15.117:0/2157386446 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c0092e0 con 0x7f4c44107670 2024-03-19T03:31:51.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.698+0000 7f4c41600700 1 -- 172.21.15.117:0/2157386446 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c3c004bb0 con 0x7f4c44107670 2024-03-19T03:31:51.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.698+0000 7f4c4b400700 1 -- 172.21.15.117:0/2157386446 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c441a72d0 con 0x7f4c44107670 2024-03-19T03:31:51.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.698+0000 7f4c41600700 1 -- 172.21.15.117:0/2157386446 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c004280 con 0x7f4c44107670 2024-03-19T03:31:51.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.698+0000 7f4c4b400700 1 -- 172.21.15.117:0/2157386446 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c441a7810 con 0x7f4c44107670 2024-03-19T03:31:51.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.699+0000 7f4c4b400700 1 -- 172.21.15.117:0/2157386446 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4c440629e0 con 0x7f4c44107670 2024-03-19T03:31:51.704 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.703+0000 7f4c41600700 1 -- 172.21.15.117:0/2157386446 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f4c3c01b030 con 0x7f4c44107670 2024-03-19T03:31:51.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.706+0000 7f4c41600700 1 --2- 172.21.15.117:0/2157386446 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4c2c0a2a70 0x7f4c2c0a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:51.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.706+0000 7f4c41600700 1 -- 172.21.15.117:0/2157386446 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f4c3c0cf350 con 0x7f4c44107670 2024-03-19T03:31:51.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.706+0000 7f4c48a00700 1 --2- 172.21.15.117:0/2157386446 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4c2c0a2a70 0x7f4c2c0a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:51.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.706+0000 7f4c41600700 1 --2- 172.21.15.117:0/2157386446 >> [v2:172.21.15.178:6808/1259631912,v1:172.21.15.178:6809/1259631912] conn(0x7f4c2c0adb50 0x7f4c2c0afff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:51.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.706+0000 7f4c41600700 1 -- 172.21.15.117:0/2157386446 --> [v2:172.21.15.178:6808/1259631912,v1:172.21.15.178:6809/1259631912] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4c3c01fe50 con 0x7f4c2c0adb50 2024-03-19T03:31:51.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.706+0000 7f4c41600700 1 -- 172.21.15.117:0/2157386446 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f4c3c0cf740 con 0x7f4c44107670 2024-03-19T03:31:51.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.707+0000 7f4c43e00700 1 --2- 172.21.15.117:0/2157386446 >> [v2:172.21.15.178:6808/1259631912,v1:172.21.15.178:6809/1259631912] conn(0x7f4c2c0adb50 0x7f4c2c0afff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:51.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.707+0000 7f4c48a00700 1 --2- 172.21.15.117:0/2157386446 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4c2c0a2a70 0x7f4c2c0a4f30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4c38006010 tx=0x7f4c38009590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:51.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.708+0000 7f4c43e00700 1 --2- 172.21.15.117:0/2157386446 >> [v2:172.21.15.178:6808/1259631912,v1:172.21.15.178:6809/1259631912] conn(0x7f4c2c0adb50 0x7f4c2c0afff0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:51.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.709+0000 7f4c41600700 1 -- 172.21.15.117:0/2157386446 <== osd.5 v2:172.21.15.178:6808/1259631912 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4c3c01fe50 con 0x7f4c2c0adb50 2024-03-19T03:31:51.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:51 smithi119 ceph-mon[66861]: pgmap v357: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s; 10 B/s, 0 objects/s recovering 2024-03-19T03:31:51.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.734+0000 7f4c4b400700 1 -- 172.21.15.117:0/2157386446 --> [v2:172.21.15.178:6808/1259631912,v1:172.21.15.178:6809/1259631912] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4c44062de0 con 0x7f4c2c0adb50 2024-03-19T03:31:51.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.734+0000 7f4c41600700 1 -- 172.21.15.117:0/2157386446 <== osd.5 v2:172.21.15.178:6808/1259631912 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4c44062de0 con 0x7f4c2c0adb50 2024-03-19T03:31:51.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.735+0000 7f4c4b400700 1 -- 172.21.15.117:0/2157386446 >> [v2:172.21.15.178:6808/1259631912,v1:172.21.15.178:6809/1259631912] conn(0x7f4c2c0adb50 msgr2=0x7f4c2c0afff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:51.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.735+0000 7f4c4b400700 1 --2- 172.21.15.117:0/2157386446 >> [v2:172.21.15.178:6808/1259631912,v1:172.21.15.178:6809/1259631912] conn(0x7f4c2c0adb50 0x7f4c2c0afff0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:51.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.735+0000 7f4c4b400700 1 -- 172.21.15.117:0/2157386446 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4c2c0a2a70 msgr2=0x7f4c2c0a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:51.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.735+0000 7f4c4b400700 1 --2- 172.21.15.117:0/2157386446 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4c2c0a2a70 0x7f4c2c0a4f30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4c38006010 tx=0x7f4c38009590 comp rx=0 tx=0).stop 2024-03-19T03:31:51.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.735+0000 7f4c4b400700 1 -- 172.21.15.117:0/2157386446 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c44107670 msgr2=0x7f4c44075090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:51.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.735+0000 7f4c4b400700 1 --2- 172.21.15.117:0/2157386446 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c44107670 0x7f4c44075090 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c00efc0 tx=0x7f4c3c00c3a0 comp rx=0 tx=0).stop 2024-03-19T03:31:51.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.735+0000 7f4c49400700 1 -- 172.21.15.117:0/2157386446 reap_dead start 2024-03-19T03:31:51.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.735+0000 7f4c4b400700 1 -- 172.21.15.117:0/2157386446 shutdown_connections 2024-03-19T03:31:51.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.735+0000 7f4c4b400700 1 -- 172.21.15.117:0/2157386446 >> 172.21.15.117:0/2157386446 conn(0x7f4c440fdde0 msgr2=0x7f4c440ffb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:51.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.735+0000 7f4c4b400700 1 -- 172.21.15.117:0/2157386446 shutdown_connections 2024-03-19T03:31:51.737 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.735+0000 7f4c4b400700 1 -- 172.21.15.117:0/2157386446 wait complete. 2024-03-19T03:31:51.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:51 smithi178 ceph-mon[67578]: pgmap v357: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s; 10 B/s, 0 objects/s recovering 2024-03-19T03:31:52.246 INFO:teuthology.orchestra.run.smithi117.stdout:107374182474 2024-03-19T03:31:52.246 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-19T03:31:53.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.046+0000 7f9be9600700 1 -- 172.21.15.117:0/461037359 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9be4102e00 msgr2=0x7f9be4106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:53.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.046+0000 7f9be9600700 1 --2- 172.21.15.117:0/461037359 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9be4102e00 0x7f9be4106e50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0009990 tx=0x7f9bd0009ca0 comp rx=0 tx=0).stop 2024-03-19T03:31:53.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.047+0000 7f9be9600700 1 -- 172.21.15.117:0/461037359 shutdown_connections 2024-03-19T03:31:53.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.047+0000 7f9be9600700 1 --2- 172.21.15.117:0/461037359 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9be4107530 0x7f9be410f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:53.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.047+0000 7f9be9600700 1 --2- 172.21.15.117:0/461037359 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9be4102450 0x7f9be4102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:53.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.047+0000 7f9be9600700 1 --2- 172.21.15.117:0/461037359 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9be4102e00 0x7f9be4106e50 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:53.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.047+0000 7f9be9600700 1 -- 172.21.15.117:0/461037359 >> 172.21.15.117:0/461037359 conn(0x7f9be40fdce0 msgr2=0x7f9be4100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:53.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.047+0000 7f9be9600700 1 -- 172.21.15.117:0/461037359 shutdown_connections 2024-03-19T03:31:53.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.047+0000 7f9be9600700 1 -- 172.21.15.117:0/461037359 wait complete. 2024-03-19T03:31:53.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.048+0000 7f9be9600700 1 Processor -- start 2024-03-19T03:31:53.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.048+0000 7f9be9600700 1 -- start start 2024-03-19T03:31:53.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.049+0000 7f9be9600700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9be4102450 0x7f9be41a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:53.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.049+0000 7f9be9600700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9be4102e00 0x7f9be41a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:53.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.049+0000 7f9be9600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9be4107530 0x7f9be419aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:53.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.049+0000 7f9be9600700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9be4112560 con 0x7f9be4102e00 2024-03-19T03:31:53.167 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.049+0000 7f9be9600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9be41123e0 con 0x7f9be4107530 2024-03-19T03:31:53.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.049+0000 7f9be9600700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f9be41126e0 con 0x7f9be4102450 2024-03-19T03:31:53.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.049+0000 7f9be3e00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9be4102450 0x7f9be41a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:53.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.049+0000 7f9be3e00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9be4102450 0x7f9be41a08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:34388/0 (socket says 172.21.15.117:34388) 2024-03-19T03:31:53.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.049+0000 7f9be3e00700 1 -- 172.21.15.117:0/2938343932 learned_addr learned my addr 172.21.15.117:0/2938343932 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:53.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.049+0000 7f9be8c00700 1 --2- 172.21.15.117:0/2938343932 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9be4107530 0x7f9be419aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:53.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.049+0000 7f9be3e00700 1 -- 172.21.15.117:0/2938343932 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9be4107530 msgr2=0x7f9be419aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:53.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.049+0000 7f9be3e00700 1 --2- 172.21.15.117:0/2938343932 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9be4107530 0x7f9be419aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:53.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.049+0000 7f9be3e00700 1 -- 172.21.15.117:0/2938343932 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9be4102e00 msgr2=0x7f9be41a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:53.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.049+0000 7f9be3e00700 1 --2- 172.21.15.117:0/2938343932 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9be4102e00 0x7f9be41a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:53.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.049+0000 7f9be3e00700 1 -- 172.21.15.117:0/2938343932 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9be419b1c0 con 0x7f9be4102450 2024-03-19T03:31:53.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.050+0000 7f9be3e00700 1 --2- 172.21.15.117:0/2938343932 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9be4102450 0x7f9be41a08a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400e9a0 tx=0x7f9bd400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:53.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.050+0000 7f9be0c00700 1 -- 172.21.15.117:0/2938343932 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd400cbd0 con 0x7f9be4102450 2024-03-19T03:31:53.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.050+0000 7f9be0c00700 1 -- 172.21.15.117:0/2938343932 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bd400cd70 con 0x7f9be4102450 2024-03-19T03:31:53.168 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.050+0000 7f9be0c00700 1 -- 172.21.15.117:0/2938343932 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4018750 con 0x7f9be4102450 2024-03-19T03:31:53.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.050+0000 7f9be9600700 1 -- 172.21.15.117:0/2938343932 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9be419b4b0 con 0x7f9be4102450 2024-03-19T03:31:53.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.050+0000 7f9be9600700 1 -- 172.21.15.117:0/2938343932 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9be41a7770 con 0x7f9be4102450 2024-03-19T03:31:53.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.051+0000 7f9be9600700 1 -- 172.21.15.117:0/2938343932 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9be410e7e0 con 0x7f9be4102450 2024-03-19T03:31:53.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.053+0000 7f9be0c00700 1 -- 172.21.15.117:0/2938343932 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f9bd40188f0 con 0x7f9be4102450 2024-03-19T03:31:53.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.054+0000 7f9be0c00700 1 --2- 172.21.15.117:0/2938343932 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f9bc80a2920 0x7f9bc80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:53.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.054+0000 7f9be0c00700 1 -- 172.21.15.117:0/2938343932 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f9bd40cfde0 con 0x7f9be4102450 2024-03-19T03:31:53.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.054+0000 7f9be3400700 1 --2- 172.21.15.117:0/2938343932 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f9bc80a2920 0x7f9bc80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:53.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.055+0000 7f9be0c00700 1 --2- 172.21.15.117:0/2938343932 >> [v2:172.21.15.119:6808/3455398086,v1:172.21.15.119:6809/3455398086] conn(0x7f9bc80ada00 0x7f9bc80afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:53.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.055+0000 7f9be0c00700 1 -- 172.21.15.117:0/2938343932 --> [v2:172.21.15.119:6808/3455398086,v1:172.21.15.119:6809/3455398086] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9bd4011e50 con 0x7f9bc80ada00 2024-03-19T03:31:53.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.055+0000 7f9be0c00700 1 -- 172.21.15.117:0/2938343932 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f9bd40d0190 con 0x7f9be4102450 2024-03-19T03:31:53.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.055+0000 7f9be8c00700 1 --2- 172.21.15.117:0/2938343932 >> [v2:172.21.15.119:6808/3455398086,v1:172.21.15.119:6809/3455398086] conn(0x7f9bc80ada00 0x7f9bc80afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:53.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.055+0000 7f9be3400700 1 --2- 172.21.15.117:0/2938343932 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f9bc80a2920 0x7f9bc80a4de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9bd00098e0 tx=0x7f9bd0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:53.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.056+0000 7f9be8c00700 1 --2- 172.21.15.117:0/2938343932 >> [v2:172.21.15.119:6808/3455398086,v1:172.21.15.119:6809/3455398086] conn(0x7f9bc80ada00 0x7f9bc80afea0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:53.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.057+0000 7f9be0c00700 1 -- 172.21.15.117:0/2938343932 <== osd.4 v2:172.21.15.119:6808/3455398086 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9bd4011e50 con 0x7f9bc80ada00 2024-03-19T03:31:53.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.075+0000 7f9be9600700 1 -- 172.21.15.117:0/2938343932 --> [v2:172.21.15.119:6808/3455398086,v1:172.21.15.119:6809/3455398086] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9be4103930 con 0x7f9bc80ada00 2024-03-19T03:31:53.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.075+0000 7f9be0c00700 1 -- 172.21.15.117:0/2938343932 <== osd.4 v2:172.21.15.119:6808/3455398086 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9be4103930 con 0x7f9bc80ada00 2024-03-19T03:31:53.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.076+0000 7f9be9600700 1 -- 172.21.15.117:0/2938343932 >> [v2:172.21.15.119:6808/3455398086,v1:172.21.15.119:6809/3455398086] conn(0x7f9bc80ada00 msgr2=0x7f9bc80afea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:53.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.076+0000 7f9be9600700 1 --2- 172.21.15.117:0/2938343932 >> [v2:172.21.15.119:6808/3455398086,v1:172.21.15.119:6809/3455398086] conn(0x7f9bc80ada00 0x7f9bc80afea0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:53.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.076+0000 7f9be9600700 1 -- 172.21.15.117:0/2938343932 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f9bc80a2920 msgr2=0x7f9bc80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:53.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.076+0000 7f9be9600700 1 --2- 172.21.15.117:0/2938343932 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f9bc80a2920 0x7f9bc80a4de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9bd00098e0 tx=0x7f9bd0005fb0 comp rx=0 tx=0).stop 2024-03-19T03:31:53.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.076+0000 7f9be9600700 1 -- 172.21.15.117:0/2938343932 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9be4102450 msgr2=0x7f9be41a08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:53.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.076+0000 7f9be9600700 1 --2- 172.21.15.117:0/2938343932 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9be4102450 0x7f9be41a08a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400e9a0 tx=0x7f9bd400ecb0 comp rx=0 tx=0).stop 2024-03-19T03:31:53.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.076+0000 7f9be8c00700 1 -- 172.21.15.117:0/2938343932 reap_dead start 2024-03-19T03:31:53.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.076+0000 7f9be9600700 1 -- 172.21.15.117:0/2938343932 shutdown_connections 2024-03-19T03:31:53.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.076+0000 7f9be9600700 1 -- 172.21.15.117:0/2938343932 >> 172.21.15.117:0/2938343932 conn(0x7f9be40fdce0 msgr2=0x7f9be40fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:53.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.076+0000 7f9be9600700 1 -- 172.21.15.117:0/2938343932 shutdown_connections 2024-03-19T03:31:53.171 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.076+0000 7f9be9600700 1 -- 172.21.15.117:0/2938343932 wait complete. 2024-03-19T03:31:53.723 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:53 smithi117 ceph-mon[31838]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s; 9 B/s, 0 objects/s recovering 2024-03-19T03:31:53.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:53 smithi119 ceph-mon[66861]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s; 9 B/s, 0 objects/s recovering 2024-03-19T03:31:53.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:53 smithi178 ceph-mon[67578]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s; 9 B/s, 0 objects/s recovering 2024-03-19T03:31:54.105 INFO:teuthology.orchestra.run.smithi117.stdout:287762808872 2024-03-19T03:31:54.106 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-19T03:31:54.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.939+0000 7fc22f000700 1 -- 172.21.15.117:0/4187455698 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2281134e0 msgr2=0x7fc2281158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:54.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.939+0000 7fc22f000700 1 --2- 172.21.15.117:0/4187455698 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2281134e0 0x7fc2281158d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc22000b0a0 tx=0x7fc22000b3b0 comp rx=0 tx=0).stop 2024-03-19T03:31:54.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.939+0000 7fc22f000700 1 -- 172.21.15.117:0/4187455698 shutdown_connections 2024-03-19T03:31:54.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.939+0000 7fc22f000700 1 --2- 172.21.15.117:0/4187455698 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2281134e0 0x7fc2281158d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:54.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.939+0000 7fc22f000700 1 --2- 172.21.15.117:0/4187455698 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc228074540 0x7fc228074920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:54.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.939+0000 7fc22f000700 1 --2- 172.21.15.117:0/4187455698 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc228074e60 0x7fc228112fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:54.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.939+0000 7fc22f000700 1 -- 172.21.15.117:0/4187455698 >> 172.21.15.117:0/4187455698 conn(0x7fc228079470 msgr2=0x7fc228079880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:54.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.939+0000 7fc22f000700 1 -- 172.21.15.117:0/4187455698 shutdown_connections 2024-03-19T03:31:54.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.940+0000 7fc22f000700 1 -- 172.21.15.117:0/4187455698 wait complete. 2024-03-19T03:31:54.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.941+0000 7fc22f000700 1 Processor -- start 2024-03-19T03:31:54.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.941+0000 7fc22f000700 1 -- start start 2024-03-19T03:31:54.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.942+0000 7fc22f000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc228074540 0x7fc22819f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:54.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.942+0000 7fc22f000700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc228074e60 0x7fc22819f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:54.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.942+0000 7fc22f000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc2281134e0 0x7fc2281a3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:54.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.942+0000 7fc22f000700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc228118d10 con 0x7fc2281134e0 2024-03-19T03:31:54.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.942+0000 7fc22f000700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc228118b90 con 0x7fc228074540 2024-03-19T03:31:54.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.942+0000 7fc22f000700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fc228118e90 con 0x7fc228074e60 2024-03-19T03:31:54.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.942+0000 7fc227e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc228074540 0x7fc22819f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:54.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.942+0000 7fc227e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc228074540 0x7fc22819f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57914/0 (socket says 172.21.15.117:57914) 2024-03-19T03:31:54.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.942+0000 7fc227e00700 1 -- 172.21.15.117:0/3383318619 learned_addr learned my addr 172.21.15.117:0/3383318619 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:54.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.942+0000 7fc227400700 1 --2- 172.21.15.117:0/3383318619 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc228074e60 0x7fc22819f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:54.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.942+0000 7fc22d000700 1 --2- 172.21.15.117:0/3383318619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc2281134e0 0x7fc2281a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:54.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.942+0000 7fc227e00700 1 -- 172.21.15.117:0/3383318619 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc228074e60 msgr2=0x7fc22819f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:54.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.942+0000 7fc227e00700 1 --2- 172.21.15.117:0/3383318619 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc228074e60 0x7fc22819f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:54.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.942+0000 7fc227e00700 1 -- 172.21.15.117:0/3383318619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc2281134e0 msgr2=0x7fc2281a3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:54.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.942+0000 7fc227e00700 1 --2- 172.21.15.117:0/3383318619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc2281134e0 0x7fc2281a3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:54.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.942+0000 7fc227e00700 1 -- 172.21.15.117:0/3383318619 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2281a40b0 con 0x7fc228074540 2024-03-19T03:31:54.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.942+0000 7fc22d000700 1 --2- 172.21.15.117:0/3383318619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc2281134e0 0x7fc2281a3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:31:54.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.943+0000 7fc227e00700 1 --2- 172.21.15.117:0/3383318619 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc228074540 0x7fc22819f0a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc21800d980 tx=0x7fc21800dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:54.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.943+0000 7fc224c00700 1 -- 172.21.15.117:0/3383318619 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc218016070 con 0x7fc228074540 2024-03-19T03:31:54.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.943+0000 7fc224c00700 1 -- 172.21.15.117:0/3383318619 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc218004d10 con 0x7fc228074540 2024-03-19T03:31:54.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.943+0000 7fc22f000700 1 -- 172.21.15.117:0/3383318619 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2281a43a0 con 0x7fc228074540 2024-03-19T03:31:54.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.943+0000 7fc224c00700 1 -- 172.21.15.117:0/3383318619 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2180055f0 con 0x7fc228074540 2024-03-19T03:31:54.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.943+0000 7fc22f000700 1 -- 172.21.15.117:0/3383318619 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2281abc20 con 0x7fc228074540 2024-03-19T03:31:54.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.944+0000 7fc22f000700 1 -- 172.21.15.117:0/3383318619 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc228112c10 con 0x7fc228074540 2024-03-19T03:31:54.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.947+0000 7fc224c00700 1 -- 172.21.15.117:0/3383318619 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fc218004750 con 0x7fc228074540 2024-03-19T03:31:54.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.949+0000 7fc224c00700 1 --2- 172.21.15.117:0/3383318619 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc20c0a29a0 0x7fc20c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:54.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.949+0000 7fc224c00700 1 -- 172.21.15.117:0/3383318619 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fc2180cf4a0 con 0x7fc228074540 2024-03-19T03:31:54.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.949+0000 7fc227400700 1 --2- 172.21.15.117:0/3383318619 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc20c0a29a0 0x7fc20c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:54.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.949+0000 7fc224c00700 1 --2- 172.21.15.117:0/3383318619 >> [v2:172.21.15.178:6824/2915080368,v1:172.21.15.178:6825/2915080368] conn(0x7fc20c0ada80 0x7fc20c0aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:54.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.949+0000 7fc224c00700 1 -- 172.21.15.117:0/3383318619 --> [v2:172.21.15.178:6824/2915080368,v1:172.21.15.178:6825/2915080368] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc218015ec0 con 0x7fc20c0ada80 2024-03-19T03:31:54.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.949+0000 7fc224c00700 1 -- 172.21.15.117:0/3383318619 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fc2180cf850 con 0x7fc228074540 2024-03-19T03:31:54.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.950+0000 7fc22d000700 1 --2- 172.21.15.117:0/3383318619 >> [v2:172.21.15.178:6824/2915080368,v1:172.21.15.178:6825/2915080368] conn(0x7fc20c0ada80 0x7fc20c0aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:54.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.950+0000 7fc227400700 1 --2- 172.21.15.117:0/3383318619 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc20c0a29a0 0x7fc20c0a4e60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc22810fdb0 tx=0x7fc214009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:54.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.950+0000 7fc22d000700 1 --2- 172.21.15.117:0/3383318619 >> [v2:172.21.15.178:6824/2915080368,v1:172.21.15.178:6825/2915080368] conn(0x7fc20c0ada80 0x7fc20c0aff20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:54.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.951+0000 7fc224c00700 1 -- 172.21.15.117:0/3383318619 <== osd.11 v2:172.21.15.178:6824/2915080368 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc218015ec0 con 0x7fc20c0ada80 2024-03-19T03:31:54.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.969+0000 7fc22f000700 1 -- 172.21.15.117:0/3383318619 --> [v2:172.21.15.178:6824/2915080368,v1:172.21.15.178:6825/2915080368] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc228075c70 con 0x7fc20c0ada80 2024-03-19T03:31:54.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.969+0000 7fc224c00700 1 -- 172.21.15.117:0/3383318619 <== osd.11 v2:172.21.15.178:6824/2915080368 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc228075c70 con 0x7fc20c0ada80 2024-03-19T03:31:54.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.970+0000 7fc22f000700 1 -- 172.21.15.117:0/3383318619 >> [v2:172.21.15.178:6824/2915080368,v1:172.21.15.178:6825/2915080368] conn(0x7fc20c0ada80 msgr2=0x7fc20c0aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:54.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.970+0000 7fc22f000700 1 --2- 172.21.15.117:0/3383318619 >> [v2:172.21.15.178:6824/2915080368,v1:172.21.15.178:6825/2915080368] conn(0x7fc20c0ada80 0x7fc20c0aff20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:54.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.970+0000 7fc22f000700 1 -- 172.21.15.117:0/3383318619 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc20c0a29a0 msgr2=0x7fc20c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:54.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.970+0000 7fc22f000700 1 --2- 172.21.15.117:0/3383318619 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc20c0a29a0 0x7fc20c0a4e60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc22810fdb0 tx=0x7fc214009260 comp rx=0 tx=0).stop 2024-03-19T03:31:54.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.970+0000 7fc22f000700 1 -- 172.21.15.117:0/3383318619 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc228074540 msgr2=0x7fc22819f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:54.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.970+0000 7fc22f000700 1 --2- 172.21.15.117:0/3383318619 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc228074540 0x7fc22819f0a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc21800d980 tx=0x7fc21800dc90 comp rx=0 tx=0).stop 2024-03-19T03:31:54.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.970+0000 7fc22d000700 1 -- 172.21.15.117:0/3383318619 reap_dead start 2024-03-19T03:31:54.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.970+0000 7fc22f000700 1 -- 172.21.15.117:0/3383318619 shutdown_connections 2024-03-19T03:31:54.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.970+0000 7fc22f000700 1 -- 172.21.15.117:0/3383318619 >> 172.21.15.117:0/3383318619 conn(0x7fc228079470 msgr2=0x7fc228112230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:54.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.970+0000 7fc22f000700 1 -- 172.21.15.117:0/3383318619 shutdown_connections 2024-03-19T03:31:54.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:53.970+0000 7fc22f000700 1 -- 172.21.15.117:0/3383318619 wait complete. 2024-03-19T03:31:55.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.122+0000 7f67c0400700 1 -- 172.21.15.117:0/2837496586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67b8108360 msgr2=0x7f67b8108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:55.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.122+0000 7f67c0400700 1 --2- 172.21.15.117:0/2837496586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67b8108360 0x7f67b8108740 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009990 tx=0x7f67ac009ca0 comp rx=0 tx=0).stop 2024-03-19T03:31:55.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.122+0000 7f67c0400700 1 -- 172.21.15.117:0/2837496586 shutdown_connections 2024-03-19T03:31:55.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.122+0000 7f67c0400700 1 --2- 172.21.15.117:0/2837496586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67b806a960 0x7f67b810f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:55.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.122+0000 7f67c0400700 1 --2- 172.21.15.117:0/2837496586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67b8108360 0x7f67b8108740 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:55.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.122+0000 7f67c0400700 1 --2- 172.21.15.117:0/2837496586 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f67b8069fc0 0x7f67b806a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:55.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.122+0000 7f67c0400700 1 -- 172.21.15.117:0/2837496586 >> 172.21.15.117:0/2837496586 conn(0x7f67b8078140 msgr2=0x7f67b8078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:55.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.122+0000 7f67c0400700 1 -- 172.21.15.117:0/2837496586 shutdown_connections 2024-03-19T03:31:55.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.122+0000 7f67c0400700 1 -- 172.21.15.117:0/2837496586 wait complete. 2024-03-19T03:31:55.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.124+0000 7f67c0400700 1 Processor -- start 2024-03-19T03:31:55.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.124+0000 7f67c0400700 1 -- start start 2024-03-19T03:31:55.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.124+0000 7f67c0400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67b8069fc0 0x7f67b81a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:55.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.124+0000 7f67c0400700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f67b806a960 0x7f67b81a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:55.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.124+0000 7f67c0400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67b8108360 0x7f67b819aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:55.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.124+0000 7f67c0400700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f67b81125f0 con 0x7f67b8108360 2024-03-19T03:31:55.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.124+0000 7f67c0400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f67b8112470 con 0x7f67b8069fc0 2024-03-19T03:31:55.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.124+0000 7f67c0400700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f67b8112770 con 0x7f67b806a960 2024-03-19T03:31:55.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.125+0000 7f67bda00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67b8069fc0 0x7f67b81a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:55.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.125+0000 7f67bda00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67b8069fc0 0x7f67b81a08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57934/0 (socket says 172.21.15.117:57934) 2024-03-19T03:31:55.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.125+0000 7f67bda00700 1 -- 172.21.15.117:0/3871053065 learned_addr learned my addr 172.21.15.117:0/3871053065 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:55.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.125+0000 7f67be400700 1 --2- 172.21.15.117:0/3871053065 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67b8108360 0x7f67b819aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:55.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.125+0000 7f67bd000700 1 --2- 172.21.15.117:0/3871053065 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f67b806a960 0x7f67b81a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:55.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.125+0000 7f67bda00700 1 -- 172.21.15.117:0/3871053065 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f67b806a960 msgr2=0x7f67b81a0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:55.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.125+0000 7f67bda00700 1 --2- 172.21.15.117:0/3871053065 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f67b806a960 0x7f67b81a0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:55.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.125+0000 7f67bda00700 1 -- 172.21.15.117:0/3871053065 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67b8108360 msgr2=0x7f67b819aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:55.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.125+0000 7f67bda00700 1 --2- 172.21.15.117:0/3871053065 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67b8108360 0x7f67b819aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:55.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.125+0000 7f67bda00700 1 -- 172.21.15.117:0/3871053065 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67b819b1d0 con 0x7f67b8069fc0 2024-03-19T03:31:55.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.125+0000 7f67bda00700 1 --2- 172.21.15.117:0/3871053065 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67b8069fc0 0x7f67b81a08b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f67ac000c00 tx=0x7f67ac0057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:55.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.126+0000 7f67b6000700 1 -- 172.21.15.117:0/3871053065 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac027070 con 0x7f67b8069fc0 2024-03-19T03:31:55.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.126+0000 7f67b6000700 1 -- 172.21.15.117:0/3871053065 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67ac00be70 con 0x7f67b8069fc0 2024-03-19T03:31:55.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.126+0000 7f67c0400700 1 -- 172.21.15.117:0/3871053065 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67b819b460 con 0x7f67b8069fc0 2024-03-19T03:31:55.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.126+0000 7f67b6000700 1 -- 172.21.15.117:0/3871053065 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac017650 con 0x7f67b8069fc0 2024-03-19T03:31:55.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.126+0000 7f67c0400700 1 -- 172.21.15.117:0/3871053065 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67b81a7780 con 0x7f67b8069fc0 2024-03-19T03:31:55.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.127+0000 7f67c0400700 1 -- 172.21.15.117:0/3871053065 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f67b810df50 con 0x7f67b8069fc0 2024-03-19T03:31:55.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.129+0000 7f67b6000700 1 -- 172.21.15.117:0/3871053065 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f67ac02a070 con 0x7f67b8069fc0 2024-03-19T03:31:55.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.131+0000 7f67b6000700 1 --2- 172.21.15.117:0/3871053065 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f67a00a29a0 0x7f67a00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:55.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.132+0000 7f67b6000700 1 -- 172.21.15.117:0/3871053065 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f67ac0d0340 con 0x7f67b8069fc0 2024-03-19T03:31:55.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.132+0000 7f67bd000700 1 --2- 172.21.15.117:0/3871053065 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f67a00a29a0 0x7f67a00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:55.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.132+0000 7f67b6000700 1 --2- 172.21.15.117:0/3871053065 >> [v2:172.21.15.178:6816/1059587501,v1:172.21.15.178:6817/1059587501] conn(0x7f67a00ada80 0x7f67a00aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:55.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.132+0000 7f67b6000700 1 -- 172.21.15.117:0/3871053065 --> [v2:172.21.15.178:6816/1059587501,v1:172.21.15.178:6817/1059587501] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f67ac001d30 con 0x7f67a00ada80 2024-03-19T03:31:55.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.132+0000 7f67b6000700 1 -- 172.21.15.117:0/3871053065 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f67ac0d06f0 con 0x7f67b8069fc0 2024-03-19T03:31:55.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.133+0000 7f67bd000700 1 --2- 172.21.15.117:0/3871053065 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f67a00a29a0 0x7f67a00a4e60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f67b8103280 tx=0x7f67a400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:55.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.133+0000 7f67be400700 1 --2- 172.21.15.117:0/3871053065 >> [v2:172.21.15.178:6816/1059587501,v1:172.21.15.178:6817/1059587501] conn(0x7f67a00ada80 0x7f67a00aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:55.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.133+0000 7f67be400700 1 --2- 172.21.15.117:0/3871053065 >> [v2:172.21.15.178:6816/1059587501,v1:172.21.15.178:6817/1059587501] conn(0x7f67a00ada80 0x7f67a00aff20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:55.135 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.134+0000 7f67b6000700 1 -- 172.21.15.117:0/3871053065 <== osd.8 v2:172.21.15.178:6816/1059587501 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f67ac001d30 con 0x7f67a00ada80 2024-03-19T03:31:55.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.152+0000 7f67c0400700 1 -- 172.21.15.117:0/3871053065 --> [v2:172.21.15.178:6816/1059587501,v1:172.21.15.178:6817/1059587501] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f67b8103a10 con 0x7f67a00ada80 2024-03-19T03:31:55.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.152+0000 7f67b6000700 1 -- 172.21.15.117:0/3871053065 <== osd.8 v2:172.21.15.178:6816/1059587501 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f67b8103a10 con 0x7f67a00ada80 2024-03-19T03:31:55.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.153+0000 7f67c0400700 1 -- 172.21.15.117:0/3871053065 >> [v2:172.21.15.178:6816/1059587501,v1:172.21.15.178:6817/1059587501] conn(0x7f67a00ada80 msgr2=0x7f67a00aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:55.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.153+0000 7f67c0400700 1 --2- 172.21.15.117:0/3871053065 >> [v2:172.21.15.178:6816/1059587501,v1:172.21.15.178:6817/1059587501] conn(0x7f67a00ada80 0x7f67a00aff20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:55.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.153+0000 7f67c0400700 1 -- 172.21.15.117:0/3871053065 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f67a00a29a0 msgr2=0x7f67a00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:55.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.153+0000 7f67c0400700 1 --2- 172.21.15.117:0/3871053065 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f67a00a29a0 0x7f67a00a4e60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f67b8103280 tx=0x7f67a400b410 comp rx=0 tx=0).stop 2024-03-19T03:31:55.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.153+0000 7f67c0400700 1 -- 172.21.15.117:0/3871053065 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67b8069fc0 msgr2=0x7f67b81a08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:55.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.153+0000 7f67c0400700 1 --2- 172.21.15.117:0/3871053065 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67b8069fc0 0x7f67b81a08b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f67ac000c00 tx=0x7f67ac0057c0 comp rx=0 tx=0).stop 2024-03-19T03:31:55.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.153+0000 7f67be400700 1 -- 172.21.15.117:0/3871053065 reap_dead start 2024-03-19T03:31:55.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.153+0000 7f67c0400700 1 -- 172.21.15.117:0/3871053065 shutdown_connections 2024-03-19T03:31:55.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.153+0000 7f67c0400700 1 -- 172.21.15.117:0/3871053065 >> 172.21.15.117:0/3871053065 conn(0x7f67b8078140 msgr2=0x7f67b810d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:55.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.153+0000 7f67c0400700 1 -- 172.21.15.117:0/3871053065 shutdown_connections 2024-03-19T03:31:55.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:55.154+0000 7f67c0400700 1 -- 172.21.15.117:0/3871053065 wait complete. 2024-03-19T03:31:55.683 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:55 smithi117 ceph-mon[31838]: pgmap v359: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-03-19T03:31:55.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:55 smithi119 ceph-mon[66861]: pgmap v359: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-03-19T03:31:55.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.980+0000 7eff6ee00700 1 -- 172.21.15.117:0/84594588 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff68107630 msgr2=0x7eff6810f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:55.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.980+0000 7eff6ee00700 1 --2- 172.21.15.117:0/84594588 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff68107630 0x7eff6810f200 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7eff6000b0a0 tx=0x7eff6000b3b0 comp rx=0 tx=0).stop 2024-03-19T03:31:55.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.980+0000 7eff6ee00700 1 -- 172.21.15.117:0/84594588 shutdown_connections 2024-03-19T03:31:55.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.980+0000 7eff6ee00700 1 --2- 172.21.15.117:0/84594588 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff68107630 0x7eff6810f200 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:55.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.980+0000 7eff6ee00700 1 --2- 172.21.15.117:0/84594588 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff68102550 0x7eff68102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:55.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.980+0000 7eff6ee00700 1 --2- 172.21.15.117:0/84594588 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7eff68102f00 0x7eff68106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:55.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.980+0000 7eff6ee00700 1 -- 172.21.15.117:0/84594588 >> 172.21.15.117:0/84594588 conn(0x7eff680fddc0 msgr2=0x7eff681001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:55.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.980+0000 7eff6ee00700 1 -- 172.21.15.117:0/84594588 shutdown_connections 2024-03-19T03:31:55.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.980+0000 7eff6ee00700 1 -- 172.21.15.117:0/84594588 wait complete. 2024-03-19T03:31:55.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.981+0000 7eff6ee00700 1 Processor -- start 2024-03-19T03:31:55.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.981+0000 7eff6ee00700 1 -- start start 2024-03-19T03:31:55.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.981+0000 7eff6ee00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff68102550 0x7eff68074610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:55.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.982+0000 7eff6ee00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7eff68102f00 0x7eff68074b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:55.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.982+0000 7eff6ee00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff68107630 0x7eff68075090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:55.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.982+0000 7eff6ee00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7eff68112620 con 0x7eff68102550 2024-03-19T03:31:55.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.982+0000 7eff6ee00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7eff681124a0 con 0x7eff68107630 2024-03-19T03:31:55.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.982+0000 7eff6ee00700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7eff681127a0 con 0x7eff68102f00 2024-03-19T03:31:55.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.982+0000 7eff6ce00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff68107630 0x7eff68075090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:55.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.982+0000 7eff67e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff68102550 0x7eff68074610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:55.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.982+0000 7eff6ce00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff68107630 0x7eff68075090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57884/0 (socket says 172.21.15.117:57884) 2024-03-19T03:31:55.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.982+0000 7eff6ce00700 1 -- 172.21.15.117:0/785212086 learned_addr learned my addr 172.21.15.117:0/785212086 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:31:55.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.982+0000 7eff67400700 1 --2- 172.21.15.117:0/785212086 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7eff68102f00 0x7eff68074b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:55.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.982+0000 7eff6ce00700 1 -- 172.21.15.117:0/785212086 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7eff68102f00 msgr2=0x7eff68074b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:55.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.982+0000 7eff6ce00700 1 --2- 172.21.15.117:0/785212086 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7eff68102f00 0x7eff68074b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:55.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.982+0000 7eff6ce00700 1 -- 172.21.15.117:0/785212086 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff68102550 msgr2=0x7eff68074610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:55.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.982+0000 7eff6ce00700 1 --2- 172.21.15.117:0/785212086 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff68102550 0x7eff68074610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:55.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.982+0000 7eff6ce00700 1 -- 172.21.15.117:0/785212086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff681a6f90 con 0x7eff68107630 2024-03-19T03:31:55.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.983+0000 7eff6ce00700 1 --2- 172.21.15.117:0/785212086 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff68107630 0x7eff68075090 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7eff60006070 tx=0x7eff60004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:55.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.983+0000 7eff64c00700 1 -- 172.21.15.117:0/785212086 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff60021070 con 0x7eff68107630 2024-03-19T03:31:55.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.983+0000 7eff64c00700 1 -- 172.21.15.117:0/785212086 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff60003c00 con 0x7eff68107630 2024-03-19T03:31:55.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.983+0000 7eff64c00700 1 -- 172.21.15.117:0/785212086 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff60004790 con 0x7eff68107630 2024-03-19T03:31:55.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.983+0000 7eff6ee00700 1 -- 172.21.15.117:0/785212086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff681a7220 con 0x7eff68107630 2024-03-19T03:31:55.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.983+0000 7eff6ee00700 1 -- 172.21.15.117:0/785212086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff681a7680 con 0x7eff68107630 2024-03-19T03:31:55.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.985+0000 7eff6ee00700 1 -- 172.21.15.117:0/785212086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7eff68062a20 con 0x7eff68107630 2024-03-19T03:31:55.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.986+0000 7eff64c00700 1 -- 172.21.15.117:0/785212086 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7eff60007980 con 0x7eff68107630 2024-03-19T03:31:55.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.988+0000 7eff64c00700 1 --2- 172.21.15.117:0/785212086 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7eff4c0a2920 0x7eff4c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:55.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.988+0000 7eff64c00700 1 -- 172.21.15.117:0/785212086 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7eff600cfe80 con 0x7eff68107630 2024-03-19T03:31:55.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.988+0000 7eff67e00700 1 --2- 172.21.15.117:0/785212086 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7eff4c0a2920 0x7eff4c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:55.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.988+0000 7eff64c00700 1 --2- 172.21.15.117:0/785212086 >> [v2:172.21.15.119:6800/2425927341,v1:172.21.15.119:6801/2425927341] conn(0x7eff4c0ada00 0x7eff4c0afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:31:55.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.988+0000 7eff64c00700 1 -- 172.21.15.117:0/785212086 --> [v2:172.21.15.119:6800/2425927341,v1:172.21.15.119:6801/2425927341] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7eff600020b0 con 0x7eff4c0ada00 2024-03-19T03:31:55.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.988+0000 7eff64c00700 1 -- 172.21.15.117:0/785212086 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7eff600d24a0 con 0x7eff68107630 2024-03-19T03:31:55.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.988+0000 7eff67400700 1 --2- 172.21.15.117:0/785212086 >> [v2:172.21.15.119:6800/2425927341,v1:172.21.15.119:6801/2425927341] conn(0x7eff4c0ada00 0x7eff4c0afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:31:55.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.989+0000 7eff67e00700 1 --2- 172.21.15.117:0/785212086 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7eff4c0a2920 0x7eff4c0a4de0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7eff58009c60 tx=0x7eff58009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:55.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.989+0000 7eff67400700 1 --2- 172.21.15.117:0/785212086 >> [v2:172.21.15.119:6800/2425927341,v1:172.21.15.119:6801/2425927341] conn(0x7eff4c0ada00 0x7eff4c0afea0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:31:55.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:51.990+0000 7eff64c00700 1 -- 172.21.15.117:0/785212086 <== osd.1 v2:172.21.15.119:6800/2425927341 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7eff600020b0 con 0x7eff4c0ada00 2024-03-19T03:31:55.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:52.008+0000 7eff6ee00700 1 -- 172.21.15.117:0/785212086 --> [v2:172.21.15.119:6800/2425927341,v1:172.21.15.119:6801/2425927341] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7eff68062ea0 con 0x7eff4c0ada00 2024-03-19T03:31:55.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:52.008+0000 7eff64c00700 1 -- 172.21.15.117:0/785212086 <== osd.1 v2:172.21.15.119:6800/2425927341 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7eff68062ea0 con 0x7eff4c0ada00 2024-03-19T03:31:55.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:52.009+0000 7eff6ee00700 1 -- 172.21.15.117:0/785212086 >> [v2:172.21.15.119:6800/2425927341,v1:172.21.15.119:6801/2425927341] conn(0x7eff4c0ada00 msgr2=0x7eff4c0afea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:55.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:52.009+0000 7eff6ee00700 1 --2- 172.21.15.117:0/785212086 >> [v2:172.21.15.119:6800/2425927341,v1:172.21.15.119:6801/2425927341] conn(0x7eff4c0ada00 0x7eff4c0afea0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:31:55.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:52.009+0000 7eff6ee00700 1 -- 172.21.15.117:0/785212086 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7eff4c0a2920 msgr2=0x7eff4c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:55.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:52.009+0000 7eff6ee00700 1 --2- 172.21.15.117:0/785212086 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7eff4c0a2920 0x7eff4c0a4de0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7eff58009c60 tx=0x7eff58009260 comp rx=0 tx=0).stop 2024-03-19T03:31:55.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:52.009+0000 7eff6ee00700 1 -- 172.21.15.117:0/785212086 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff68107630 msgr2=0x7eff68075090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:31:55.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:52.009+0000 7eff6ee00700 1 --2- 172.21.15.117:0/785212086 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff68107630 0x7eff68075090 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7eff60006070 tx=0x7eff60004d00 comp rx=0 tx=0).stop 2024-03-19T03:31:55.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:52.009+0000 7eff6ce00700 1 -- 172.21.15.117:0/785212086 reap_dead start 2024-03-19T03:31:55.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:52.009+0000 7eff6ee00700 1 -- 172.21.15.117:0/785212086 shutdown_connections 2024-03-19T03:31:55.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:52.009+0000 7eff6ee00700 1 -- 172.21.15.117:0/785212086 >> 172.21.15.117:0/785212086 conn(0x7eff680fddc0 msgr2=0x7eff680fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:31:55.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:52.009+0000 7eff6ee00700 1 -- 172.21.15.117:0/785212086 shutdown_connections 2024-03-19T03:31:55.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:31:52.009+0000 7eff6ee00700 1 -- 172.21.15.117:0/785212086 wait complete. 2024-03-19T03:31:55.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:55 smithi178 ceph-mon[67578]: pgmap v359: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-03-19T03:31:56.691 INFO:teuthology.orchestra.run.smithi117.stdout:141733920839 2024-03-19T03:31:56.691 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-19T03:31:57.111 INFO:teuthology.orchestra.run.smithi117.stdout:253403070511 2024-03-19T03:31:57.112 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-19T03:31:57.262 INFO:teuthology.orchestra.run.smithi117.stdout:163208757312 2024-03-19T03:31:57.263 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-19T03:31:57.658 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:57 smithi117 ceph-mon[31838]: pgmap v360: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T03:31:57.717 INFO:teuthology.orchestra.run.smithi117.stdout:343597383712 2024-03-19T03:31:57.717 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-19T03:31:57.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:57 smithi119 ceph-mon[66861]: pgmap v360: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T03:31:57.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:57 smithi178 ceph-mon[67578]: pgmap v360: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T03:31:58.393 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:31:58.393 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:31:58.410 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:31:58.426 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:31:58.429 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:31:58.487 INFO:teuthology.orchestra.run.smithi117.stdout:55834574935 2024-03-19T03:31:58.487 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-19T03:31:58.663 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:31:58.664 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:31:58.677 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:31:59.732 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:31:59 smithi117 ceph-mon[31838]: pgmap v361: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:31:59.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:31:59 smithi119 ceph-mon[66861]: pgmap v361: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:31:59.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:31:59 smithi178 ceph-mon[67578]: pgmap v361: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:32:01.722 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:01 smithi117 ceph-mon[31838]: pgmap v362: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:32:01.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:01 smithi119 ceph-mon[66861]: pgmap v362: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:32:01.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:01 smithi178 ceph-mon[67578]: pgmap v362: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:32:02.270 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:32:02.271 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:32:03.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:03 smithi117 ceph-mon[31838]: pgmap v363: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-19T03:32:03.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:03 smithi119 ceph-mon[66861]: pgmap v363: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-19T03:32:03.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:03 smithi178 ceph-mon[67578]: pgmap v363: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-19T03:32:05.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:05 smithi119 ceph-mon[66861]: pgmap v364: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-19T03:32:05.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:05 smithi117 ceph-mon[31838]: pgmap v364: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-19T03:32:05.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:05 smithi178 ceph-mon[67578]: pgmap v364: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-19T03:32:06.825 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:32:07.573 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:07 smithi117 ceph-mon[31838]: pgmap v365: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:32:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:07 smithi119 ceph-mon[66861]: pgmap v365: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:32:07.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:07 smithi178 ceph-mon[67578]: pgmap v365: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:32:08.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.132+0000 7f1fcea00700 1 -- 172.21.15.117:0/2927451399 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fc810ed10 msgr2=0x7f1fc8111100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:08.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.132+0000 7f1fcea00700 1 --2- 172.21.15.117:0/2927451399 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fc810ed10 0x7f1fc8111100 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1fc000b0a0 tx=0x7f1fc000b3b0 comp rx=0 tx=0).stop 2024-03-19T03:32:08.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.133+0000 7f1fcea00700 1 -- 172.21.15.117:0/2927451399 shutdown_connections 2024-03-19T03:32:08.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.133+0000 7f1fcea00700 1 --2- 172.21.15.117:0/2927451399 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fc810ed10 0x7f1fc8111100 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:08.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.133+0000 7f1fcea00700 1 --2- 172.21.15.117:0/2927451399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1fc8074540 0x7f1fc8074920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:08.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.133+0000 7f1fcea00700 1 --2- 172.21.15.117:0/2927451399 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1fc8074e60 0x7f1fc810e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:08.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.133+0000 7f1fcea00700 1 -- 172.21.15.117:0/2927451399 >> 172.21.15.117:0/2927451399 conn(0x7f1fc8079440 msgr2=0x7f1fc8079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:08.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.133+0000 7f1fcea00700 1 -- 172.21.15.117:0/2927451399 shutdown_connections 2024-03-19T03:32:08.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.133+0000 7f1fcea00700 1 -- 172.21.15.117:0/2927451399 wait complete. 2024-03-19T03:32:08.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.134+0000 7f1fcea00700 1 Processor -- start 2024-03-19T03:32:08.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.134+0000 7f1fcea00700 1 -- start start 2024-03-19T03:32:08.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.135+0000 7f1fcea00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1fc8074540 0x7f1fc81a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:08.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.135+0000 7f1fcea00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1fc8074e60 0x7f1fc81a0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:08.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.135+0000 7f1fcea00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fc810ed10 0x7f1fc819aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:08.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.135+0000 7f1fcea00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc81147d0 con 0x7f1fc8074540 2024-03-19T03:32:08.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.135+0000 7f1fcea00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc8114650 con 0x7f1fc810ed10 2024-03-19T03:32:08.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.135+0000 7f1fcea00700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc8114950 con 0x7f1fc8074e60 2024-03-19T03:32:08.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.135+0000 7f1fc7400700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1fc8074e60 0x7f1fc81a0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:08.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.135+0000 7f1fc7400700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1fc8074e60 0x7f1fc81a0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:52864/0 (socket says 172.21.15.117:52864) 2024-03-19T03:32:08.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.135+0000 7f1fc7400700 1 -- 172.21.15.117:0/2759258386 learned_addr learned my addr 172.21.15.117:0/2759258386 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:08.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.135+0000 7f1fcca00700 1 --2- 172.21.15.117:0/2759258386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fc810ed10 0x7f1fc819aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:08.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.135+0000 7f1fc7e00700 1 --2- 172.21.15.117:0/2759258386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1fc8074540 0x7f1fc81a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:08.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.136+0000 7f1fcca00700 1 -- 172.21.15.117:0/2759258386 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1fc8074e60 msgr2=0x7f1fc81a0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:08.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.136+0000 7f1fcca00700 1 --2- 172.21.15.117:0/2759258386 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1fc8074e60 0x7f1fc81a0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:08.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.136+0000 7f1fcca00700 1 -- 172.21.15.117:0/2759258386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1fc8074540 msgr2=0x7f1fc81a08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:08.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.136+0000 7f1fcca00700 1 --2- 172.21.15.117:0/2759258386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1fc8074540 0x7f1fc81a08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:08.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.136+0000 7f1fcca00700 1 -- 172.21.15.117:0/2759258386 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fc819b210 con 0x7f1fc810ed10 2024-03-19T03:32:08.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.136+0000 7f1fcca00700 1 --2- 172.21.15.117:0/2759258386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fc810ed10 0x7f1fc819aab0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1fc000b0a0 tx=0x7f1fc0004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:08.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.136+0000 7f1fc4c00700 1 -- 172.21.15.117:0/2759258386 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc0021070 con 0x7f1fc810ed10 2024-03-19T03:32:08.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.136+0000 7f1fc4c00700 1 -- 172.21.15.117:0/2759258386 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fc0007b80 con 0x7f1fc810ed10 2024-03-19T03:32:08.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.136+0000 7f1fc4c00700 1 -- 172.21.15.117:0/2759258386 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc00052a0 con 0x7f1fc810ed10 2024-03-19T03:32:08.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.136+0000 7f1fcea00700 1 -- 172.21.15.117:0/2759258386 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fc819b4a0 con 0x7f1fc810ed10 2024-03-19T03:32:08.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.137+0000 7f1fcea00700 1 -- 172.21.15.117:0/2759258386 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fc81a77c0 con 0x7f1fc810ed10 2024-03-19T03:32:08.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.138+0000 7f1fcea00700 1 -- 172.21.15.117:0/2759258386 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fc8068fd0 con 0x7f1fc810ed10 2024-03-19T03:32:08.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.141+0000 7f1fc4c00700 1 -- 172.21.15.117:0/2759258386 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f1fc0024070 con 0x7f1fc810ed10 2024-03-19T03:32:08.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.143+0000 7f1fc4c00700 1 --2- 172.21.15.117:0/2759258386 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1fac0a29a0 0x7f1fac0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:08.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.143+0000 7f1fc4c00700 1 -- 172.21.15.117:0/2759258386 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f1fc00cfe30 con 0x7f1fc810ed10 2024-03-19T03:32:08.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.144+0000 7f1fc7e00700 1 --2- 172.21.15.117:0/2759258386 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1fac0a29a0 0x7f1fac0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:08.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.144+0000 7f1fc7e00700 1 --2- 172.21.15.117:0/2759258386 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1fac0a29a0 0x7f1fac0a4e60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1fc819bbf0 tx=0x7f1fb8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:08.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.151+0000 7f1fc4c00700 1 -- 172.21.15.117:0/2759258386 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1fc0007d30 con 0x7f1fc810ed10 2024-03-19T03:32:08.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.544+0000 7f1fcea00700 1 -- 172.21.15.117:0/2759258386 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f1fc802dbb0 con 0x7f1fc810ed10 2024-03-19T03:32:08.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.544+0000 7f1fc4c00700 1 -- 172.21.15.117:0/2759258386 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f1fc009ca60 con 0x7f1fc810ed10 2024-03-19T03:32:08.545 INFO:teuthology.orchestra.run.smithi117.stdout:309237645350 2024-03-19T03:32:08.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.548+0000 7f1fcea00700 1 -- 172.21.15.117:0/2759258386 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1fac0a29a0 msgr2=0x7f1fac0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:08.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.548+0000 7f1fcea00700 1 --2- 172.21.15.117:0/2759258386 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1fac0a29a0 0x7f1fac0a4e60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1fc819bbf0 tx=0x7f1fb8009260 comp rx=0 tx=0).stop 2024-03-19T03:32:08.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.548+0000 7f1fcea00700 1 -- 172.21.15.117:0/2759258386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fc810ed10 msgr2=0x7f1fc819aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:08.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.548+0000 7f1fcea00700 1 --2- 172.21.15.117:0/2759258386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fc810ed10 0x7f1fc819aab0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1fc000b0a0 tx=0x7f1fc0004d00 comp rx=0 tx=0).stop 2024-03-19T03:32:08.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.548+0000 7f1fcea00700 1 -- 172.21.15.117:0/2759258386 shutdown_connections 2024-03-19T03:32:08.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.548+0000 7f1fcea00700 1 --2- 172.21.15.117:0/2759258386 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1fac0a29a0 0x7f1fac0a4e60 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:08.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.548+0000 7f1fcea00700 1 --2- 172.21.15.117:0/2759258386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fc810ed10 0x7f1fc819aab0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:08.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.548+0000 7f1fcea00700 1 --2- 172.21.15.117:0/2759258386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1fc8074540 0x7f1fc81a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:08.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.548+0000 7f1fcea00700 1 --2- 172.21.15.117:0/2759258386 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1fc8074e60 0x7f1fc81a0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:08.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.548+0000 7f1fcea00700 1 -- 172.21.15.117:0/2759258386 >> 172.21.15.117:0/2759258386 conn(0x7f1fc8079440 msgr2=0x7f1fc80ff520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:08.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.548+0000 7f1fcea00700 1 -- 172.21.15.117:0/2759258386 shutdown_connections 2024-03-19T03:32:08.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:08.548+0000 7f1fcea00700 1 -- 172.21.15.117:0/2759258386 wait complete. 2024-03-19T03:32:09.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:09 smithi119 ceph-mon[66861]: pgmap v366: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:32:09.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:09 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2759258386' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-19T03:32:09.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:09 smithi117 ceph-mon[31838]: pgmap v366: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:32:09.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:09 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2759258386' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-19T03:32:09.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:09 smithi178 ceph-mon[67578]: pgmap v366: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:32:09.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:09 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2759258386' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-19T03:32:10.794 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:10 smithi117 ceph-mon[31838]: pgmap v367: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:32:10.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:10 smithi178 ceph-mon[67578]: pgmap v367: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:32:10.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:10 smithi119 ceph-mon[66861]: pgmap v367: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:32:10.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.531+0000 7f55f9a00700 1 -- 172.21.15.117:0/4157348679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55f4102d20 msgr2=0x7f55f410f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:10.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.531+0000 7f55f9a00700 1 --2- 172.21.15.117:0/4157348679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55f4102d20 0x7f55f410f160 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f55ec00b0d0 tx=0x7f55ec00b3e0 comp rx=0 tx=0).stop 2024-03-19T03:32:10.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.531+0000 7f55f9a00700 1 -- 172.21.15.117:0/4157348679 shutdown_connections 2024-03-19T03:32:10.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.531+0000 7f55f9a00700 1 --2- 172.21.15.117:0/4157348679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55f4102d20 0x7f55f410f160 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:10.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.531+0000 7f55f9a00700 1 --2- 172.21.15.117:0/4157348679 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55f4108340 0x7f55f4108720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:10.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.531+0000 7f55f9a00700 1 --2- 172.21.15.117:0/4157348679 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55f4102360 0x7f55f41027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:10.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.531+0000 7f55f9a00700 1 -- 172.21.15.117:0/4157348679 >> 172.21.15.117:0/4157348679 conn(0x7f55f4079440 msgr2=0x7f55f4079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:10.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.531+0000 7f55f9a00700 1 -- 172.21.15.117:0/4157348679 shutdown_connections 2024-03-19T03:32:10.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.531+0000 7f55f9a00700 1 -- 172.21.15.117:0/4157348679 wait complete. 2024-03-19T03:32:10.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.532+0000 7f55f9a00700 1 Processor -- start 2024-03-19T03:32:10.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.532+0000 7f55f9a00700 1 -- start start 2024-03-19T03:32:10.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.533+0000 7f55f9a00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55f4102360 0x7f55f41a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:10.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.533+0000 7f55f9a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55f4102d20 0x7f55f41a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:10.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.533+0000 7f55f9a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55f4108340 0x7f55f419aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:10.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.533+0000 7f55f9a00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f55f4112630 con 0x7f55f4108340 2024-03-19T03:32:10.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.533+0000 7f55f9a00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f55f41124b0 con 0x7f55f4102d20 2024-03-19T03:32:10.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.533+0000 7f55f9a00700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f55f41127b0 con 0x7f55f4102360 2024-03-19T03:32:10.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.533+0000 7f55ebe00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55f4102d20 0x7f55f41a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:10.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.533+0000 7f55ebe00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55f4102d20 0x7f55f41a0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:58006/0 (socket says 172.21.15.117:58006) 2024-03-19T03:32:10.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.533+0000 7f55ebe00700 1 -- 172.21.15.117:0/60233039 learned_addr learned my addr 172.21.15.117:0/60233039 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:10.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.533+0000 7f55f2a00700 1 --2- 172.21.15.117:0/60233039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55f4102360 0x7f55f41a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:10.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.533+0000 7f55f3400700 1 --2- 172.21.15.117:0/60233039 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55f4108340 0x7f55f419aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:10.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.533+0000 7f55ebe00700 1 -- 172.21.15.117:0/60233039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55f4102360 msgr2=0x7f55f41a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:10.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.533+0000 7f55ebe00700 1 --2- 172.21.15.117:0/60233039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55f4102360 0x7f55f41a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:10.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.533+0000 7f55ebe00700 1 -- 172.21.15.117:0/60233039 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55f4108340 msgr2=0x7f55f419aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:10.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.533+0000 7f55ebe00700 1 --2- 172.21.15.117:0/60233039 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55f4108340 0x7f55f419aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:10.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.533+0000 7f55ebe00700 1 -- 172.21.15.117:0/60233039 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55f419b1c0 con 0x7f55f4102d20 2024-03-19T03:32:10.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.534+0000 7f55ebe00700 1 --2- 172.21.15.117:0/60233039 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55f4102d20 0x7f55f41a0de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f55dc00e9a0 tx=0x7f55dc00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:10.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.534+0000 7f55eb400700 1 -- 172.21.15.117:0/60233039 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55dc00cbd0 con 0x7f55f4102d20 2024-03-19T03:32:10.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.534+0000 7f55eb400700 1 -- 172.21.15.117:0/60233039 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55dc00cd70 con 0x7f55f4102d20 2024-03-19T03:32:10.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.534+0000 7f55f9a00700 1 -- 172.21.15.117:0/60233039 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55f419b4b0 con 0x7f55f4102d20 2024-03-19T03:32:10.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.534+0000 7f55eb400700 1 -- 172.21.15.117:0/60233039 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55dc018750 con 0x7f55f4102d20 2024-03-19T03:32:10.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.534+0000 7f55f9a00700 1 -- 172.21.15.117:0/60233039 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55f41a7770 con 0x7f55f4102d20 2024-03-19T03:32:10.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.536+0000 7f55f9a00700 1 -- 172.21.15.117:0/60233039 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55f4068fd0 con 0x7f55f4102d20 2024-03-19T03:32:10.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.537+0000 7f55eb400700 1 -- 172.21.15.117:0/60233039 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f55dc010430 con 0x7f55f4102d20 2024-03-19T03:32:10.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.538+0000 7f55eb400700 1 --2- 172.21.15.117:0/60233039 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f55d80a29a0 0x7f55d80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:10.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.539+0000 7f55eb400700 1 -- 172.21.15.117:0/60233039 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f55dc0cf200 con 0x7f55f4102d20 2024-03-19T03:32:10.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.539+0000 7f55f2a00700 1 --2- 172.21.15.117:0/60233039 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f55d80a29a0 0x7f55d80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:10.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.540+0000 7f55f2a00700 1 --2- 172.21.15.117:0/60233039 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f55d80a29a0 0x7f55d80a4e60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f55f419bba0 tx=0x7f55e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:10.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.547+0000 7f55eb400700 1 -- 172.21.15.117:0/60233039 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f55dc09bda0 con 0x7f55f4102d20 2024-03-19T03:32:10.998 INFO:teuthology.orchestra.run.smithi117.stdout:197568495678 2024-03-19T03:32:10.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.841+0000 7f55f9a00700 1 -- 172.21.15.117:0/60233039 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f55f41a79a0 con 0x7f55f4102d20 2024-03-19T03:32:10.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.841+0000 7f55eb400700 1 -- 172.21.15.117:0/60233039 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f55dc09b6a0 con 0x7f55f4102d20 2024-03-19T03:32:10.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.845+0000 7f55f9a00700 1 -- 172.21.15.117:0/60233039 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f55d80a29a0 msgr2=0x7f55d80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:10.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.845+0000 7f55f9a00700 1 --2- 172.21.15.117:0/60233039 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f55d80a29a0 0x7f55d80a4e60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f55f419bba0 tx=0x7f55e4009260 comp rx=0 tx=0).stop 2024-03-19T03:32:10.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.845+0000 7f55f9a00700 1 -- 172.21.15.117:0/60233039 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55f4102d20 msgr2=0x7f55f41a0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:10.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.845+0000 7f55f9a00700 1 --2- 172.21.15.117:0/60233039 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55f4102d20 0x7f55f41a0de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f55dc00e9a0 tx=0x7f55dc00ecb0 comp rx=0 tx=0).stop 2024-03-19T03:32:10.998 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.845+0000 7f55f9a00700 1 -- 172.21.15.117:0/60233039 shutdown_connections 2024-03-19T03:32:10.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.845+0000 7f55f9a00700 1 --2- 172.21.15.117:0/60233039 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f55d80a29a0 0x7f55d80a4e60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:10.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.845+0000 7f55f9a00700 1 --2- 172.21.15.117:0/60233039 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55f4108340 0x7f55f419aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:10.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.845+0000 7f55f9a00700 1 --2- 172.21.15.117:0/60233039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55f4102360 0x7f55f41a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:10.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.845+0000 7f55f9a00700 1 --2- 172.21.15.117:0/60233039 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55f4102d20 0x7f55f41a0de0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:10.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.845+0000 7f55f9a00700 1 -- 172.21.15.117:0/60233039 >> 172.21.15.117:0/60233039 conn(0x7f55f4079440 msgr2=0x7f55f410d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:10.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.845+0000 7f55f9a00700 1 -- 172.21.15.117:0/60233039 shutdown_connections 2024-03-19T03:32:10.999 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:10.845+0000 7f55f9a00700 1 -- 172.21.15.117:0/60233039 wait complete. 2024-03-19T03:32:11.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:11 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/60233039' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-19T03:32:11.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:11 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/60233039' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-19T03:32:11.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:11 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/60233039' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-19T03:32:12.700 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:12 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3141883204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-19T03:32:12.701 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:12 smithi117 ceph-mon[31838]: pgmap v368: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-03-19T03:32:12.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:12 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3141883204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-19T03:32:12.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:12 smithi178 ceph-mon[67578]: pgmap v368: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-03-19T03:32:12.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:12 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3141883204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-19T03:32:12.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:12 smithi119 ceph-mon[66861]: pgmap v368: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-03-19T03:32:13.306 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645347 got 309237645350 for osd.10 2024-03-19T03:32:13.307 DEBUG:teuthology.parallel:result is None 2024-03-19T03:32:13.779 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:13 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/282989295' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-19T03:32:13.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.678+0000 7f838d200700 1 -- 172.21.15.117:0/2633029400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8388103cd0 msgr2=0x7f838810c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:13.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.678+0000 7f838d200700 1 --2- 172.21.15.117:0/2633029400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8388103cd0 0x7f838810c1c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8374009990 tx=0x7f8374009ca0 comp rx=0 tx=0).stop 2024-03-19T03:32:13.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.679+0000 7f838d200700 1 -- 172.21.15.117:0/2633029400 shutdown_connections 2024-03-19T03:32:13.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.679+0000 7f838d200700 1 --2- 172.21.15.117:0/2633029400 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f838810c7e0 0x7f838810eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:13.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.679+0000 7f838d200700 1 --2- 172.21.15.117:0/2633029400 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f83881033b0 0x7f8388103790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:13.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.679+0000 7f838d200700 1 --2- 172.21.15.117:0/2633029400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8388103cd0 0x7f838810c1c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:13.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.679+0000 7f838d200700 1 -- 172.21.15.117:0/2633029400 >> 172.21.15.117:0/2633029400 conn(0x7f83880fcbe0 msgr2=0x7f83880ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:13.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.679+0000 7f838d200700 1 -- 172.21.15.117:0/2633029400 shutdown_connections 2024-03-19T03:32:13.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.679+0000 7f838d200700 1 -- 172.21.15.117:0/2633029400 wait complete. 2024-03-19T03:32:13.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.680+0000 7f838d200700 1 Processor -- start 2024-03-19T03:32:13.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.680+0000 7f838d200700 1 -- start start 2024-03-19T03:32:13.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.681+0000 7f838d200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83881033b0 0x7f83881a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:13.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.681+0000 7f838d200700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8388103cd0 0x7f83881a0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:13.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.681+0000 7f838d200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f838810c7e0 0x7f838819aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:13.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.681+0000 7f838d200700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8388112570 con 0x7f83881033b0 2024-03-19T03:32:13.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.681+0000 7f838d200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f83881123f0 con 0x7f838810c7e0 2024-03-19T03:32:13.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.681+0000 7f838d200700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f83881126f0 con 0x7f8388103cd0 2024-03-19T03:32:13.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.681+0000 7f8387400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83881033b0 0x7f83881a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:13.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.681+0000 7f8387400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83881033b0 0x7f83881a08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:35804/0 (socket says 172.21.15.117:35804) 2024-03-19T03:32:13.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.681+0000 7f8386a00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8388103cd0 0x7f83881a0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:13.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.681+0000 7f8387400700 1 -- 172.21.15.117:0/2391381387 learned_addr learned my addr 172.21.15.117:0/2391381387 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:13.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.682+0000 7f8386a00700 1 -- 172.21.15.117:0/2391381387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f838810c7e0 msgr2=0x7f838819aab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T03:32:13.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.682+0000 7f8386a00700 1 --2- 172.21.15.117:0/2391381387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f838810c7e0 0x7f838819aab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:13.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.682+0000 7f8386a00700 1 -- 172.21.15.117:0/2391381387 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83881033b0 msgr2=0x7f83881a08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:13.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.682+0000 7f8386a00700 1 --2- 172.21.15.117:0/2391381387 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83881033b0 0x7f83881a08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:13.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.682+0000 7f8386a00700 1 -- 172.21.15.117:0/2391381387 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f838819b210 con 0x7f8388103cd0 2024-03-19T03:32:13.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.682+0000 7f8386a00700 1 --2- 172.21.15.117:0/2391381387 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8388103cd0 0x7f83881a0e30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8374000c00 tx=0x7f8374004790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:13.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.682+0000 7f837fe00700 1 -- 172.21.15.117:0/2391381387 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8374004800 con 0x7f8388103cd0 2024-03-19T03:32:13.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.682+0000 7f837fe00700 1 -- 172.21.15.117:0/2391381387 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8374005030 con 0x7f8388103cd0 2024-03-19T03:32:13.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.682+0000 7f838d200700 1 -- 172.21.15.117:0/2391381387 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f838819b4a0 con 0x7f8388103cd0 2024-03-19T03:32:13.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.682+0000 7f837fe00700 1 -- 172.21.15.117:0/2391381387 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8374003a00 con 0x7f8388103cd0 2024-03-19T03:32:13.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.682+0000 7f838d200700 1 -- 172.21.15.117:0/2391381387 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f838819b8f0 con 0x7f8388103cd0 2024-03-19T03:32:13.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.683+0000 7f838d200700 1 -- 172.21.15.117:0/2391381387 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8388068fd0 con 0x7f8388103cd0 2024-03-19T03:32:13.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.685+0000 7f837fe00700 1 -- 172.21.15.117:0/2391381387 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f8374028060 con 0x7f8388103cd0 2024-03-19T03:32:13.841 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.686+0000 7f837fe00700 1 --2- 172.21.15.117:0/2391381387 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f836c0a2920 0x7f836c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:13.841 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.687+0000 7f837fe00700 1 -- 172.21.15.117:0/2391381387 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f83740cf980 con 0x7f8388103cd0 2024-03-19T03:32:13.841 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.687+0000 7f8387400700 1 --2- 172.21.15.117:0/2391381387 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f836c0a2920 0x7f836c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:13.841 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.688+0000 7f8387400700 1 --2- 172.21.15.117:0/2391381387 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f836c0a2920 0x7f836c0a4de0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f838819bbf0 tx=0x7f8378009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:13.841 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.691+0000 7f837fe00700 1 -- 172.21.15.117:0/2391381387 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8374013070 con 0x7f8388103cd0 2024-03-19T03:32:13.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:13 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/282989295' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-19T03:32:13.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:13 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/282989295' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-19T03:32:14.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.063+0000 7f838d200700 1 -- 172.21.15.117:0/2391381387 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f83881a78b0 con 0x7f8388103cd0 2024-03-19T03:32:14.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.064+0000 7f837fe00700 1 -- 172.21.15.117:0/2391381387 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f837409c5b0 con 0x7f8388103cd0 2024-03-19T03:32:14.065 INFO:teuthology.orchestra.run.smithi117.stdout:77309411413 2024-03-19T03:32:14.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.067+0000 7f838d200700 1 -- 172.21.15.117:0/2391381387 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f836c0a2920 msgr2=0x7f836c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:14.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.067+0000 7f838d200700 1 --2- 172.21.15.117:0/2391381387 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f836c0a2920 0x7f836c0a4de0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f838819bbf0 tx=0x7f8378009260 comp rx=0 tx=0).stop 2024-03-19T03:32:14.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.067+0000 7f838d200700 1 -- 172.21.15.117:0/2391381387 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8388103cd0 msgr2=0x7f83881a0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:14.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.067+0000 7f838d200700 1 --2- 172.21.15.117:0/2391381387 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8388103cd0 0x7f83881a0e30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8374000c00 tx=0x7f8374004790 comp rx=0 tx=0).stop 2024-03-19T03:32:14.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.067+0000 7f838d200700 1 -- 172.21.15.117:0/2391381387 shutdown_connections 2024-03-19T03:32:14.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.067+0000 7f838d200700 1 --2- 172.21.15.117:0/2391381387 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f836c0a2920 0x7f836c0a4de0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:14.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.067+0000 7f838d200700 1 --2- 172.21.15.117:0/2391381387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f838810c7e0 0x7f838819aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:14.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.067+0000 7f838d200700 1 --2- 172.21.15.117:0/2391381387 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83881033b0 0x7f83881a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:14.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.067+0000 7f838d200700 1 --2- 172.21.15.117:0/2391381387 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8388103cd0 0x7f83881a0e30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:14.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.067+0000 7f838d200700 1 -- 172.21.15.117:0/2391381387 >> 172.21.15.117:0/2391381387 conn(0x7f83880fcbe0 msgr2=0x7f83880fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:14.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.068+0000 7f838d200700 1 -- 172.21.15.117:0/2391381387 shutdown_connections 2024-03-19T03:32:14.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.068+0000 7f838d200700 1 -- 172.21.15.117:0/2391381387 wait complete. 2024-03-19T03:32:14.141 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.140+0000 7f1d27c00700 1 -- 172.21.15.117:0/3234254689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d20113070 msgr2=0x7f1d20115460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:14.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.140+0000 7f1d27c00700 1 --2- 172.21.15.117:0/3234254689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d20113070 0x7f1d20115460 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1d18009990 tx=0x7f1d18009ca0 comp rx=0 tx=0).stop 2024-03-19T03:32:14.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.140+0000 7f1d27c00700 1 -- 172.21.15.117:0/3234254689 shutdown_connections 2024-03-19T03:32:14.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.140+0000 7f1d27c00700 1 --2- 172.21.15.117:0/3234254689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d20113070 0x7f1d20115460 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:14.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.140+0000 7f1d27c00700 1 --2- 172.21.15.117:0/3234254689 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1d201031c0 0x7f1d20069fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:14.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.140+0000 7f1d27c00700 1 --2- 172.21.15.117:0/3234254689 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d2006a590 0x7f1d2006a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:14.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.140+0000 7f1d27c00700 1 -- 172.21.15.117:0/3234254689 >> 172.21.15.117:0/3234254689 conn(0x7f1d20078140 msgr2=0x7f1d20078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:14.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.140+0000 7f1d27c00700 1 -- 172.21.15.117:0/3234254689 shutdown_connections 2024-03-19T03:32:14.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.140+0000 7f1d27c00700 1 -- 172.21.15.117:0/3234254689 wait complete. 2024-03-19T03:32:14.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.141+0000 7f1d27c00700 1 Processor -- start 2024-03-19T03:32:14.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.141+0000 7f1d27c00700 1 -- start start 2024-03-19T03:32:14.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.142+0000 7f1d27c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d2006a590 0x7f1d2019f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:14.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.142+0000 7f1d27c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d201031c0 0x7f1d2019f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:14.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.142+0000 7f1d27c00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1d20113070 0x7f1d201a39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:14.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.142+0000 7f1d27c00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1d20118c00 con 0x7f1d2006a590 2024-03-19T03:32:14.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.142+0000 7f1d27c00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f1d20118a80 con 0x7f1d201031c0 2024-03-19T03:32:14.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.142+0000 7f1d27c00700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f1d20118d80 con 0x7f1d20113070 2024-03-19T03:32:14.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.142+0000 7f1d17e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d201031c0 0x7f1d2019f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:14.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.142+0000 7f1d17e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d201031c0 0x7f1d2019f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:58096/0 (socket says 172.21.15.117:58096) 2024-03-19T03:32:14.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.142+0000 7f1d17e00700 1 -- 172.21.15.117:0/1737335864 learned_addr learned my addr 172.21.15.117:0/1737335864 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:14.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.142+0000 7f1d25200700 1 --2- 172.21.15.117:0/1737335864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d2006a590 0x7f1d2019f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:14.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.142+0000 7f1d25c00700 1 --2- 172.21.15.117:0/1737335864 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1d20113070 0x7f1d201a39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:14.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.143+0000 7f1d17e00700 1 -- 172.21.15.117:0/1737335864 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1d20113070 msgr2=0x7f1d201a39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:14.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.143+0000 7f1d17e00700 1 --2- 172.21.15.117:0/1737335864 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1d20113070 0x7f1d201a39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:14.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.143+0000 7f1d17e00700 1 -- 172.21.15.117:0/1737335864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d2006a590 msgr2=0x7f1d2019f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:14.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.143+0000 7f1d17e00700 1 --2- 172.21.15.117:0/1737335864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d2006a590 0x7f1d2019f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:14.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.143+0000 7f1d17e00700 1 -- 172.21.15.117:0/1737335864 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d201a4100 con 0x7f1d201031c0 2024-03-19T03:32:14.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.143+0000 7f1d25200700 1 --2- 172.21.15.117:0/1737335864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d2006a590 0x7f1d2019f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:32:14.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.143+0000 7f1d25c00700 1 --2- 172.21.15.117:0/1737335864 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1d20113070 0x7f1d201a39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:32:14.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.143+0000 7f1d17e00700 1 --2- 172.21.15.117:0/1737335864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d201031c0 0x7f1d2019f630 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1d0800d950 tx=0x7f1d0800dd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:14.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.144+0000 7f1d1e000700 1 -- 172.21.15.117:0/1737335864 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d08016070 con 0x7f1d201031c0 2024-03-19T03:32:14.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.144+0000 7f1d27c00700 1 -- 172.21.15.117:0/1737335864 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d201a43f0 con 0x7f1d201031c0 2024-03-19T03:32:14.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.144+0000 7f1d1e000700 1 -- 172.21.15.117:0/1737335864 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d08009b40 con 0x7f1d201031c0 2024-03-19T03:32:14.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.144+0000 7f1d27c00700 1 -- 172.21.15.117:0/1737335864 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d201abc70 con 0x7f1d201031c0 2024-03-19T03:32:14.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.145+0000 7f1d1e000700 1 -- 172.21.15.117:0/1737335864 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d08016070 con 0x7f1d201031c0 2024-03-19T03:32:14.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.146+0000 7f1d27c00700 1 -- 172.21.15.117:0/1737335864 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d20068fd0 con 0x7f1d201031c0 2024-03-19T03:32:14.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.151+0000 7f1d1e000700 1 -- 172.21.15.117:0/1737335864 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f1d08019070 con 0x7f1d201031c0 2024-03-19T03:32:14.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.154+0000 7f1d1e000700 1 --2- 172.21.15.117:0/1737335864 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1d040a2ba0 0x7f1d040a5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:14.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.154+0000 7f1d1e000700 1 -- 172.21.15.117:0/1737335864 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f1d080ce510 con 0x7f1d201031c0 2024-03-19T03:32:14.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.154+0000 7f1d25200700 1 --2- 172.21.15.117:0/1737335864 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1d040a2ba0 0x7f1d040a5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:14.155 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.154+0000 7f1d1e000700 1 -- 172.21.15.117:0/1737335864 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1d08020060 con 0x7f1d201031c0 2024-03-19T03:32:14.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.155+0000 7f1d25200700 1 --2- 172.21.15.117:0/1737335864 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1d040a2ba0 0x7f1d040a5060 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1d100093a0 tx=0x7f1d1000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:14.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.449+0000 7f1d27c00700 1 -- 172.21.15.117:0/1737335864 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f1d201abf60 con 0x7f1d201031c0 2024-03-19T03:32:14.451 INFO:teuthology.orchestra.run.smithi117.stdout:219043332152 2024-03-19T03:32:14.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.450+0000 7f1d1e000700 1 -- 172.21.15.117:0/1737335864 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1d0809b140 con 0x7f1d201031c0 2024-03-19T03:32:14.453 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.452+0000 7f1d27c00700 1 -- 172.21.15.117:0/1737335864 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1d040a2ba0 msgr2=0x7f1d040a5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:14.453 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.452+0000 7f1d27c00700 1 --2- 172.21.15.117:0/1737335864 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1d040a2ba0 0x7f1d040a5060 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1d100093a0 tx=0x7f1d1000b040 comp rx=0 tx=0).stop 2024-03-19T03:32:14.453 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.452+0000 7f1d27c00700 1 -- 172.21.15.117:0/1737335864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d201031c0 msgr2=0x7f1d2019f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:14.453 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.452+0000 7f1d27c00700 1 --2- 172.21.15.117:0/1737335864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d201031c0 0x7f1d2019f630 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1d0800d950 tx=0x7f1d0800dd10 comp rx=0 tx=0).stop 2024-03-19T03:32:14.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.452+0000 7f1d27c00700 1 -- 172.21.15.117:0/1737335864 shutdown_connections 2024-03-19T03:32:14.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.452+0000 7f1d27c00700 1 --2- 172.21.15.117:0/1737335864 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f1d040a2ba0 0x7f1d040a5060 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:14.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.452+0000 7f1d27c00700 1 --2- 172.21.15.117:0/1737335864 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1d20113070 0x7f1d201a39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:14.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.452+0000 7f1d27c00700 1 --2- 172.21.15.117:0/1737335864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d2006a590 0x7f1d2019f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:14.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.452+0000 7f1d27c00700 1 --2- 172.21.15.117:0/1737335864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d201031c0 0x7f1d2019f630 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:14.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.452+0000 7f1d27c00700 1 -- 172.21.15.117:0/1737335864 >> 172.21.15.117:0/1737335864 conn(0x7f1d20078140 msgr2=0x7f1d20100200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:14.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.453+0000 7f1d27c00700 1 -- 172.21.15.117:0/1737335864 shutdown_connections 2024-03-19T03:32:14.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.453+0000 7f1d27c00700 1 -- 172.21.15.117:0/1737335864 wait complete. 2024-03-19T03:32:14.720 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:14 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2391381387' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-19T03:32:14.720 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:14 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/195361703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-19T03:32:14.720 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:14 smithi117 ceph-mon[31838]: pgmap v369: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-19T03:32:14.720 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:14 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1737335864' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-19T03:32:14.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:14 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2391381387' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-19T03:32:14.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:14 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/195361703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-19T03:32:14.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:14 smithi178 ceph-mon[67578]: pgmap v369: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-19T03:32:14.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:14 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1737335864' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-19T03:32:14.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.699+0000 7f7468200700 1 -- 172.21.15.117:0/3928602260 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7460107690 msgr2=0x7f746010f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:14.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.699+0000 7f7468200700 1 --2- 172.21.15.117:0/3928602260 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7460107690 0x7f746010f260 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f745800b0a0 tx=0x7f745800b3b0 comp rx=0 tx=0).stop 2024-03-19T03:32:14.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.699+0000 7f7468200700 1 -- 172.21.15.117:0/3928602260 shutdown_connections 2024-03-19T03:32:14.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.699+0000 7f7468200700 1 --2- 172.21.15.117:0/3928602260 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7460107690 0x7f746010f260 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:14.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.699+0000 7f7468200700 1 --2- 172.21.15.117:0/3928602260 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74601025c0 0x7f74601029a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:14.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.699+0000 7f7468200700 1 --2- 172.21.15.117:0/3928602260 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7460102f70 0x7f7460106fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:14.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.699+0000 7f7468200700 1 -- 172.21.15.117:0/3928602260 >> 172.21.15.117:0/3928602260 conn(0x7f7460079470 msgr2=0x7f7460079880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:14.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.699+0000 7f7468200700 1 -- 172.21.15.117:0/3928602260 shutdown_connections 2024-03-19T03:32:14.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.699+0000 7f7468200700 1 -- 172.21.15.117:0/3928602260 wait complete. 2024-03-19T03:32:14.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.701+0000 7f7468200700 1 Processor -- start 2024-03-19T03:32:14.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.701+0000 7f7468200700 1 -- start start 2024-03-19T03:32:14.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.701+0000 7f7468200700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f74601025c0 0x7f7460074610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:14.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.701+0000 7f7468200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7460102f70 0x7f7460074b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:14.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.701+0000 7f7468200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7460107690 0x7f7460075090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:14.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.701+0000 7f7468200700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7460112700 con 0x7f7460107690 2024-03-19T03:32:14.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.701+0000 7f7468200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7460112580 con 0x7f7460102f70 2024-03-19T03:32:14.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.701+0000 7f7468200700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f7460112880 con 0x7f74601025c0 2024-03-19T03:32:14.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.702+0000 7f7464e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7460102f70 0x7f7460074b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:14.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.702+0000 7f7464e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7460102f70 0x7f7460074b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:58052/0 (socket says 172.21.15.117:58052) 2024-03-19T03:32:14.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.702+0000 7f7464e00700 1 -- 172.21.15.117:0/282989295 learned_addr learned my addr 172.21.15.117:0/282989295 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:14.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.702+0000 7f7465800700 1 --2- 172.21.15.117:0/282989295 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f74601025c0 0x7f7460074610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:14.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.702+0000 7f7466200700 1 --2- 172.21.15.117:0/282989295 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7460107690 0x7f7460075090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:14.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.702+0000 7f7464e00700 1 -- 172.21.15.117:0/282989295 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f74601025c0 msgr2=0x7f7460074610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:14.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.702+0000 7f7464e00700 1 --2- 172.21.15.117:0/282989295 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f74601025c0 0x7f7460074610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:14.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.702+0000 7f7464e00700 1 -- 172.21.15.117:0/282989295 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7460107690 msgr2=0x7f7460075090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:14.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.702+0000 7f7464e00700 1 --2- 172.21.15.117:0/282989295 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7460107690 0x7f7460075090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:14.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.702+0000 7f7464e00700 1 -- 172.21.15.117:0/282989295 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74601a70b0 con 0x7f7460102f70 2024-03-19T03:32:14.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.702+0000 7f7465800700 1 --2- 172.21.15.117:0/282989295 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f74601025c0 0x7f7460074610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:32:14.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.702+0000 7f7464e00700 1 --2- 172.21.15.117:0/282989295 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7460102f70 0x7f7460074b50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f744c00e9f0 tx=0x7f744c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:14.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.703+0000 7f745e000700 1 -- 172.21.15.117:0/282989295 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744c00cbd0 con 0x7f7460102f70 2024-03-19T03:32:14.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.703+0000 7f745e000700 1 -- 172.21.15.117:0/282989295 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f744c00cd70 con 0x7f7460102f70 2024-03-19T03:32:14.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.703+0000 7f745e000700 1 -- 172.21.15.117:0/282989295 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744c010430 con 0x7f7460102f70 2024-03-19T03:32:14.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.703+0000 7f7468200700 1 -- 172.21.15.117:0/282989295 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74601a73a0 con 0x7f7460102f70 2024-03-19T03:32:14.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.703+0000 7f7468200700 1 -- 172.21.15.117:0/282989295 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74601a7890 con 0x7f7460102f70 2024-03-19T03:32:14.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.704+0000 7f7468200700 1 -- 172.21.15.117:0/282989295 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7460068fd0 con 0x7f7460102f70 2024-03-19T03:32:14.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.706+0000 7f745e000700 1 -- 172.21.15.117:0/282989295 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f744c004750 con 0x7f7460102f70 2024-03-19T03:32:14.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.707+0000 7f745e000700 1 --2- 172.21.15.117:0/282989295 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f74480a29a0 0x7f74480a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:14.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.707+0000 7f745e000700 1 -- 172.21.15.117:0/282989295 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f744c0cf0f0 con 0x7f7460102f70 2024-03-19T03:32:14.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.707+0000 7f7465800700 1 --2- 172.21.15.117:0/282989295 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f74480a29a0 0x7f74480a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:14.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.708+0000 7f7465800700 1 --2- 172.21.15.117:0/282989295 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f74480a29a0 0x7f74480a4e60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7454011490 tx=0x7f7454009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:14.861 INFO:teuthology.orchestra.run.smithi117.stdout:107374182479 2024-03-19T03:32:14.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.711+0000 7f745e000700 1 -- 172.21.15.117:0/282989295 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f744c09bc90 con 0x7f7460102f70 2024-03-19T03:32:14.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.942+0000 7f7468200700 1 -- 172.21.15.117:0/282989295 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f746002dbb0 con 0x7f7460102f70 2024-03-19T03:32:14.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.942+0000 7f745e000700 1 -- 172.21.15.117:0/282989295 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f744c09b590 con 0x7f7460102f70 2024-03-19T03:32:14.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.945+0000 7f7468200700 1 -- 172.21.15.117:0/282989295 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f74480a29a0 msgr2=0x7f74480a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:14.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.945+0000 7f7468200700 1 --2- 172.21.15.117:0/282989295 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f74480a29a0 0x7f74480a4e60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7454011490 tx=0x7f7454009260 comp rx=0 tx=0).stop 2024-03-19T03:32:14.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.945+0000 7f7468200700 1 -- 172.21.15.117:0/282989295 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7460102f70 msgr2=0x7f7460074b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:14.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.945+0000 7f7468200700 1 --2- 172.21.15.117:0/282989295 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7460102f70 0x7f7460074b50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f744c00e9f0 tx=0x7f744c00ed00 comp rx=0 tx=0).stop 2024-03-19T03:32:14.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.945+0000 7f7468200700 1 -- 172.21.15.117:0/282989295 shutdown_connections 2024-03-19T03:32:14.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.945+0000 7f7468200700 1 --2- 172.21.15.117:0/282989295 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f74480a29a0 0x7f74480a4e60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:14.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.945+0000 7f7468200700 1 --2- 172.21.15.117:0/282989295 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7460107690 0x7f7460075090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:14.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.946+0000 7f7468200700 1 --2- 172.21.15.117:0/282989295 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f74601025c0 0x7f7460074610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:14.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.946+0000 7f7468200700 1 --2- 172.21.15.117:0/282989295 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7460102f70 0x7f7460074b50 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:14.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.946+0000 7f7468200700 1 -- 172.21.15.117:0/282989295 >> 172.21.15.117:0/282989295 conn(0x7f7460079470 msgr2=0x7f74601003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:14.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.946+0000 7f7468200700 1 -- 172.21.15.117:0/282989295 shutdown_connections 2024-03-19T03:32:14.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.946+0000 7f7468200700 1 -- 172.21.15.117:0/282989295 wait complete. 2024-03-19T03:32:14.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:14 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2391381387' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-19T03:32:14.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:14 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/195361703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-19T03:32:14.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:14 smithi119 ceph-mon[66861]: pgmap v369: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-19T03:32:14.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:14 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1737335864' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-19T03:32:15.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.792+0000 7f8fb1600700 1 -- 172.21.15.117:0/3286047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8fac104ec0 msgr2=0x7f8fac111380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:15.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.792+0000 7f8fb1600700 1 --2- 172.21.15.117:0/3286047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8fac104ec0 0x7f8fac111380 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8fa400b0d0 tx=0x7f8fa400b3e0 comp rx=0 tx=0).stop 2024-03-19T03:32:15.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.792+0000 7f8fb1600700 1 -- 172.21.15.117:0/3286047 shutdown_connections 2024-03-19T03:32:15.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.792+0000 7f8fb1600700 1 --2- 172.21.15.117:0/3286047 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8fac104ec0 0x7f8fac111380 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:15.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.792+0000 7f8fb1600700 1 --2- 172.21.15.117:0/3286047 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8fac10a5b0 0x7f8fac10a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:15.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.792+0000 7f8fb1600700 1 --2- 172.21.15.117:0/3286047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8fac104500 0x7f8fac104980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:15.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.792+0000 7f8fb1600700 1 -- 172.21.15.117:0/3286047 >> 172.21.15.117:0/3286047 conn(0x7f8fac0ffff0 msgr2=0x7f8fac102410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:15.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.793+0000 7f8fb1600700 1 -- 172.21.15.117:0/3286047 shutdown_connections 2024-03-19T03:32:15.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.793+0000 7f8fb1600700 1 -- 172.21.15.117:0/3286047 wait complete. 2024-03-19T03:32:15.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.794+0000 7f8fb1600700 1 Processor -- start 2024-03-19T03:32:15.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.794+0000 7f8fb1600700 1 -- start start 2024-03-19T03:32:15.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.795+0000 7f8fb1600700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8fac104500 0x7f8fac1a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:15.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.795+0000 7f8fb1600700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8fac104ec0 0x7f8fac1a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:15.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.795+0000 7f8fb1600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8fac10a5b0 0x7f8fac19a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:15.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.795+0000 7f8fb1600700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac114840 con 0x7f8fac104500 2024-03-19T03:32:15.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.795+0000 7f8fb1600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac1146c0 con 0x7f8fac10a5b0 2024-03-19T03:32:15.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.795+0000 7f8fb1600700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac1149c0 con 0x7f8fac104ec0 2024-03-19T03:32:15.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.795+0000 7f8faaa00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8fac104500 0x7f8fac1a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:15.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.795+0000 7f8faa000700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8fac104ec0 0x7f8fac1a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:15.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.795+0000 7f8faa000700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8fac104ec0 0x7f8fac1a0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:52952/0 (socket says 172.21.15.117:52952) 2024-03-19T03:32:15.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.795+0000 7f8faaa00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8fac104500 0x7f8fac1a0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:35834/0 (socket says 172.21.15.117:35834) 2024-03-19T03:32:15.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.795+0000 7f8faa000700 1 -- 172.21.15.117:0/195361703 learned_addr learned my addr 172.21.15.117:0/195361703 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:15.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.796+0000 7f8fab400700 1 --2- 172.21.15.117:0/195361703 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8fac10a5b0 0x7f8fac19a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:15.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.796+0000 7f8faa000700 1 -- 172.21.15.117:0/195361703 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8fac10a5b0 msgr2=0x7f8fac19a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:15.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.796+0000 7f8faa000700 1 --2- 172.21.15.117:0/195361703 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8fac10a5b0 0x7f8fac19a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:15.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.796+0000 7f8faa000700 1 -- 172.21.15.117:0/195361703 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8fac104500 msgr2=0x7f8fac1a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:15.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.796+0000 7f8faa000700 1 --2- 172.21.15.117:0/195361703 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8fac104500 0x7f8fac1a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:15.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.796+0000 7f8faa000700 1 -- 172.21.15.117:0/195361703 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fac19b120 con 0x7f8fac104ec0 2024-03-19T03:32:15.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.796+0000 7f8faaa00700 1 --2- 172.21.15.117:0/195361703 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8fac104500 0x7f8fac1a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:32:15.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.796+0000 7f8faa000700 1 --2- 172.21.15.117:0/195361703 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8fac104ec0 0x7f8fac1a0d40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800e9a0 tx=0x7f8f9800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:15.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.797+0000 7f8fa3400700 1 -- 172.21.15.117:0/195361703 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f9800cbd0 con 0x7f8fac104ec0 2024-03-19T03:32:15.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.797+0000 7f8fa3400700 1 -- 172.21.15.117:0/195361703 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f9800cd70 con 0x7f8fac104ec0 2024-03-19T03:32:15.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.797+0000 7f8fb1600700 1 -- 172.21.15.117:0/195361703 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fac19b410 con 0x7f8fac104ec0 2024-03-19T03:32:15.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.797+0000 7f8fa3400700 1 -- 172.21.15.117:0/195361703 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f98018750 con 0x7f8fac104ec0 2024-03-19T03:32:15.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.797+0000 7f8fb1600700 1 -- 172.21.15.117:0/195361703 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fac1a76d0 con 0x7f8fac104ec0 2024-03-19T03:32:15.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.798+0000 7f8fb1600700 1 -- 172.21.15.117:0/195361703 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fac068fd0 con 0x7f8fac104ec0 2024-03-19T03:32:15.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.802+0000 7f8fa3400700 1 -- 172.21.15.117:0/195361703 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f8f98010430 con 0x7f8fac104ec0 2024-03-19T03:32:15.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.803+0000 7f8fa3400700 1 --2- 172.21.15.117:0/195361703 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8f900a2920 0x7f8f900a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:15.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.804+0000 7f8fa3400700 1 -- 172.21.15.117:0/195361703 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f8f980cf250 con 0x7f8fac104ec0 2024-03-19T03:32:15.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.804+0000 7f8faaa00700 1 --2- 172.21.15.117:0/195361703 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8f900a2920 0x7f8f900a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:15.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.805+0000 7f8faaa00700 1 --2- 172.21.15.117:0/195361703 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8f900a2920 0x7f8f900a4de0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c005950 tx=0x7f8f9c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:15.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:13.807+0000 7f8fa3400700 1 -- 172.21.15.117:0/195361703 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8f9809bdf0 con 0x7f8fac104ec0 2024-03-19T03:32:15.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.139+0000 7f8fb1600700 1 -- 172.21.15.117:0/195361703 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f8fac1a7900 con 0x7f8fac104ec0 2024-03-19T03:32:15.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.140+0000 7f8fa3400700 1 -- 172.21.15.117:0/195361703 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8f9809b6f0 con 0x7f8fac104ec0 2024-03-19T03:32:15.464 INFO:teuthology.orchestra.run.smithi117.stdout:34359738463 2024-03-19T03:32:15.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.143+0000 7f8fb1600700 1 -- 172.21.15.117:0/195361703 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8f900a2920 msgr2=0x7f8f900a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:15.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.143+0000 7f8fb1600700 1 --2- 172.21.15.117:0/195361703 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8f900a2920 0x7f8f900a4de0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c005950 tx=0x7f8f9c0058e0 comp rx=0 tx=0).stop 2024-03-19T03:32:15.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.143+0000 7f8fb1600700 1 -- 172.21.15.117:0/195361703 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8fac104ec0 msgr2=0x7f8fac1a0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:15.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.143+0000 7f8fb1600700 1 --2- 172.21.15.117:0/195361703 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8fac104ec0 0x7f8fac1a0d40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800e9a0 tx=0x7f8f9800ecb0 comp rx=0 tx=0).stop 2024-03-19T03:32:15.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.144+0000 7f8fb1600700 1 -- 172.21.15.117:0/195361703 shutdown_connections 2024-03-19T03:32:15.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.144+0000 7f8fb1600700 1 --2- 172.21.15.117:0/195361703 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8f900a2920 0x7f8f900a4de0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:15.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.144+0000 7f8fb1600700 1 --2- 172.21.15.117:0/195361703 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8fac10a5b0 0x7f8fac19a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:15.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.144+0000 7f8fb1600700 1 --2- 172.21.15.117:0/195361703 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8fac104500 0x7f8fac1a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:15.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.144+0000 7f8fb1600700 1 --2- 172.21.15.117:0/195361703 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8fac104ec0 0x7f8fac1a0d40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:15.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.144+0000 7f8fb1600700 1 -- 172.21.15.117:0/195361703 >> 172.21.15.117:0/195361703 conn(0x7f8fac0ffff0 msgr2=0x7f8fac101430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:15.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.145+0000 7f8fb1600700 1 -- 172.21.15.117:0/195361703 shutdown_connections 2024-03-19T03:32:15.466 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:14.145+0000 7f8fb1600700 1 -- 172.21.15.117:0/195361703 wait complete. 2024-03-19T03:32:16.280 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411408 got 77309411413 for osd.2 2024-03-19T03:32:16.280 DEBUG:teuthology.parallel:result is None 2024-03-19T03:32:17.709 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:17 smithi117 ceph-mon[31838]: pgmap v370: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-19T03:32:17.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:17 smithi119 ceph-mon[66861]: pgmap v370: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-19T03:32:17.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.736+0000 7fd284000700 1 -- 172.21.15.117:0/2353539507 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd27c069fc0 msgr2=0x7fd27c06a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:17.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.736+0000 7fd284000700 1 --2- 172.21.15.117:0/2353539507 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd27c069fc0 0x7fd27c06a420 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd268009990 tx=0x7fd268009ca0 comp rx=0 tx=0).stop 2024-03-19T03:32:17.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.736+0000 7fd284000700 1 -- 172.21.15.117:0/2353539507 shutdown_connections 2024-03-19T03:32:17.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.736+0000 7fd284000700 1 --2- 172.21.15.117:0/2353539507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd27c06a960 0x7fd27c10f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:17.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.736+0000 7fd284000700 1 --2- 172.21.15.117:0/2353539507 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd27c108360 0x7fd27c108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:17.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.736+0000 7fd284000700 1 --2- 172.21.15.117:0/2353539507 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd27c069fc0 0x7fd27c06a420 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:17.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.736+0000 7fd284000700 1 -- 172.21.15.117:0/2353539507 >> 172.21.15.117:0/2353539507 conn(0x7fd27c078140 msgr2=0x7fd27c078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:17.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.736+0000 7fd284000700 1 -- 172.21.15.117:0/2353539507 shutdown_connections 2024-03-19T03:32:17.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.736+0000 7fd284000700 1 -- 172.21.15.117:0/2353539507 wait complete. 2024-03-19T03:32:17.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.737+0000 7fd284000700 1 Processor -- start 2024-03-19T03:32:17.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.738+0000 7fd284000700 1 -- start start 2024-03-19T03:32:17.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.738+0000 7fd284000700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd27c069fc0 0x7fd27c1a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:17.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.738+0000 7fd284000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd27c06a960 0x7fd27c1a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:17.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.738+0000 7fd284000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd27c108360 0x7fd27c19aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:17.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.738+0000 7fd284000700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd27c1125e0 con 0x7fd27c108360 2024-03-19T03:32:17.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.738+0000 7fd284000700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd27c112460 con 0x7fd27c06a960 2024-03-19T03:32:17.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.738+0000 7fd284000700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fd27c112760 con 0x7fd27c069fc0 2024-03-19T03:32:17.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.738+0000 7fd280c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd27c06a960 0x7fd27c1a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:17.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.738+0000 7fd280c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd27c06a960 0x7fd27c1a0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:58024/0 (socket says 172.21.15.117:58024) 2024-03-19T03:32:17.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.739+0000 7fd280c00700 1 -- 172.21.15.117:0/3141883204 learned_addr learned my addr 172.21.15.117:0/3141883204 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:17.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.739+0000 7fd281600700 1 --2- 172.21.15.117:0/3141883204 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd27c069fc0 0x7fd27c1a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:17.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.739+0000 7fd282000700 1 --2- 172.21.15.117:0/3141883204 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd27c108360 0x7fd27c19aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:17.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.739+0000 7fd280c00700 1 -- 172.21.15.117:0/3141883204 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd27c069fc0 msgr2=0x7fd27c1a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:17.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.739+0000 7fd280c00700 1 --2- 172.21.15.117:0/3141883204 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd27c069fc0 0x7fd27c1a08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:17.786 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.739+0000 7fd280c00700 1 -- 172.21.15.117:0/3141883204 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd27c108360 msgr2=0x7fd27c19aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:17.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.739+0000 7fd280c00700 1 --2- 172.21.15.117:0/3141883204 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd27c108360 0x7fd27c19aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:17.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.739+0000 7fd280c00700 1 -- 172.21.15.117:0/3141883204 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd27c19b1e0 con 0x7fd27c06a960 2024-03-19T03:32:17.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.739+0000 7fd282000700 1 --2- 172.21.15.117:0/3141883204 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd27c108360 0x7fd27c19aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:32:17.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.739+0000 7fd280c00700 1 --2- 172.21.15.117:0/3141883204 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd27c06a960 0x7fd27c1a0e00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd2680095b0 tx=0x7fd268005590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:17.787 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.740+0000 7fd27a000700 1 -- 172.21.15.117:0/3141883204 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd268026070 con 0x7fd27c06a960 2024-03-19T03:32:17.787 INFO:teuthology.orchestra.run.smithi117.stdout:287762808877 2024-03-19T03:32:17.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.740+0000 7fd27a000700 1 -- 172.21.15.117:0/3141883204 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2680056b0 con 0x7fd27c06a960 2024-03-19T03:32:17.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.740+0000 7fd284000700 1 -- 172.21.15.117:0/3141883204 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd27c19b470 con 0x7fd27c06a960 2024-03-19T03:32:17.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.740+0000 7fd27a000700 1 -- 172.21.15.117:0/3141883204 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2680173f0 con 0x7fd27c06a960 2024-03-19T03:32:17.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.740+0000 7fd284000700 1 -- 172.21.15.117:0/3141883204 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd27c1a7790 con 0x7fd27c06a960 2024-03-19T03:32:17.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.741+0000 7fd284000700 1 -- 172.21.15.117:0/3141883204 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd27c068fd0 con 0x7fd27c06a960 2024-03-19T03:32:17.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.743+0000 7fd27a000700 1 -- 172.21.15.117:0/3141883204 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fd268029070 con 0x7fd27c06a960 2024-03-19T03:32:17.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.744+0000 7fd27a000700 1 --2- 172.21.15.117:0/3141883204 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd2640a29a0 0x7fd2640a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:17.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.744+0000 7fd27a000700 1 -- 172.21.15.117:0/3141883204 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fd2680cfb80 con 0x7fd27c06a960 2024-03-19T03:32:17.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.745+0000 7fd281600700 1 --2- 172.21.15.117:0/3141883204 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd2640a29a0 0x7fd2640a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:17.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.745+0000 7fd281600700 1 --2- 172.21.15.117:0/3141883204 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd2640a29a0 0x7fd2640a4e60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd270009680 tx=0x7fd270009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:17.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:11.751+0000 7fd27a000700 1 -- 172.21.15.117:0/3141883204 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd268013070 con 0x7fd27c06a960 2024-03-19T03:32:17.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.032+0000 7fd284000700 1 -- 172.21.15.117:0/3141883204 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fd27c1a7990 con 0x7fd27c06a960 2024-03-19T03:32:17.788 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.033+0000 7fd27a000700 1 -- 172.21.15.117:0/3141883204 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd26809c7b0 con 0x7fd27c06a960 2024-03-19T03:32:17.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.035+0000 7fd284000700 1 -- 172.21.15.117:0/3141883204 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd2640a29a0 msgr2=0x7fd2640a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:17.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.035+0000 7fd284000700 1 --2- 172.21.15.117:0/3141883204 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd2640a29a0 0x7fd2640a4e60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd270009680 tx=0x7fd270009260 comp rx=0 tx=0).stop 2024-03-19T03:32:17.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.035+0000 7fd284000700 1 -- 172.21.15.117:0/3141883204 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd27c06a960 msgr2=0x7fd27c1a0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:17.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.035+0000 7fd284000700 1 --2- 172.21.15.117:0/3141883204 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd27c06a960 0x7fd27c1a0e00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd2680095b0 tx=0x7fd268005590 comp rx=0 tx=0).stop 2024-03-19T03:32:17.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.036+0000 7fd284000700 1 -- 172.21.15.117:0/3141883204 shutdown_connections 2024-03-19T03:32:17.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.036+0000 7fd284000700 1 --2- 172.21.15.117:0/3141883204 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd2640a29a0 0x7fd2640a4e60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:17.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.036+0000 7fd284000700 1 --2- 172.21.15.117:0/3141883204 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd27c108360 0x7fd27c19aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:17.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.036+0000 7fd284000700 1 --2- 172.21.15.117:0/3141883204 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd27c069fc0 0x7fd27c1a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:17.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.036+0000 7fd284000700 1 --2- 172.21.15.117:0/3141883204 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd27c06a960 0x7fd27c1a0e00 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:17.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.036+0000 7fd284000700 1 -- 172.21.15.117:0/3141883204 >> 172.21.15.117:0/3141883204 conn(0x7fd27c078140 msgr2=0x7fd27c10d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:17.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.036+0000 7fd284000700 1 -- 172.21.15.117:0/3141883204 shutdown_connections 2024-03-19T03:32:17.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:12.036+0000 7fd284000700 1 -- 172.21.15.117:0/3141883204 wait complete. 2024-03-19T03:32:17.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:17 smithi178 ceph-mon[67578]: pgmap v370: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-19T03:32:18.284 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738456 got 34359738463 for osd.0 2024-03-19T03:32:18.285 DEBUG:teuthology.parallel:result is None 2024-03-19T03:32:19.334 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182474 got 107374182479 for osd.3 2024-03-19T03:32:19.334 DEBUG:teuthology.parallel:result is None 2024-03-19T03:32:19.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:19 smithi117 ceph-mon[31838]: pgmap v371: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-19T03:32:19.564 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:32:19.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:19 smithi119 ceph-mon[66861]: pgmap v371: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-19T03:32:19.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:19 smithi178 ceph-mon[67578]: pgmap v371: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-19T03:32:19.947 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808872 got 287762808877 for osd.9 2024-03-19T03:32:19.947 DEBUG:teuthology.parallel:result is None 2024-03-19T03:32:20.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.517+0000 7f51f3000700 1 -- 172.21.15.117:0/3754472358 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51ec1134e0 msgr2=0x7f51ec1158d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:20.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.517+0000 7f51f3000700 1 --2- 172.21.15.117:0/3754472358 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51ec1134e0 0x7f51ec1158d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f51e400b0a0 tx=0x7f51e400b3b0 comp rx=0 tx=0).stop 2024-03-19T03:32:20.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.517+0000 7f51f3000700 1 -- 172.21.15.117:0/3754472358 shutdown_connections 2024-03-19T03:32:20.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.517+0000 7f51f3000700 1 --2- 172.21.15.117:0/3754472358 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51ec1134e0 0x7f51ec1158d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:20.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.517+0000 7f51f3000700 1 --2- 172.21.15.117:0/3754472358 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51ec074540 0x7f51ec074920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:20.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.517+0000 7f51f3000700 1 --2- 172.21.15.117:0/3754472358 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51ec074e60 0x7f51ec112fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:20.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.517+0000 7f51f3000700 1 -- 172.21.15.117:0/3754472358 >> 172.21.15.117:0/3754472358 conn(0x7f51ec079470 msgr2=0x7f51ec079880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:20.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.517+0000 7f51f3000700 1 -- 172.21.15.117:0/3754472358 shutdown_connections 2024-03-19T03:32:20.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.517+0000 7f51f3000700 1 -- 172.21.15.117:0/3754472358 wait complete. 2024-03-19T03:32:20.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.519+0000 7f51f3000700 1 Processor -- start 2024-03-19T03:32:20.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.519+0000 7f51f3000700 1 -- start start 2024-03-19T03:32:20.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.519+0000 7f51f3000700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51ec074540 0x7f51ec19f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:20.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.519+0000 7f51f3000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51ec074e60 0x7f51ec19f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:20.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.519+0000 7f51f3000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51ec1134e0 0x7f51ec1a3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:20.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.519+0000 7f51f3000700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec118d10 con 0x7f51ec1134e0 2024-03-19T03:32:20.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.519+0000 7f51f3000700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec118b90 con 0x7f51ec074e60 2024-03-19T03:32:20.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.519+0000 7f51f3000700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec118e90 con 0x7f51ec074540 2024-03-19T03:32:20.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.519+0000 7f51e3e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51ec074e60 0x7f51ec19f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:20.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.520+0000 7f51e3e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51ec074e60 0x7f51ec19f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:41746/0 (socket says 172.21.15.117:41746) 2024-03-19T03:32:20.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.520+0000 7f51e3e00700 1 -- 172.21.15.117:0/965720996 learned_addr learned my addr 172.21.15.117:0/965720996 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:20.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.520+0000 7f51ebe00700 1 --2- 172.21.15.117:0/965720996 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51ec074540 0x7f51ec19f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:20.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.520+0000 7f51e3e00700 1 -- 172.21.15.117:0/965720996 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51ec074540 msgr2=0x7f51ec19f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:20.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.520+0000 7f51e3e00700 1 --2- 172.21.15.117:0/965720996 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51ec074540 0x7f51ec19f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:20.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.520+0000 7f51e3e00700 1 -- 172.21.15.117:0/965720996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51ec1134e0 msgr2=0x7f51ec1a3950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T03:32:20.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.520+0000 7f51e3e00700 1 --2- 172.21.15.117:0/965720996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51ec1134e0 0x7f51ec1a3950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:20.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.520+0000 7f51e3e00700 1 -- 172.21.15.117:0/965720996 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51ec1a4090 con 0x7f51ec074e60 2024-03-19T03:32:20.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.520+0000 7f51e3e00700 1 --2- 172.21.15.117:0/965720996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51ec074e60 0x7f51ec19f5c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f51d400e970 tx=0x7f51d400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:20.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.520+0000 7f51e9600700 1 -- 172.21.15.117:0/965720996 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d400cbd0 con 0x7f51ec074e60 2024-03-19T03:32:20.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.520+0000 7f51e9600700 1 -- 172.21.15.117:0/965720996 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51d4004500 con 0x7f51ec074e60 2024-03-19T03:32:20.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.520+0000 7f51f3000700 1 -- 172.21.15.117:0/965720996 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51ec1a4380 con 0x7f51ec074e60 2024-03-19T03:32:20.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.520+0000 7f51e9600700 1 -- 172.21.15.117:0/965720996 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d4003e40 con 0x7f51ec074e60 2024-03-19T03:32:20.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.521+0000 7f51f3000700 1 -- 172.21.15.117:0/965720996 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51ec1abc00 con 0x7f51ec074e60 2024-03-19T03:32:20.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.522+0000 7f51f3000700 1 -- 172.21.15.117:0/965720996 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51ec068fd0 con 0x7f51ec074e60 2024-03-19T03:32:20.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.523+0000 7f51e9600700 1 -- 172.21.15.117:0/965720996 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f51d4004000 con 0x7f51ec074e60 2024-03-19T03:32:20.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.525+0000 7f51e9600700 1 --2- 172.21.15.117:0/965720996 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f51d00a2920 0x7f51d00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:20.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.525+0000 7f51e9600700 1 -- 172.21.15.117:0/965720996 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f51d40cf350 con 0x7f51ec074e60 2024-03-19T03:32:20.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.525+0000 7f51ebe00700 1 --2- 172.21.15.117:0/965720996 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f51d00a2920 0x7f51d00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:20.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.526+0000 7f51ebe00700 1 --2- 172.21.15.117:0/965720996 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f51d00a2920 0x7f51d00a4de0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f51dc009c60 tx=0x7f51dc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:20.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.532+0000 7f51e9600700 1 -- 172.21.15.117:0/965720996 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f51d409bef0 con 0x7f51ec074e60 2024-03-19T03:32:20.765 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.763+0000 7f51f3000700 1 -- 172.21.15.117:0/965720996 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f51ec1abe30 con 0x7f51ec074e60 2024-03-19T03:32:20.765 INFO:teuthology.orchestra.run.smithi117.stdout:163208757317 2024-03-19T03:32:20.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.764+0000 7f51e9600700 1 -- 172.21.15.117:0/965720996 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f51d409b7f0 con 0x7f51ec074e60 2024-03-19T03:32:20.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.767+0000 7f51f3000700 1 -- 172.21.15.117:0/965720996 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f51d00a2920 msgr2=0x7f51d00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:20.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.767+0000 7f51f3000700 1 --2- 172.21.15.117:0/965720996 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f51d00a2920 0x7f51d00a4de0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f51dc009c60 tx=0x7f51dc009260 comp rx=0 tx=0).stop 2024-03-19T03:32:20.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.767+0000 7f51f3000700 1 -- 172.21.15.117:0/965720996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51ec074e60 msgr2=0x7f51ec19f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:20.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.767+0000 7f51f3000700 1 --2- 172.21.15.117:0/965720996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51ec074e60 0x7f51ec19f5c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f51d400e970 tx=0x7f51d400ec80 comp rx=0 tx=0).stop 2024-03-19T03:32:20.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.768+0000 7f51f3000700 1 -- 172.21.15.117:0/965720996 shutdown_connections 2024-03-19T03:32:20.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.768+0000 7f51f3000700 1 --2- 172.21.15.117:0/965720996 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f51d00a2920 0x7f51d00a4de0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:20.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.768+0000 7f51f3000700 1 --2- 172.21.15.117:0/965720996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51ec1134e0 0x7f51ec1a3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:20.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.768+0000 7f51f3000700 1 --2- 172.21.15.117:0/965720996 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51ec074540 0x7f51ec19f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:20.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.768+0000 7f51f3000700 1 --2- 172.21.15.117:0/965720996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51ec074e60 0x7f51ec19f5c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:20.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.768+0000 7f51f3000700 1 -- 172.21.15.117:0/965720996 >> 172.21.15.117:0/965720996 conn(0x7f51ec079470 msgr2=0x7f51ec112230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:20.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.768+0000 7f51f3000700 1 -- 172.21.15.117:0/965720996 shutdown_connections 2024-03-19T03:32:20.770 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.768+0000 7f51f3000700 1 -- 172.21.15.117:0/965720996 wait complete. 2024-03-19T03:32:20.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.858+0000 7f7e47200700 1 -- 172.21.15.117:0/3376771996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e40069fc0 msgr2=0x7f7e4006a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:20.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.858+0000 7f7e47200700 1 --2- 172.21.15.117:0/3376771996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e40069fc0 0x7f7e4006a3a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7e30009990 tx=0x7f7e30009ca0 comp rx=0 tx=0).stop 2024-03-19T03:32:20.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.858+0000 7f7e47200700 1 -- 172.21.15.117:0/3376771996 shutdown_connections 2024-03-19T03:32:20.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.858+0000 7f7e47200700 1 --2- 172.21.15.117:0/3376771996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e4010edc0 0x7f7e401111b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:20.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.858+0000 7f7e47200700 1 --2- 172.21.15.117:0/3376771996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e40069fc0 0x7f7e4006a3a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:20.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.858+0000 7f7e47200700 1 --2- 172.21.15.117:0/3376771996 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e4006a8e0 0x7f7e4010e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:20.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.858+0000 7f7e47200700 1 -- 172.21.15.117:0/3376771996 >> 172.21.15.117:0/3376771996 conn(0x7f7e40078140 msgr2=0x7f7e40078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:20.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.858+0000 7f7e47200700 1 -- 172.21.15.117:0/3376771996 shutdown_connections 2024-03-19T03:32:20.860 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.860+0000 7f7e47200700 1 -- 172.21.15.117:0/3376771996 wait complete. 2024-03-19T03:32:20.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.861+0000 7f7e47200700 1 Processor -- start 2024-03-19T03:32:20.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.861+0000 7f7e47200700 1 -- start start 2024-03-19T03:32:20.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.861+0000 7f7e47200700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e40069fc0 0x7f7e401a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:20.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.861+0000 7f7e47200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e4006a8e0 0x7f7e401a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:20.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.861+0000 7f7e47200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e4010edc0 0x7f7e4019ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:20.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.861+0000 7f7e47200700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7e40114840 con 0x7f7e4010edc0 2024-03-19T03:32:20.862 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.861+0000 7f7e47200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7e401146c0 con 0x7f7e4006a8e0 2024-03-19T03:32:20.863 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.861+0000 7f7e47200700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f7e401149c0 con 0x7f7e40069fc0 2024-03-19T03:32:20.864 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.862+0000 7f7e3f400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e4006a8e0 0x7f7e401a0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:20.864 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.862+0000 7f7e3f400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e4006a8e0 0x7f7e401a0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:41764/0 (socket says 172.21.15.117:41764) 2024-03-19T03:32:20.864 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.862+0000 7f7e3f400700 1 -- 172.21.15.117:0/941179685 learned_addr learned my addr 172.21.15.117:0/941179685 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:20.864 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.862+0000 7f7e3fe00700 1 --2- 172.21.15.117:0/941179685 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e40069fc0 0x7f7e401a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:20.864 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.862+0000 7f7e45200700 1 --2- 172.21.15.117:0/941179685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e4010edc0 0x7f7e4019ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:20.864 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.862+0000 7f7e3f400700 1 -- 172.21.15.117:0/941179685 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e40069fc0 msgr2=0x7f7e401a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:20.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.862+0000 7f7e3f400700 1 --2- 172.21.15.117:0/941179685 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e40069fc0 0x7f7e401a0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:20.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.862+0000 7f7e3f400700 1 -- 172.21.15.117:0/941179685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e4010edc0 msgr2=0x7f7e4019ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:20.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.862+0000 7f7e3f400700 1 --2- 172.21.15.117:0/941179685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e4010edc0 0x7f7e4019ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:20.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.862+0000 7f7e3f400700 1 -- 172.21.15.117:0/941179685 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e4019b2a0 con 0x7f7e4006a8e0 2024-03-19T03:32:20.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.863+0000 7f7e3f400700 1 --2- 172.21.15.117:0/941179685 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e4006a8e0 0x7f7e401a0e70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c00e9c0 tx=0x7f7e2c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:20.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.863+0000 7f7e3cc00700 1 -- 172.21.15.117:0/941179685 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c00cbd0 con 0x7f7e4006a8e0 2024-03-19T03:32:20.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.863+0000 7f7e3cc00700 1 -- 172.21.15.117:0/941179685 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e2c00cd70 con 0x7f7e4006a8e0 2024-03-19T03:32:20.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.863+0000 7f7e3cc00700 1 -- 172.21.15.117:0/941179685 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c010430 con 0x7f7e4006a8e0 2024-03-19T03:32:20.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.863+0000 7f7e47200700 1 -- 172.21.15.117:0/941179685 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e4019b590 con 0x7f7e4006a8e0 2024-03-19T03:32:20.865 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.863+0000 7f7e47200700 1 -- 172.21.15.117:0/941179685 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e401a7800 con 0x7f7e4006a8e0 2024-03-19T03:32:20.866 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.864+0000 7f7e47200700 1 -- 172.21.15.117:0/941179685 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e401017f0 con 0x7f7e4006a8e0 2024-03-19T03:32:20.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.867+0000 7f7e3cc00700 1 -- 172.21.15.117:0/941179685 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f7e2c004750 con 0x7f7e4006a8e0 2024-03-19T03:32:20.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.868+0000 7f7e3cc00700 1 --2- 172.21.15.117:0/941179685 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7e240a29a0 0x7f7e240a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:20.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.869+0000 7f7e3cc00700 1 -- 172.21.15.117:0/941179685 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f7e2c0cf820 con 0x7f7e4006a8e0 2024-03-19T03:32:20.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.869+0000 7f7e3fe00700 1 --2- 172.21.15.117:0/941179685 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7e240a29a0 0x7f7e240a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:20.870 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.869+0000 7f7e3fe00700 1 --2- 172.21.15.117:0/941179685 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7e240a29a0 0x7f7e240a4e60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7e3000b2e0 tx=0x7f7e30005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:20.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:20.871+0000 7f7e3cc00700 1 -- 172.21.15.117:0/941179685 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7e2c026020 con 0x7f7e4006a8e0 2024-03-19T03:32:21.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.107+0000 7f7e47200700 1 -- 172.21.15.117:0/941179685 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f7e4019bfe0 con 0x7f7e4006a8e0 2024-03-19T03:32:21.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.107+0000 7f7e3cc00700 1 -- 172.21.15.117:0/941179685 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f7e2c09c3c0 con 0x7f7e4006a8e0 2024-03-19T03:32:21.109 INFO:teuthology.orchestra.run.smithi117.stdout:141733920844 2024-03-19T03:32:21.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.110+0000 7f7e47200700 1 -- 172.21.15.117:0/941179685 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7e240a29a0 msgr2=0x7f7e240a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:21.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.110+0000 7f7e47200700 1 --2- 172.21.15.117:0/941179685 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7e240a29a0 0x7f7e240a4e60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7e3000b2e0 tx=0x7f7e30005fd0 comp rx=0 tx=0).stop 2024-03-19T03:32:21.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.110+0000 7f7e47200700 1 -- 172.21.15.117:0/941179685 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e4006a8e0 msgr2=0x7f7e401a0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:21.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.110+0000 7f7e47200700 1 --2- 172.21.15.117:0/941179685 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e4006a8e0 0x7f7e401a0e70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c00e9c0 tx=0x7f7e2c00ecd0 comp rx=0 tx=0).stop 2024-03-19T03:32:21.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.111+0000 7f7e47200700 1 -- 172.21.15.117:0/941179685 shutdown_connections 2024-03-19T03:32:21.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.111+0000 7f7e47200700 1 --2- 172.21.15.117:0/941179685 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7e240a29a0 0x7f7e240a4e60 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:21.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.111+0000 7f7e47200700 1 --2- 172.21.15.117:0/941179685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e4010edc0 0x7f7e4019ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:21.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.111+0000 7f7e47200700 1 --2- 172.21.15.117:0/941179685 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e40069fc0 0x7f7e401a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:21.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.111+0000 7f7e47200700 1 --2- 172.21.15.117:0/941179685 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e4006a8e0 0x7f7e401a0e70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:21.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.111+0000 7f7e47200700 1 -- 172.21.15.117:0/941179685 >> 172.21.15.117:0/941179685 conn(0x7f7e40078140 msgr2=0x7f7e40110250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:21.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.111+0000 7f7e47200700 1 -- 172.21.15.117:0/941179685 shutdown_connections 2024-03-19T03:32:21.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.111+0000 7f7e47200700 1 -- 172.21.15.117:0/941179685 wait complete. 2024-03-19T03:32:21.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.361+0000 7fd189c00700 1 -- 172.21.15.117:0/4294847745 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd184102450 msgr2=0x7fd184102830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:21.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.361+0000 7fd189c00700 1 --2- 172.21.15.117:0/4294847745 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd184102450 0x7fd184102830 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd174009990 tx=0x7fd174009ca0 comp rx=0 tx=0).stop 2024-03-19T03:32:21.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.361+0000 7fd189c00700 1 -- 172.21.15.117:0/4294847745 shutdown_connections 2024-03-19T03:32:21.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.361+0000 7fd189c00700 1 --2- 172.21.15.117:0/4294847745 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd184107530 0x7fd18410f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:21.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.361+0000 7fd189c00700 1 --2- 172.21.15.117:0/4294847745 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd184102450 0x7fd184102830 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:21.388 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.361+0000 7fd189c00700 1 --2- 172.21.15.117:0/4294847745 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd184102e00 0x7fd184106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:21.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.361+0000 7fd189c00700 1 -- 172.21.15.117:0/4294847745 >> 172.21.15.117:0/4294847745 conn(0x7fd1840fdce0 msgr2=0x7fd184100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:21.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.361+0000 7fd189c00700 1 -- 172.21.15.117:0/4294847745 shutdown_connections 2024-03-19T03:32:21.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.361+0000 7fd189c00700 1 -- 172.21.15.117:0/4294847745 wait complete. 2024-03-19T03:32:21.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.362+0000 7fd189c00700 1 Processor -- start 2024-03-19T03:32:21.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.362+0000 7fd189c00700 1 -- start start 2024-03-19T03:32:21.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.363+0000 7fd189c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd184102450 0x7fd1841a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:21.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.363+0000 7fd189c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd184102e00 0x7fd1841a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:21.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.363+0000 7fd189c00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd184107530 0x7fd18419aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:21.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.363+0000 7fd189c00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd184112560 con 0x7fd184102450 2024-03-19T03:32:21.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.363+0000 7fd189c00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd1841123e0 con 0x7fd184102e00 2024-03-19T03:32:21.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.363+0000 7fd189c00700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fd1841126e0 con 0x7fd184107530 2024-03-19T03:32:21.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.363+0000 7fd183e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd184102450 0x7fd1841a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:21.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.363+0000 7fd183e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd184102450 0x7fd1841a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39498/0 (socket says 172.21.15.117:39498) 2024-03-19T03:32:21.389 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.363+0000 7fd183e00700 1 -- 172.21.15.117:0/2350408679 learned_addr learned my addr 172.21.15.117:0/2350408679 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:21.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.363+0000 7fd189200700 1 --2- 172.21.15.117:0/2350408679 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd184107530 0x7fd18419aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:21.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.364+0000 7fd183e00700 1 -- 172.21.15.117:0/2350408679 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd184107530 msgr2=0x7fd18419aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:21.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.364+0000 7fd183e00700 1 --2- 172.21.15.117:0/2350408679 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd184107530 0x7fd18419aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:21.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.364+0000 7fd183e00700 1 -- 172.21.15.117:0/2350408679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd184102e00 msgr2=0x7fd1841a0d80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-19T03:32:21.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.364+0000 7fd183e00700 1 --2- 172.21.15.117:0/2350408679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd184102e00 0x7fd1841a0d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:21.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.364+0000 7fd183e00700 1 -- 172.21.15.117:0/2350408679 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd18419b160 con 0x7fd184102450 2024-03-19T03:32:21.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.364+0000 7fd183e00700 1 --2- 172.21.15.117:0/2350408679 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd184102450 0x7fd1841a0840 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd174000c00 tx=0x7fd1740057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:21.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.364+0000 7fd180c00700 1 -- 172.21.15.117:0/2350408679 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd174027070 con 0x7fd184102450 2024-03-19T03:32:21.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.364+0000 7fd189c00700 1 -- 172.21.15.117:0/2350408679 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd18419b3f0 con 0x7fd184102450 2024-03-19T03:32:21.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.364+0000 7fd180c00700 1 -- 172.21.15.117:0/2350408679 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd174022470 con 0x7fd184102450 2024-03-19T03:32:21.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.364+0000 7fd180c00700 1 -- 172.21.15.117:0/2350408679 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd174017650 con 0x7fd184102450 2024-03-19T03:32:21.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.365+0000 7fd189c00700 1 -- 172.21.15.117:0/2350408679 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1841a7710 con 0x7fd184102450 2024-03-19T03:32:21.390 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.365+0000 7fd189c00700 1 -- 172.21.15.117:0/2350408679 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd184068fd0 con 0x7fd184102450 2024-03-19T03:32:21.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.368+0000 7fd180c00700 1 -- 172.21.15.117:0/2350408679 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fd17402a070 con 0x7fd184102450 2024-03-19T03:32:21.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.370+0000 7fd180c00700 1 --2- 172.21.15.117:0/2350408679 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd1680a2920 0x7fd1680a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:21.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.370+0000 7fd180c00700 1 -- 172.21.15.117:0/2350408679 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fd1740d0120 con 0x7fd184102450 2024-03-19T03:32:21.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.370+0000 7fd183400700 1 --2- 172.21.15.117:0/2350408679 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd1680a2920 0x7fd1680a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:21.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.371+0000 7fd183400700 1 --2- 172.21.15.117:0/2350408679 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd1680a2920 0x7fd1680a4de0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd1840ff4c0 tx=0x7fd170009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:21.391 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.373+0000 7fd180c00700 1 -- 172.21.15.117:0/2350408679 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd174013070 con 0x7fd184102450 2024-03-19T03:32:21.619 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:21 smithi117 ceph-mon[31838]: pgmap v372: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-19T03:32:21.619 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:21 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/965720996' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-19T03:32:21.620 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:21 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/941179685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-19T03:32:21.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.623+0000 7fd189c00700 1 -- 172.21.15.117:0/2350408679 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fd1841a78e0 con 0x7fd184102450 2024-03-19T03:32:21.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.624+0000 7fd180c00700 1 -- 172.21.15.117:0/2350408679 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd17409cd50 con 0x7fd184102450 2024-03-19T03:32:21.625 INFO:teuthology.orchestra.run.smithi117.stdout:253403070516 2024-03-19T03:32:21.628 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.627+0000 7fd189c00700 1 -- 172.21.15.117:0/2350408679 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd1680a2920 msgr2=0x7fd1680a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:21.628 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.627+0000 7fd189c00700 1 --2- 172.21.15.117:0/2350408679 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd1680a2920 0x7fd1680a4de0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd1840ff4c0 tx=0x7fd170009260 comp rx=0 tx=0).stop 2024-03-19T03:32:21.628 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.627+0000 7fd189c00700 1 -- 172.21.15.117:0/2350408679 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd184102450 msgr2=0x7fd1841a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:21.628 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.627+0000 7fd189c00700 1 --2- 172.21.15.117:0/2350408679 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd184102450 0x7fd1841a0840 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd174000c00 tx=0x7fd1740057c0 comp rx=0 tx=0).stop 2024-03-19T03:32:21.628 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.627+0000 7fd189c00700 1 -- 172.21.15.117:0/2350408679 shutdown_connections 2024-03-19T03:32:21.628 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.627+0000 7fd189c00700 1 --2- 172.21.15.117:0/2350408679 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd1680a2920 0x7fd1680a4de0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:21.628 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.627+0000 7fd189c00700 1 --2- 172.21.15.117:0/2350408679 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd184107530 0x7fd18419aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:21.628 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.627+0000 7fd189c00700 1 --2- 172.21.15.117:0/2350408679 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd184102450 0x7fd1841a0840 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:21.628 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.627+0000 7fd189c00700 1 --2- 172.21.15.117:0/2350408679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd184102e00 0x7fd1841a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:21.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.627+0000 7fd189c00700 1 -- 172.21.15.117:0/2350408679 >> 172.21.15.117:0/2350408679 conn(0x7fd1840fdce0 msgr2=0x7fd18410e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:21.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.627+0000 7fd189c00700 1 -- 172.21.15.117:0/2350408679 shutdown_connections 2024-03-19T03:32:21.629 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:21.627+0000 7fd189c00700 1 -- 172.21.15.117:0/2350408679 wait complete. 2024-03-19T03:32:21.649 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:21 smithi178 ceph-mon[67578]: pgmap v372: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-19T03:32:21.649 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:21 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/965720996' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-19T03:32:21.649 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:21 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/941179685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-19T03:32:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:21 smithi119 ceph-mon[66861]: pgmap v372: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-19T03:32:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:21 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/965720996' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-19T03:32:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:21 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/941179685' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-19T03:32:22.013 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495672 got 197568495678 for osd.6 2024-03-19T03:32:22.013 DEBUG:teuthology.parallel:result is None 2024-03-19T03:32:22.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:22 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2350408679' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-19T03:32:22.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:22 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2350408679' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-19T03:32:22.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:22 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2350408679' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-19T03:32:23.272 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070511 got 253403070516 for osd.8 2024-03-19T03:32:23.273 DEBUG:teuthology.parallel:result is None 2024-03-19T03:32:23.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:23 smithi117 ceph-mon[31838]: pgmap v373: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-03-19T03:32:23.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:23 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:32:23.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:23 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:32:23.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:23 smithi119 ceph-mon[66861]: pgmap v373: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-03-19T03:32:23.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:23 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:32:23.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:23 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:32:23.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:23 smithi178 ceph-mon[67578]: pgmap v373: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-03-19T03:32:23.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:23 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:32:23.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:23 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:32:23.855 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757312 got 163208757317 for osd.5 2024-03-19T03:32:23.855 DEBUG:teuthology.parallel:result is None 2024-03-19T03:32:24.081 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332146 got 219043332152 for osd.7 2024-03-19T03:32:24.081 DEBUG:teuthology.parallel:result is None 2024-03-19T03:32:25.454 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920839 got 141733920844 for osd.4 2024-03-19T03:32:25.455 DEBUG:teuthology.parallel:result is None 2024-03-19T03:32:25.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.726+0000 7f5444600700 1 -- 172.21.15.117:0/666881449 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f543c06a8e0 msgr2=0x7f543c10e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:25.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.726+0000 7f5444600700 1 --2- 172.21.15.117:0/666881449 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f543c06a8e0 0x7f543c10e880 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5428009990 tx=0x7f5428009ca0 comp rx=0 tx=0).stop 2024-03-19T03:32:25.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.726+0000 7f5444600700 1 -- 172.21.15.117:0/666881449 shutdown_connections 2024-03-19T03:32:25.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.726+0000 7f5444600700 1 --2- 172.21.15.117:0/666881449 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f543c10edc0 0x7f543c1111b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:25.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.726+0000 7f5444600700 1 --2- 172.21.15.117:0/666881449 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f543c069fc0 0x7f543c06a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:25.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.726+0000 7f5444600700 1 --2- 172.21.15.117:0/666881449 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f543c06a8e0 0x7f543c10e880 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:25.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.726+0000 7f5444600700 1 -- 172.21.15.117:0/666881449 >> 172.21.15.117:0/666881449 conn(0x7f543c078140 msgr2=0x7f543c078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:25.727 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:25 smithi117 ceph-mon[31838]: pgmap v374: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:32:25.727 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:25 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:32:25.727 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:25 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:32:25.727 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:25 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:32:25.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.726+0000 7f5444600700 1 -- 172.21.15.117:0/666881449 shutdown_connections 2024-03-19T03:32:25.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.726+0000 7f5444600700 1 -- 172.21.15.117:0/666881449 wait complete. 2024-03-19T03:32:25.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.728+0000 7f5444600700 1 Processor -- start 2024-03-19T03:32:25.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.728+0000 7f5444600700 1 -- start start 2024-03-19T03:32:25.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.729+0000 7f5444600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f543c069fc0 0x7f543c19ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:25.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.729+0000 7f5444600700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f543c06a8e0 0x7f543c19b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:25.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.729+0000 7f5444600700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f543c10edc0 0x7f543c19f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:25.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.729+0000 7f5444600700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f543c114850 con 0x7f543c10edc0 2024-03-19T03:32:25.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.729+0000 7f5444600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f543c1146d0 con 0x7f543c069fc0 2024-03-19T03:32:25.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.729+0000 7f5444600700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f543c1149d0 con 0x7f543c06a8e0 2024-03-19T03:32:25.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.729+0000 7f5441c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f543c069fc0 0x7f543c19ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:25.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.729+0000 7f5441c00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f543c069fc0 0x7f543c19ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:41804/0 (socket says 172.21.15.117:41804) 2024-03-19T03:32:25.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.729+0000 7f5442600700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f543c10edc0 0x7f543c19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:25.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.729+0000 7f5441c00700 1 -- 172.21.15.117:0/1731758939 learned_addr learned my addr 172.21.15.117:0/1731758939 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:25.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.729+0000 7f5441200700 1 --2- 172.21.15.117:0/1731758939 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f543c06a8e0 0x7f543c19b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:25.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.729+0000 7f5441c00700 1 -- 172.21.15.117:0/1731758939 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f543c06a8e0 msgr2=0x7f543c19b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:25.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.729+0000 7f5441c00700 1 --2- 172.21.15.117:0/1731758939 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f543c06a8e0 0x7f543c19b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:25.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.729+0000 7f5441c00700 1 -- 172.21.15.117:0/1731758939 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f543c10edc0 msgr2=0x7f543c19f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:25.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.729+0000 7f5441c00700 1 --2- 172.21.15.117:0/1731758939 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f543c10edc0 0x7f543c19f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:25.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.729+0000 7f5441c00700 1 -- 172.21.15.117:0/1731758939 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f543c19fe60 con 0x7f543c069fc0 2024-03-19T03:32:25.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.730+0000 7f5441c00700 1 --2- 172.21.15.117:0/1731758939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f543c069fc0 0x7f543c19ae50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f543000e9a0 tx=0x7f543000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:25.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.730+0000 7f543a000700 1 -- 172.21.15.117:0/1731758939 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f543000ca10 con 0x7f543c069fc0 2024-03-19T03:32:25.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.730+0000 7f543a000700 1 -- 172.21.15.117:0/1731758939 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f543000cbb0 con 0x7f543c069fc0 2024-03-19T03:32:25.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.730+0000 7f5444600700 1 -- 172.21.15.117:0/1731758939 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f543c1a0150 con 0x7f543c069fc0 2024-03-19T03:32:25.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.730+0000 7f543a000700 1 -- 172.21.15.117:0/1731758939 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5430018660 con 0x7f543c069fc0 2024-03-19T03:32:25.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.730+0000 7f5444600700 1 -- 172.21.15.117:0/1731758939 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f543c1a79d0 con 0x7f543c069fc0 2024-03-19T03:32:25.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.731+0000 7f5444600700 1 -- 172.21.15.117:0/1731758939 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f543c068fd0 con 0x7f543c069fc0 2024-03-19T03:32:25.733 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.733+0000 7f543a000700 1 -- 172.21.15.117:0/1731758939 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f5430010430 con 0x7f543c069fc0 2024-03-19T03:32:25.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:25 smithi119 ceph-mon[66861]: pgmap v374: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:32:25.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:25 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:32:25.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:25 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:32:25.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:25 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:32:25.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.735+0000 7f543a000700 1 --2- 172.21.15.117:0/1731758939 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f54240a29a0 0x7f54240a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:25.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.735+0000 7f543a000700 1 -- 172.21.15.117:0/1731758939 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f54300cf0b0 con 0x7f543c069fc0 2024-03-19T03:32:25.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.735+0000 7f5441200700 1 --2- 172.21.15.117:0/1731758939 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f54240a29a0 0x7f54240a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:25.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.736+0000 7f5441200700 1 --2- 172.21.15.117:0/1731758939 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f54240a29a0 0x7f54240a4e60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f543c102690 tx=0x7f542800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:25.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:25.739+0000 7f543a000700 1 -- 172.21.15.117:0/1731758939 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f543009bcb0 con 0x7f543c069fc0 2024-03-19T03:32:25.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:25 smithi178 ceph-mon[67578]: pgmap v374: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:32:25.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:25 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:32:25.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:25 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:32:25.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:25 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:32:26.045 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:26.045+0000 7f5444600700 1 -- 172.21.15.117:0/1731758939 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f543c02dbb0 con 0x7f543c069fc0 2024-03-19T03:32:26.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:26.046+0000 7f543a000700 1 -- 172.21.15.117:0/1731758939 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f543009b5b0 con 0x7f543c069fc0 2024-03-19T03:32:26.046 INFO:teuthology.orchestra.run.smithi117.stdout:343597383718 2024-03-19T03:32:26.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:26.049+0000 7f5444600700 1 -- 172.21.15.117:0/1731758939 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f54240a29a0 msgr2=0x7f54240a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:26.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:26.049+0000 7f5444600700 1 --2- 172.21.15.117:0/1731758939 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f54240a29a0 0x7f54240a4e60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f543c102690 tx=0x7f542800b260 comp rx=0 tx=0).stop 2024-03-19T03:32:26.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:26.049+0000 7f5444600700 1 -- 172.21.15.117:0/1731758939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f543c069fc0 msgr2=0x7f543c19ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:26.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:26.049+0000 7f5444600700 1 --2- 172.21.15.117:0/1731758939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f543c069fc0 0x7f543c19ae50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f543000e9a0 tx=0x7f543000ecb0 comp rx=0 tx=0).stop 2024-03-19T03:32:26.049 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:26.049+0000 7f5444600700 1 -- 172.21.15.117:0/1731758939 shutdown_connections 2024-03-19T03:32:26.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:26.049+0000 7f5444600700 1 --2- 172.21.15.117:0/1731758939 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f54240a29a0 0x7f54240a4e60 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:26.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:26.049+0000 7f5444600700 1 --2- 172.21.15.117:0/1731758939 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f543c10edc0 0x7f543c19f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:26.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:26.049+0000 7f5444600700 1 --2- 172.21.15.117:0/1731758939 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f543c069fc0 0x7f543c19ae50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:26.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:26.049+0000 7f5444600700 1 --2- 172.21.15.117:0/1731758939 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f543c06a8e0 0x7f543c19b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:26.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:26.049+0000 7f5444600700 1 -- 172.21.15.117:0/1731758939 >> 172.21.15.117:0/1731758939 conn(0x7f543c078140 msgr2=0x7f543c1101d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:26.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:26.049+0000 7f5444600700 1 -- 172.21.15.117:0/1731758939 shutdown_connections 2024-03-19T03:32:26.050 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:26.049+0000 7f5444600700 1 -- 172.21.15.117:0/1731758939 wait complete. 2024-03-19T03:32:26.671 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:26 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1731758939' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-19T03:32:26.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:26 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1731758939' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-19T03:32:26.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:26 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1731758939' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-19T03:32:27.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.019+0000 7f512d800700 1 -- 172.21.15.117:0/642114682 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5128075060 msgr2=0x7f512810e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:27.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.019+0000 7f512d800700 1 --2- 172.21.15.117:0/642114682 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5128075060 0x7f512810e4f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5114009990 tx=0x7f5114009ca0 comp rx=0 tx=0).stop 2024-03-19T03:32:27.019 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.019+0000 7f512d800700 1 -- 172.21.15.117:0/642114682 shutdown_connections 2024-03-19T03:32:27.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.019+0000 7f512d800700 1 --2- 172.21.15.117:0/642114682 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f512810ea30 0x7f5128110ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:27.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.019+0000 7f512d800700 1 --2- 172.21.15.117:0/642114682 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5128074740 0x7f5128074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:27.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.019+0000 7f512d800700 1 --2- 172.21.15.117:0/642114682 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5128075060 0x7f512810e4f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:27.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.019+0000 7f512d800700 1 -- 172.21.15.117:0/642114682 >> 172.21.15.117:0/642114682 conn(0x7f51280fdce0 msgr2=0x7f5128100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:27.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.019+0000 7f512d800700 1 -- 172.21.15.117:0/642114682 shutdown_connections 2024-03-19T03:32:27.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.019+0000 7f512d800700 1 -- 172.21.15.117:0/642114682 wait complete. 2024-03-19T03:32:27.020 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.020+0000 7f512d800700 1 Processor -- start 2024-03-19T03:32:27.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.021+0000 7f512d800700 1 -- start start 2024-03-19T03:32:27.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.021+0000 7f512d800700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5128074740 0x7f51281a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:27.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.021+0000 7f512d800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5128075060 0x7f51281a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:27.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.021+0000 7f512d800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f512810ea30 0x7f512819aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:27.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.021+0000 7f512d800700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5128114770 con 0x7f5128075060 2024-03-19T03:32:27.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.021+0000 7f512d800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f51281145f0 con 0x7f512810ea30 2024-03-19T03:32:27.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.021+0000 7f512d800700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f51281148f0 con 0x7f5128074740 2024-03-19T03:32:27.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.021+0000 7f512ce00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f512810ea30 0x7f512819aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:27.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.021+0000 7f5127400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5128075060 0x7f51281a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:27.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.021+0000 7f512ce00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f512810ea30 0x7f512819aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:48788/0 (socket says 172.21.15.117:48788) 2024-03-19T03:32:27.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.021+0000 7f512ce00700 1 -- 172.21.15.117:0/908742036 learned_addr learned my addr 172.21.15.117:0/908742036 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:27.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.021+0000 7f5127400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5128075060 0x7f51281a0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:56034/0 (socket says 172.21.15.117:56034) 2024-03-19T03:32:27.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.022+0000 7f5127e00700 1 --2- 172.21.15.117:0/908742036 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5128074740 0x7f51281a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:27.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.022+0000 7f512ce00700 1 -- 172.21.15.117:0/908742036 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5128074740 msgr2=0x7f51281a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:27.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.022+0000 7f512ce00700 1 --2- 172.21.15.117:0/908742036 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5128074740 0x7f51281a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:27.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.022+0000 7f512ce00700 1 -- 172.21.15.117:0/908742036 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5128075060 msgr2=0x7f51281a0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:27.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.022+0000 7f512ce00700 1 --2- 172.21.15.117:0/908742036 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5128075060 0x7f51281a0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:27.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.022+0000 7f512ce00700 1 -- 172.21.15.117:0/908742036 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f512819b1f0 con 0x7f512810ea30 2024-03-19T03:32:27.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.022+0000 7f512ce00700 1 --2- 172.21.15.117:0/908742036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f512810ea30 0x7f512819aa90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f512000ef30 tx=0x7f512000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:27.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.023+0000 7f5124c00700 1 -- 172.21.15.117:0/908742036 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51200092e0 con 0x7f512810ea30 2024-03-19T03:32:27.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.023+0000 7f5124c00700 1 -- 172.21.15.117:0/908742036 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f512000ce20 con 0x7f512810ea30 2024-03-19T03:32:27.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.023+0000 7f512d800700 1 -- 172.21.15.117:0/908742036 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f512819b480 con 0x7f512810ea30 2024-03-19T03:32:27.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.023+0000 7f5124c00700 1 -- 172.21.15.117:0/908742036 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5120007b20 con 0x7f512810ea30 2024-03-19T03:32:27.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.023+0000 7f512d800700 1 -- 172.21.15.117:0/908742036 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51281a7750 con 0x7f512810ea30 2024-03-19T03:32:27.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.025+0000 7f512d800700 1 -- 172.21.15.117:0/908742036 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5128068fd0 con 0x7f512810ea30 2024-03-19T03:32:27.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.026+0000 7f5124c00700 1 -- 172.21.15.117:0/908742036 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f512001b020 con 0x7f512810ea30 2024-03-19T03:32:27.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.028+0000 7f5124c00700 1 --2- 172.21.15.117:0/908742036 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f510c0a29a0 0x7f510c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:27.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.028+0000 7f5124c00700 1 -- 172.21.15.117:0/908742036 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f51200ceac0 con 0x7f512810ea30 2024-03-19T03:32:27.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.028+0000 7f5127e00700 1 --2- 172.21.15.117:0/908742036 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f510c0a29a0 0x7f510c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:27.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.029+0000 7f5127e00700 1 --2- 172.21.15.117:0/908742036 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f510c0a29a0 0x7f510c0a4e60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5128111300 tx=0x7f5118009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:27.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.031+0000 7f5124c00700 1 -- 172.21.15.117:0/908742036 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5120010040 con 0x7f512810ea30 2024-03-19T03:32:27.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.275+0000 7f512d800700 1 -- 172.21.15.117:0/908742036 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f512802dbb0 con 0x7f512810ea30 2024-03-19T03:32:27.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.276+0000 7f5124c00700 1 -- 172.21.15.117:0/908742036 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f512009b660 con 0x7f512810ea30 2024-03-19T03:32:27.276 INFO:teuthology.orchestra.run.smithi117.stdout:55834574942 2024-03-19T03:32:27.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.280+0000 7f512d800700 1 -- 172.21.15.117:0/908742036 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f510c0a29a0 msgr2=0x7f510c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:27.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.280+0000 7f512d800700 1 --2- 172.21.15.117:0/908742036 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f510c0a29a0 0x7f510c0a4e60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5128111300 tx=0x7f5118009260 comp rx=0 tx=0).stop 2024-03-19T03:32:27.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.280+0000 7f512d800700 1 -- 172.21.15.117:0/908742036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f512810ea30 msgr2=0x7f512819aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:27.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.280+0000 7f512d800700 1 --2- 172.21.15.117:0/908742036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f512810ea30 0x7f512819aa90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f512000ef30 tx=0x7f512000c3a0 comp rx=0 tx=0).stop 2024-03-19T03:32:27.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.280+0000 7f512d800700 1 -- 172.21.15.117:0/908742036 shutdown_connections 2024-03-19T03:32:27.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.280+0000 7f512d800700 1 --2- 172.21.15.117:0/908742036 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f510c0a29a0 0x7f510c0a4e60 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:27.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.280+0000 7f512d800700 1 --2- 172.21.15.117:0/908742036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f512810ea30 0x7f512819aa90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:27.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.280+0000 7f512d800700 1 --2- 172.21.15.117:0/908742036 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5128074740 0x7f51281a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:27.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.280+0000 7f512d800700 1 --2- 172.21.15.117:0/908742036 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5128075060 0x7f51281a0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:27.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.280+0000 7f512d800700 1 -- 172.21.15.117:0/908742036 >> 172.21.15.117:0/908742036 conn(0x7f51280fdce0 msgr2=0x7f51280fe9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:27.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.280+0000 7f512d800700 1 -- 172.21.15.117:0/908742036 shutdown_connections 2024-03-19T03:32:27.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:27.280+0000 7f512d800700 1 -- 172.21.15.117:0/908742036 wait complete. 2024-03-19T03:32:27.528 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:27 smithi117 ceph-mon[31838]: pgmap v375: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:32:27.528 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:27 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/908742036' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-19T03:32:27.558 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383712 got 343597383718 for osd.11 2024-03-19T03:32:27.558 DEBUG:teuthology.parallel:result is None 2024-03-19T03:32:27.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:27 smithi119 ceph-mon[66861]: pgmap v375: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:32:27.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:27 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/908742036' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-19T03:32:27.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:27 smithi178 ceph-mon[67578]: pgmap v375: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:32:27.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:27 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/908742036' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-19T03:32:28.307 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574935 got 55834574942 for osd.1 2024-03-19T03:32:28.307 DEBUG:teuthology.parallel:result is None 2024-03-19T03:32:28.307 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-19T03:32:28.307 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-19T03:32:28.850 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:32:29.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:29 smithi119 ceph-mon[66861]: pgmap v376: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:32:29.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:29 smithi117 ceph-mon[31838]: pgmap v376: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:32:29.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:29 smithi178 ceph-mon[67578]: pgmap v376: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:32:31.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.023+0000 7feddec00700 1 -- 172.21.15.117:0/1400697171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedd8069fc0 msgr2=0x7fedd806a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:31.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.023+0000 7feddec00700 1 --2- 172.21.15.117:0/1400697171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedd8069fc0 0x7fedd806a420 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fedc4009990 tx=0x7fedc4009ca0 comp rx=0 tx=0).stop 2024-03-19T03:32:31.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.023+0000 7feddec00700 1 -- 172.21.15.117:0/1400697171 shutdown_connections 2024-03-19T03:32:31.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.023+0000 7feddec00700 1 --2- 172.21.15.117:0/1400697171 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedd806a960 0x7fedd810f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:31.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.023+0000 7feddec00700 1 --2- 172.21.15.117:0/1400697171 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fedd8108360 0x7fedd8108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:31.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.023+0000 7feddec00700 1 --2- 172.21.15.117:0/1400697171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedd8069fc0 0x7fedd806a420 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:31.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.023+0000 7feddec00700 1 -- 172.21.15.117:0/1400697171 >> 172.21.15.117:0/1400697171 conn(0x7fedd8078140 msgr2=0x7fedd8078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:31.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.023+0000 7feddec00700 1 -- 172.21.15.117:0/1400697171 shutdown_connections 2024-03-19T03:32:31.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.023+0000 7feddec00700 1 -- 172.21.15.117:0/1400697171 wait complete. 2024-03-19T03:32:31.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.025+0000 7feddec00700 1 Processor -- start 2024-03-19T03:32:31.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.025+0000 7feddec00700 1 -- start start 2024-03-19T03:32:31.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.025+0000 7feddec00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedd8069fc0 0x7fedd81012e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:31.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.025+0000 7feddec00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedd806a960 0x7fedd8101820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:31.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.025+0000 7feddec00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fedd8108360 0x7fedd8101d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:31.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.025+0000 7feddec00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fedd8112610 con 0x7fedd806a960 2024-03-19T03:32:31.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.025+0000 7feddec00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fedd8112490 con 0x7fedd8069fc0 2024-03-19T03:32:31.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.025+0000 7feddec00700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fedd8112790 con 0x7fedd8108360 2024-03-19T03:32:31.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.026+0000 7fedd7e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedd8069fc0 0x7fedd81012e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:31.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.026+0000 7fedd7400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedd806a960 0x7fedd8101820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:31.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.026+0000 7fedd7e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedd8069fc0 0x7fedd81012e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:48808/0 (socket says 172.21.15.117:48808) 2024-03-19T03:32:31.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.026+0000 7fedd7e00700 1 -- 172.21.15.117:0/3835340221 learned_addr learned my addr 172.21.15.117:0/3835340221 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:31.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.026+0000 7feddcc00700 1 --2- 172.21.15.117:0/3835340221 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fedd8108360 0x7fedd8101d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:31.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.026+0000 7fedd7e00700 1 -- 172.21.15.117:0/3835340221 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fedd8108360 msgr2=0x7fedd8101d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:31.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.026+0000 7fedd7e00700 1 --2- 172.21.15.117:0/3835340221 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fedd8108360 0x7fedd8101d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:31.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.026+0000 7fedd7e00700 1 -- 172.21.15.117:0/3835340221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedd806a960 msgr2=0x7fedd8101820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:31.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.026+0000 7fedd7e00700 1 --2- 172.21.15.117:0/3835340221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedd806a960 0x7fedd8101820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:31.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.026+0000 7fedd7e00700 1 -- 172.21.15.117:0/3835340221 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedd81a7070 con 0x7fedd8069fc0 2024-03-19T03:32:31.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.026+0000 7fedd7e00700 1 --2- 172.21.15.117:0/3835340221 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedd8069fc0 0x7fedd81012e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fedc800e9a0 tx=0x7fedc800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:31.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.027+0000 7fedd4c00700 1 -- 172.21.15.117:0/3835340221 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc800cbd0 con 0x7fedd8069fc0 2024-03-19T03:32:31.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.027+0000 7fedd4c00700 1 -- 172.21.15.117:0/3835340221 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedc800cd70 con 0x7fedd8069fc0 2024-03-19T03:32:31.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.027+0000 7fedd4c00700 1 -- 172.21.15.117:0/3835340221 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc8018750 con 0x7fedd8069fc0 2024-03-19T03:32:31.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.027+0000 7feddec00700 1 -- 172.21.15.117:0/3835340221 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedd81a7360 con 0x7fedd8069fc0 2024-03-19T03:32:31.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.027+0000 7feddec00700 1 -- 172.21.15.117:0/3835340221 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedd81a7850 con 0x7fedd8069fc0 2024-03-19T03:32:31.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.028+0000 7feddec00700 1 -- 172.21.15.117:0/3835340221 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedd8068fd0 con 0x7fedd8069fc0 2024-03-19T03:32:31.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.030+0000 7fedd4c00700 1 -- 172.21.15.117:0/3835340221 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fedc8010430 con 0x7fedd8069fc0 2024-03-19T03:32:31.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.031+0000 7fedd4c00700 1 --2- 172.21.15.117:0/3835340221 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fedbc0a2920 0x7fedbc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:31.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.032+0000 7fedd4c00700 1 -- 172.21.15.117:0/3835340221 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fedc80cf200 con 0x7fedd8069fc0 2024-03-19T03:32:31.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.032+0000 7fedd7400700 1 --2- 172.21.15.117:0/3835340221 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fedbc0a2920 0x7fedbc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:31.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.033+0000 7fedd7400700 1 --2- 172.21.15.117:0/3835340221 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fedbc0a2920 0x7fedbc0a4de0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fedd80787f0 tx=0x7fedc4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:31.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.033+0000 7fedd4c00700 1 -- 172.21.15.117:0/3835340221 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fedc809bda0 con 0x7fedd8069fc0 2024-03-19T03:32:31.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.284+0000 7feddec00700 1 -- 172.21.15.117:0/3835340221 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fedd8102810 con 0x7fedbc0a2920 2024-03-19T03:32:31.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.308+0000 7fedd4c00700 1 -- 172.21.15.117:0/3835340221 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340712 (secure 0 0 0) 0x7fedc4079300 con 0x7fedbc0a2920 2024-03-19T03:32:31.315 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:32:31.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.315+0000 7feddec00700 1 -- 172.21.15.117:0/3835340221 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fedbc0a2920 msgr2=0x7fedbc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:31.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.315+0000 7feddec00700 1 --2- 172.21.15.117:0/3835340221 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fedbc0a2920 0x7fedbc0a4de0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fedd80787f0 tx=0x7fedc4005fd0 comp rx=0 tx=0).stop 2024-03-19T03:32:31.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.315+0000 7feddec00700 1 -- 172.21.15.117:0/3835340221 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedd8069fc0 msgr2=0x7fedd81012e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:31.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.315+0000 7feddec00700 1 --2- 172.21.15.117:0/3835340221 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedd8069fc0 0x7fedd81012e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fedc800e9a0 tx=0x7fedc800ecb0 comp rx=0 tx=0).stop 2024-03-19T03:32:31.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.316+0000 7feddec00700 1 -- 172.21.15.117:0/3835340221 shutdown_connections 2024-03-19T03:32:31.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.316+0000 7feddec00700 1 --2- 172.21.15.117:0/3835340221 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fedbc0a2920 0x7fedbc0a4de0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:31.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.316+0000 7feddec00700 1 --2- 172.21.15.117:0/3835340221 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fedd8108360 0x7fedd8101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:31.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.316+0000 7feddec00700 1 --2- 172.21.15.117:0/3835340221 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedd8069fc0 0x7fedd81012e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:31.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.316+0000 7feddec00700 1 --2- 172.21.15.117:0/3835340221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedd806a960 0x7fedd8101820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:31.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.316+0000 7feddec00700 1 -- 172.21.15.117:0/3835340221 >> 172.21.15.117:0/3835340221 conn(0x7fedd8078140 msgr2=0x7fedd80fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:31.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.316+0000 7feddec00700 1 -- 172.21.15.117:0/3835340221 shutdown_connections 2024-03-19T03:32:31.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:31.316+0000 7feddec00700 1 -- 172.21.15.117:0/3835340221 wait complete. 2024-03-19T03:32:31.318 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-03-19T03:32:31.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:31 smithi117 ceph-mon[31838]: pgmap v377: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:32:31.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:31 smithi119 ceph-mon[66861]: pgmap v377: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:32:31.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:31 smithi178 ceph-mon[67578]: pgmap v377: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:32:32.092 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":377,"stamp":"2024-03-19T03:32:30.323931+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662012,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":949,"num_read_kb":599,"num_write":427,"num_write_kb":3082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1180853,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":351,"ondisk_log_size":351,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361528,"kb_used_data":12296,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580296,"statfs":{"total":1151940427776,"available":1151570223104,"internally_reserved":0,"allocated":12591104,"data_stored":7971182,"data_compressed":361626,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19138,"internal_metadata":357414206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":65,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004611"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.427095+0000","last_change":"2024-03-19T03:30:35.676095+0000","last_active":"2024-03-19T03:31:39.427095+0000","last_peered":"2024-03-19T03:31:39.427095+0000","last_clean":"2024-03-19T03:31:39.427095+0000","last_became_active":"2024-03-19T03:30:35.675772+0000","last_became_peered":"2024-03-19T03:30:35.675772+0000","last_unstale":"2024-03-19T03:31:39.427095+0000","last_undegraded":"2024-03-19T03:31:39.427095+0000","last_fullsized":"2024-03-19T03:31:39.427095+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:12:31.902482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.435407+0000","last_change":"2024-03-19T03:30:35.672451+0000","last_active":"2024-03-19T03:31:39.435407+0000","last_peered":"2024-03-19T03:31:39.435407+0000","last_clean":"2024-03-19T03:31:39.435407+0000","last_became_active":"2024-03-19T03:30:35.671516+0000","last_became_peered":"2024-03-19T03:30:35.671516+0000","last_unstale":"2024-03-19T03:31:39.435407+0000","last_undegraded":"2024-03-19T03:31:39.435407+0000","last_fullsized":"2024-03-19T03:31:39.435407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:14:07.021704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.427269+0000","last_change":"2024-03-19T03:30:35.674425+0000","last_active":"2024-03-19T03:31:39.427269+0000","last_peered":"2024-03-19T03:31:39.427269+0000","last_clean":"2024-03-19T03:31:39.427269+0000","last_became_active":"2024-03-19T03:30:35.674012+0000","last_became_peered":"2024-03-19T03:30:35.674012+0000","last_unstale":"2024-03-19T03:31:39.427269+0000","last_undegraded":"2024-03-19T03:31:39.427269+0000","last_fullsized":"2024-03-19T03:31:39.427269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:51:44.850847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.423258+0000","last_change":"2024-03-19T03:30:35.649218+0000","last_active":"2024-03-19T03:31:39.423258+0000","last_peered":"2024-03-19T03:31:39.423258+0000","last_clean":"2024-03-19T03:31:39.423258+0000","last_became_active":"2024-03-19T03:30:35.648848+0000","last_became_peered":"2024-03-19T03:30:35.648848+0000","last_unstale":"2024-03-19T03:31:39.423258+0000","last_undegraded":"2024-03-19T03:31:39.423258+0000","last_fullsized":"2024-03-19T03:31:39.423258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:24.204126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425132+0000","last_change":"2024-03-19T03:30:35.670208+0000","last_active":"2024-03-19T03:31:39.425132+0000","last_peered":"2024-03-19T03:31:39.425132+0000","last_clean":"2024-03-19T03:31:39.425132+0000","last_became_active":"2024-03-19T03:30:35.669939+0000","last_became_peered":"2024-03-19T03:30:35.669939+0000","last_unstale":"2024-03-19T03:31:39.425132+0000","last_undegraded":"2024-03-19T03:31:39.425132+0000","last_fullsized":"2024-03-19T03:31:39.425132+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:59.687262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.423789+0000","last_change":"2024-03-19T03:30:35.662329+0000","last_active":"2024-03-19T03:31:39.423789+0000","last_peered":"2024-03-19T03:31:39.423789+0000","last_clean":"2024-03-19T03:31:39.423789+0000","last_became_active":"2024-03-19T03:30:35.661969+0000","last_became_peered":"2024-03-19T03:30:35.661969+0000","last_unstale":"2024-03-19T03:31:39.423789+0000","last_undegraded":"2024-03-19T03:31:39.423789+0000","last_fullsized":"2024-03-19T03:31:39.423789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:29.265017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426342+0000","last_change":"2024-03-19T03:30:35.662586+0000","last_active":"2024-03-19T03:31:39.426342+0000","last_peered":"2024-03-19T03:31:39.426342+0000","last_clean":"2024-03-19T03:31:39.426342+0000","last_became_active":"2024-03-19T03:30:35.661774+0000","last_became_peered":"2024-03-19T03:30:35.661774+0000","last_unstale":"2024-03-19T03:31:39.426342+0000","last_undegraded":"2024-03-19T03:31:39.426342+0000","last_fullsized":"2024-03-19T03:31:39.426342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:49.258178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.420906+0000","last_change":"2024-03-19T03:30:35.638512+0000","last_active":"2024-03-19T03:31:39.420906+0000","last_peered":"2024-03-19T03:31:39.420906+0000","last_clean":"2024-03-19T03:31:39.420906+0000","last_became_active":"2024-03-19T03:30:35.637205+0000","last_became_peered":"2024-03-19T03:30:35.637205+0000","last_unstale":"2024-03-19T03:31:39.420906+0000","last_undegraded":"2024-03-19T03:31:39.420906+0000","last_fullsized":"2024-03-19T03:31:39.420906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:29:34.578520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.437823+0000","last_change":"2024-03-19T03:30:35.650681+0000","last_active":"2024-03-19T03:31:39.437823+0000","last_peered":"2024-03-19T03:31:39.437823+0000","last_clean":"2024-03-19T03:31:39.437823+0000","last_became_active":"2024-03-19T03:30:35.650324+0000","last_became_peered":"2024-03-19T03:30:35.650324+0000","last_unstale":"2024-03-19T03:31:39.437823+0000","last_undegraded":"2024-03-19T03:31:39.437823+0000","last_fullsized":"2024-03-19T03:31:39.437823+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:48:23.413530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.424504+0000","last_change":"2024-03-19T03:30:35.650240+0000","last_active":"2024-03-19T03:31:39.424504+0000","last_peered":"2024-03-19T03:31:39.424504+0000","last_clean":"2024-03-19T03:31:39.424504+0000","last_became_active":"2024-03-19T03:30:35.649888+0000","last_became_peered":"2024-03-19T03:30:35.649888+0000","last_unstale":"2024-03-19T03:31:39.424504+0000","last_undegraded":"2024-03-19T03:31:39.424504+0000","last_fullsized":"2024-03-19T03:31:39.424504+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:55:30.110289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426913+0000","last_change":"2024-03-19T03:30:35.657058+0000","last_active":"2024-03-19T03:31:39.426913+0000","last_peered":"2024-03-19T03:31:39.426913+0000","last_clean":"2024-03-19T03:31:39.426913+0000","last_became_active":"2024-03-19T03:30:35.656678+0000","last_became_peered":"2024-03-19T03:30:35.656678+0000","last_unstale":"2024-03-19T03:31:39.426913+0000","last_undegraded":"2024-03-19T03:31:39.426913+0000","last_fullsized":"2024-03-19T03:31:39.426913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:10.211852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.423141+0000","last_change":"2024-03-19T03:30:35.651543+0000","last_active":"2024-03-19T03:31:39.423141+0000","last_peered":"2024-03-19T03:31:39.423141+0000","last_clean":"2024-03-19T03:31:39.423141+0000","last_became_active":"2024-03-19T03:30:35.649514+0000","last_became_peered":"2024-03-19T03:30:35.649514+0000","last_unstale":"2024-03-19T03:31:39.423141+0000","last_undegraded":"2024-03-19T03:31:39.423141+0000","last_fullsized":"2024-03-19T03:31:39.423141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:54:37.607873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425739+0000","last_change":"2024-03-19T03:30:35.662494+0000","last_active":"2024-03-19T03:31:39.425739+0000","last_peered":"2024-03-19T03:31:39.425739+0000","last_clean":"2024-03-19T03:31:39.425739+0000","last_became_active":"2024-03-19T03:30:35.660596+0000","last_became_peered":"2024-03-19T03:30:35.660596+0000","last_unstale":"2024-03-19T03:31:39.425739+0000","last_undegraded":"2024-03-19T03:31:39.425739+0000","last_fullsized":"2024-03-19T03:31:39.425739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:25:59.184025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.433898+0000","last_change":"2024-03-19T03:30:35.675751+0000","last_active":"2024-03-19T03:31:39.433898+0000","last_peered":"2024-03-19T03:31:39.433898+0000","last_clean":"2024-03-19T03:31:39.433898+0000","last_became_active":"2024-03-19T03:30:35.674313+0000","last_became_peered":"2024-03-19T03:30:35.674313+0000","last_unstale":"2024-03-19T03:31:39.433898+0000","last_undegraded":"2024-03-19T03:31:39.433898+0000","last_fullsized":"2024-03-19T03:31:39.433898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:26:26.433958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.428301+0000","last_change":"2024-03-19T03:30:35.672628+0000","last_active":"2024-03-19T03:31:39.428301+0000","last_peered":"2024-03-19T03:31:39.428301+0000","last_clean":"2024-03-19T03:31:39.428301+0000","last_became_active":"2024-03-19T03:30:35.672246+0000","last_became_peered":"2024-03-19T03:30:35.672246+0000","last_unstale":"2024-03-19T03:31:39.428301+0000","last_undegraded":"2024-03-19T03:31:39.428301+0000","last_fullsized":"2024-03-19T03:31:39.428301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:07:59.891108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.437160+0000","last_change":"2024-03-19T03:30:35.678718+0000","last_active":"2024-03-19T03:31:39.437160+0000","last_peered":"2024-03-19T03:31:39.437160+0000","last_clean":"2024-03-19T03:31:39.437160+0000","last_became_active":"2024-03-19T03:30:35.678461+0000","last_became_peered":"2024-03-19T03:30:35.678461+0000","last_unstale":"2024-03-19T03:31:39.437160+0000","last_undegraded":"2024-03-19T03:31:39.437160+0000","last_fullsized":"2024-03-19T03:31:39.437160+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:19:39.079048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426980+0000","last_change":"2024-03-19T03:30:35.675847+0000","last_active":"2024-03-19T03:31:39.426980+0000","last_peered":"2024-03-19T03:31:39.426980+0000","last_clean":"2024-03-19T03:31:39.426980+0000","last_became_active":"2024-03-19T03:30:35.675204+0000","last_became_peered":"2024-03-19T03:30:35.675204+0000","last_unstale":"2024-03-19T03:31:39.426980+0000","last_undegraded":"2024-03-19T03:31:39.426980+0000","last_fullsized":"2024-03-19T03:31:39.426980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:08.732536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.428252+0000","last_change":"2024-03-19T03:30:35.663110+0000","last_active":"2024-03-19T03:31:39.428252+0000","last_peered":"2024-03-19T03:31:39.428252+0000","last_clean":"2024-03-19T03:31:39.428252+0000","last_became_active":"2024-03-19T03:30:35.658376+0000","last_became_peered":"2024-03-19T03:30:35.658376+0000","last_unstale":"2024-03-19T03:31:39.428252+0000","last_undegraded":"2024-03-19T03:31:39.428252+0000","last_fullsized":"2024-03-19T03:31:39.428252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:29:44.658681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421561+0000","last_change":"2024-03-19T03:30:35.661154+0000","last_active":"2024-03-19T03:31:39.421561+0000","last_peered":"2024-03-19T03:31:39.421561+0000","last_clean":"2024-03-19T03:31:39.421561+0000","last_became_active":"2024-03-19T03:30:35.660794+0000","last_became_peered":"2024-03-19T03:30:35.660794+0000","last_unstale":"2024-03-19T03:31:39.421561+0000","last_undegraded":"2024-03-19T03:31:39.421561+0000","last_fullsized":"2024-03-19T03:31:39.421561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:26:29.803827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.420977+0000","last_change":"2024-03-19T03:30:39.735971+0000","last_active":"2024-03-19T03:31:39.420977+0000","last_peered":"2024-03-19T03:31:39.420977+0000","last_clean":"2024-03-19T03:31:39.420977+0000","last_became_active":"2024-03-19T03:30:39.735585+0000","last_became_peered":"2024-03-19T03:30:39.735585+0000","last_unstale":"2024-03-19T03:31:39.420977+0000","last_undegraded":"2024-03-19T03:31:39.420977+0000","last_fullsized":"2024-03-19T03:31:39.420977+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:30:15.969148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":117,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:32:24.133723+0000","last_change":"2024-03-19T03:30:34.632177+0000","last_active":"2024-03-19T03:32:24.133723+0000","last_peered":"2024-03-19T03:32:24.133723+0000","last_clean":"2024-03-19T03:32:24.133723+0000","last_became_active":"2024-03-19T03:30:34.631836+0000","last_became_peered":"2024-03-19T03:30:34.631836+0000","last_unstale":"2024-03-19T03:32:24.133723+0000","last_undegraded":"2024-03-19T03:32:24.133723+0000","last_fullsized":"2024-03-19T03:32:24.133723+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:15:42.883089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":77,"num_read_kb":77,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.435242+0000","last_change":"2024-03-19T03:31:39.433509+0000","last_active":"2024-03-19T03:31:39.435242+0000","last_peered":"2024-03-19T03:31:39.435242+0000","last_clean":"2024-03-19T03:31:39.435242+0000","last_became_active":"2024-03-19T03:31:39.431889+0000","last_became_peered":"2024-03-19T03:31:39.431889+0000","last_unstale":"2024-03-19T03:31:39.435242+0000","last_undegraded":"2024-03-19T03:31:39.435242+0000","last_fullsized":"2024-03-19T03:31:39.435242+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:51:14.868898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.422448+0000","last_change":"2024-03-19T03:30:39.733744+0000","last_active":"2024-03-19T03:31:39.422448+0000","last_peered":"2024-03-19T03:31:39.422448+0000","last_clean":"2024-03-19T03:31:39.422448+0000","last_became_active":"2024-03-19T03:30:39.733373+0000","last_became_peered":"2024-03-19T03:30:39.733373+0000","last_unstale":"2024-03-19T03:31:39.422448+0000","last_undegraded":"2024-03-19T03:31:39.422448+0000","last_fullsized":"2024-03-19T03:31:39.422448+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:48:48.186995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.422844+0000","last_change":"2024-03-19T03:30:34.632369+0000","last_active":"2024-03-19T03:31:39.422844+0000","last_peered":"2024-03-19T03:31:39.422844+0000","last_clean":"2024-03-19T03:31:39.422844+0000","last_became_active":"2024-03-19T03:30:34.632027+0000","last_became_peered":"2024-03-19T03:30:34.632027+0000","last_unstale":"2024-03-19T03:31:39.422844+0000","last_undegraded":"2024-03-19T03:31:39.422844+0000","last_fullsized":"2024-03-19T03:31:39.422844+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:04:22.575864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.437652+0000","last_change":"2024-03-19T03:30:35.677250+0000","last_active":"2024-03-19T03:31:39.437652+0000","last_peered":"2024-03-19T03:31:39.437652+0000","last_clean":"2024-03-19T03:31:39.437652+0000","last_became_active":"2024-03-19T03:30:35.676741+0000","last_became_peered":"2024-03-19T03:30:35.676741+0000","last_unstale":"2024-03-19T03:31:39.437652+0000","last_undegraded":"2024-03-19T03:31:39.437652+0000","last_fullsized":"2024-03-19T03:31:39.437652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:52:06.095738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.422804+0000","last_change":"2024-03-19T03:30:39.733501+0000","last_active":"2024-03-19T03:31:39.422804+0000","last_peered":"2024-03-19T03:31:39.422804+0000","last_clean":"2024-03-19T03:31:39.422804+0000","last_became_active":"2024-03-19T03:30:39.733102+0000","last_became_peered":"2024-03-19T03:30:39.733102+0000","last_unstale":"2024-03-19T03:31:39.422804+0000","last_undegraded":"2024-03-19T03:31:39.422804+0000","last_fullsized":"2024-03-19T03:31:39.422804+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:35:06.004381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425441+0000","last_change":"2024-03-19T03:30:34.627192+0000","last_active":"2024-03-19T03:31:39.425441+0000","last_peered":"2024-03-19T03:31:39.425441+0000","last_clean":"2024-03-19T03:31:39.425441+0000","last_became_active":"2024-03-19T03:30:34.626730+0000","last_became_peered":"2024-03-19T03:30:34.626730+0000","last_unstale":"2024-03-19T03:31:39.425441+0000","last_undegraded":"2024-03-19T03:31:39.425441+0000","last_fullsized":"2024-03-19T03:31:39.425441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:20:25.816500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.437424+0000","last_change":"2024-03-19T03:30:35.663783+0000","last_active":"2024-03-19T03:31:39.437424+0000","last_peered":"2024-03-19T03:31:39.437424+0000","last_clean":"2024-03-19T03:31:39.437424+0000","last_became_active":"2024-03-19T03:30:35.663472+0000","last_became_peered":"2024-03-19T03:30:35.663472+0000","last_unstale":"2024-03-19T03:31:39.437424+0000","last_undegraded":"2024-03-19T03:31:39.437424+0000","last_fullsized":"2024-03-19T03:31:39.437424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:51:15.211113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.422588+0000","last_change":"2024-03-19T03:30:39.731492+0000","last_active":"2024-03-19T03:31:39.422588+0000","last_peered":"2024-03-19T03:31:39.422588+0000","last_clean":"2024-03-19T03:31:39.422588+0000","last_became_active":"2024-03-19T03:30:39.731168+0000","last_became_peered":"2024-03-19T03:30:39.731168+0000","last_unstale":"2024-03-19T03:31:39.422588+0000","last_undegraded":"2024-03-19T03:31:39.422588+0000","last_fullsized":"2024-03-19T03:31:39.422588+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:11:48.281607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425834+0000","last_change":"2024-03-19T03:30:34.627589+0000","last_active":"2024-03-19T03:31:39.425834+0000","last_peered":"2024-03-19T03:31:39.425834+0000","last_clean":"2024-03-19T03:31:39.425834+0000","last_became_active":"2024-03-19T03:30:34.627229+0000","last_became_peered":"2024-03-19T03:30:34.627229+0000","last_unstale":"2024-03-19T03:31:39.425834+0000","last_undegraded":"2024-03-19T03:31:39.425834+0000","last_fullsized":"2024-03-19T03:31:39.425834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:53:23.128932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.438748+0000","last_change":"2024-03-19T03:31:39.438748+0000","last_active":"2024-03-19T03:31:39.438748+0000","last_peered":"2024-03-19T03:31:39.438748+0000","last_clean":"2024-03-19T03:31:39.438748+0000","last_became_active":"2024-03-19T03:31:39.438163+0000","last_became_peered":"2024-03-19T03:31:39.438163+0000","last_unstale":"2024-03-19T03:31:39.438748+0000","last_undegraded":"2024-03-19T03:31:39.438748+0000","last_fullsized":"2024-03-19T03:31:39.438748+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:01.537442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421281+0000","last_change":"2024-03-19T03:30:39.727351+0000","last_active":"2024-03-19T03:31:39.421281+0000","last_peered":"2024-03-19T03:31:39.421281+0000","last_clean":"2024-03-19T03:31:39.421281+0000","last_became_active":"2024-03-19T03:30:39.726868+0000","last_became_peered":"2024-03-19T03:30:39.726868+0000","last_unstale":"2024-03-19T03:31:39.421281+0000","last_undegraded":"2024-03-19T03:31:39.421281+0000","last_fullsized":"2024-03-19T03:31:39.421281+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:40:59.003064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":111,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:32:24.097758+0000","last_change":"2024-03-19T03:30:34.623911+0000","last_active":"2024-03-19T03:32:24.097758+0000","last_peered":"2024-03-19T03:32:24.097758+0000","last_clean":"2024-03-19T03:32:24.097758+0000","last_became_active":"2024-03-19T03:30:34.623548+0000","last_became_peered":"2024-03-19T03:30:34.623548+0000","last_unstale":"2024-03-19T03:32:24.097758+0000","last_undegraded":"2024-03-19T03:32:24.097758+0000","last_fullsized":"2024-03-19T03:32:24.097758+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:29:56.616353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425315+0000","last_change":"2024-03-19T03:30:35.661874+0000","last_active":"2024-03-19T03:31:39.425315+0000","last_peered":"2024-03-19T03:31:39.425315+0000","last_clean":"2024-03-19T03:31:39.425315+0000","last_became_active":"2024-03-19T03:30:35.661521+0000","last_became_peered":"2024-03-19T03:30:35.661521+0000","last_unstale":"2024-03-19T03:31:39.425315+0000","last_undegraded":"2024-03-19T03:31:39.425315+0000","last_fullsized":"2024-03-19T03:31:39.425315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:19:35.017020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421192+0000","last_change":"2024-03-19T03:30:39.748197+0000","last_active":"2024-03-19T03:31:39.421192+0000","last_peered":"2024-03-19T03:31:39.421192+0000","last_clean":"2024-03-19T03:31:39.421192+0000","last_became_active":"2024-03-19T03:30:39.747833+0000","last_became_peered":"2024-03-19T03:30:39.747833+0000","last_unstale":"2024-03-19T03:31:39.421192+0000","last_undegraded":"2024-03-19T03:31:39.421192+0000","last_fullsized":"2024-03-19T03:31:39.421192+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:29:19.213403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.423728+0000","last_change":"2024-03-19T03:30:34.623491+0000","last_active":"2024-03-19T03:31:39.423728+0000","last_peered":"2024-03-19T03:31:39.423728+0000","last_clean":"2024-03-19T03:31:39.423728+0000","last_became_active":"2024-03-19T03:30:34.623152+0000","last_became_peered":"2024-03-19T03:30:34.623152+0000","last_unstale":"2024-03-19T03:31:39.423728+0000","last_undegraded":"2024-03-19T03:31:39.423728+0000","last_fullsized":"2024-03-19T03:31:39.423728+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:00:25.530926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.427403+0000","last_change":"2024-03-19T03:30:35.651528+0000","last_active":"2024-03-19T03:31:39.427403+0000","last_peered":"2024-03-19T03:31:39.427403+0000","last_clean":"2024-03-19T03:31:39.427403+0000","last_became_active":"2024-03-19T03:30:35.650240+0000","last_became_peered":"2024-03-19T03:30:35.650240+0000","last_unstale":"2024-03-19T03:31:39.427403+0000","last_undegraded":"2024-03-19T03:31:39.427403+0000","last_fullsized":"2024-03-19T03:31:39.427403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:25.123570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426848+0000","last_change":"2024-03-19T03:30:39.740630+0000","last_active":"2024-03-19T03:31:39.426848+0000","last_peered":"2024-03-19T03:31:39.426848+0000","last_clean":"2024-03-19T03:31:39.426848+0000","last_became_active":"2024-03-19T03:30:39.740170+0000","last_became_peered":"2024-03-19T03:30:39.740170+0000","last_unstale":"2024-03-19T03:31:39.426848+0000","last_undegraded":"2024-03-19T03:31:39.426848+0000","last_fullsized":"2024-03-19T03:31:39.426848+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:23:49.415718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425434+0000","last_change":"2024-03-19T03:30:34.635173+0000","last_active":"2024-03-19T03:31:39.425434+0000","last_peered":"2024-03-19T03:31:39.425434+0000","last_clean":"2024-03-19T03:31:39.425434+0000","last_became_active":"2024-03-19T03:30:34.633809+0000","last_became_peered":"2024-03-19T03:30:34.633809+0000","last_unstale":"2024-03-19T03:31:39.425434+0000","last_undegraded":"2024-03-19T03:31:39.425434+0000","last_fullsized":"2024-03-19T03:31:39.425434+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:03:57.409707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.420821+0000","last_change":"2024-03-19T03:30:35.665660+0000","last_active":"2024-03-19T03:31:39.420821+0000","last_peered":"2024-03-19T03:31:39.420821+0000","last_clean":"2024-03-19T03:31:39.420821+0000","last_became_active":"2024-03-19T03:30:35.665290+0000","last_became_peered":"2024-03-19T03:30:35.665290+0000","last_unstale":"2024-03-19T03:31:39.420821+0000","last_undegraded":"2024-03-19T03:31:39.420821+0000","last_fullsized":"2024-03-19T03:31:39.420821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:20.333009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.435292+0000","last_change":"2024-03-19T03:30:39.744391+0000","last_active":"2024-03-19T03:31:39.435292+0000","last_peered":"2024-03-19T03:31:39.435292+0000","last_clean":"2024-03-19T03:31:39.435292+0000","last_became_active":"2024-03-19T03:30:39.744000+0000","last_became_peered":"2024-03-19T03:30:39.744000+0000","last_unstale":"2024-03-19T03:31:39.435292+0000","last_undegraded":"2024-03-19T03:31:39.435292+0000","last_fullsized":"2024-03-19T03:31:39.435292+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:17:06.124286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425903+0000","last_change":"2024-03-19T03:30:34.621811+0000","last_active":"2024-03-19T03:31:39.425903+0000","last_peered":"2024-03-19T03:31:39.425903+0000","last_clean":"2024-03-19T03:31:39.425903+0000","last_became_active":"2024-03-19T03:30:34.621308+0000","last_became_peered":"2024-03-19T03:30:34.621308+0000","last_unstale":"2024-03-19T03:31:39.425903+0000","last_undegraded":"2024-03-19T03:31:39.425903+0000","last_fullsized":"2024-03-19T03:31:39.425903+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:36:45.895720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.429510+0000","last_change":"2024-03-19T03:30:35.673683+0000","last_active":"2024-03-19T03:31:39.429510+0000","last_peered":"2024-03-19T03:31:39.429510+0000","last_clean":"2024-03-19T03:31:39.429510+0000","last_became_active":"2024-03-19T03:30:35.670489+0000","last_became_peered":"2024-03-19T03:30:35.670489+0000","last_unstale":"2024-03-19T03:31:39.429510+0000","last_undegraded":"2024-03-19T03:31:39.429510+0000","last_fullsized":"2024-03-19T03:31:39.429510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:48:30.521454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421010+0000","last_change":"2024-03-19T03:30:39.726358+0000","last_active":"2024-03-19T03:31:39.421010+0000","last_peered":"2024-03-19T03:31:39.421010+0000","last_clean":"2024-03-19T03:31:39.421010+0000","last_became_active":"2024-03-19T03:30:39.725720+0000","last_became_peered":"2024-03-19T03:30:39.725720+0000","last_unstale":"2024-03-19T03:31:39.421010+0000","last_undegraded":"2024-03-19T03:31:39.421010+0000","last_fullsized":"2024-03-19T03:31:39.421010+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:01:08.759794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":191,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:32:28.102314+0000","last_change":"2024-03-19T03:30:34.626336+0000","last_active":"2024-03-19T03:32:28.102314+0000","last_peered":"2024-03-19T03:32:28.102314+0000","last_clean":"2024-03-19T03:32:28.102314+0000","last_became_active":"2024-03-19T03:30:34.625991+0000","last_became_peered":"2024-03-19T03:30:34.625991+0000","last_unstale":"2024-03-19T03:32:28.102314+0000","last_undegraded":"2024-03-19T03:32:28.102314+0000","last_fullsized":"2024-03-19T03:32:28.102314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:40:58.791759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":160,"num_read_kb":160,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.437787+0000","last_change":"2024-03-19T03:30:35.667001+0000","last_active":"2024-03-19T03:31:39.437787+0000","last_peered":"2024-03-19T03:31:39.437787+0000","last_clean":"2024-03-19T03:31:39.437787+0000","last_became_active":"2024-03-19T03:30:35.666586+0000","last_became_peered":"2024-03-19T03:30:35.666586+0000","last_unstale":"2024-03-19T03:31:39.437787+0000","last_undegraded":"2024-03-19T03:31:39.437787+0000","last_fullsized":"2024-03-19T03:31:39.437787+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:40:30.100860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.428857+0000","last_change":"2024-03-19T03:30:39.745267+0000","last_active":"2024-03-19T03:31:39.428857+0000","last_peered":"2024-03-19T03:31:39.428857+0000","last_clean":"2024-03-19T03:31:39.428857+0000","last_became_active":"2024-03-19T03:30:39.743434+0000","last_became_peered":"2024-03-19T03:30:39.743434+0000","last_unstale":"2024-03-19T03:31:39.428857+0000","last_undegraded":"2024-03-19T03:31:39.428857+0000","last_fullsized":"2024-03-19T03:31:39.428857+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:53:37.583886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426670+0000","last_change":"2024-03-19T03:30:34.617769+0000","last_active":"2024-03-19T03:31:39.426670+0000","last_peered":"2024-03-19T03:31:39.426670+0000","last_clean":"2024-03-19T03:31:39.426670+0000","last_became_active":"2024-03-19T03:30:34.617428+0000","last_became_peered":"2024-03-19T03:30:34.617428+0000","last_unstale":"2024-03-19T03:31:39.426670+0000","last_undegraded":"2024-03-19T03:31:39.426670+0000","last_fullsized":"2024-03-19T03:31:39.426670+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:58:09.181320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.440189+0000","last_change":"2024-03-19T03:31:39.440189+0000","last_active":"2024-03-19T03:31:39.440189+0000","last_peered":"2024-03-19T03:31:39.440189+0000","last_clean":"2024-03-19T03:31:39.440189+0000","last_became_active":"2024-03-19T03:31:39.439752+0000","last_became_peered":"2024-03-19T03:31:39.439752+0000","last_unstale":"2024-03-19T03:31:39.440189+0000","last_undegraded":"2024-03-19T03:31:39.440189+0000","last_fullsized":"2024-03-19T03:31:39.440189+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:45.028679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.422928+0000","last_change":"2024-03-19T03:30:39.733273+0000","last_active":"2024-03-19T03:31:39.422928+0000","last_peered":"2024-03-19T03:31:39.422928+0000","last_clean":"2024-03-19T03:31:39.422928+0000","last_became_active":"2024-03-19T03:30:39.732885+0000","last_became_peered":"2024-03-19T03:30:39.732885+0000","last_unstale":"2024-03-19T03:31:39.422928+0000","last_undegraded":"2024-03-19T03:31:39.422928+0000","last_fullsized":"2024-03-19T03:31:39.422928+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:48:10.446577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.427841+0000","last_change":"2024-03-19T03:30:34.637076+0000","last_active":"2024-03-19T03:31:39.427841+0000","last_peered":"2024-03-19T03:31:39.427841+0000","last_clean":"2024-03-19T03:31:39.427841+0000","last_became_active":"2024-03-19T03:30:34.636642+0000","last_became_peered":"2024-03-19T03:30:34.636642+0000","last_unstale":"2024-03-19T03:31:39.427841+0000","last_undegraded":"2024-03-19T03:31:39.427841+0000","last_fullsized":"2024-03-19T03:31:39.427841+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:26:17.494081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.437546+0000","last_change":"2024-03-19T03:30:35.651739+0000","last_active":"2024-03-19T03:31:39.437546+0000","last_peered":"2024-03-19T03:31:39.437546+0000","last_clean":"2024-03-19T03:31:39.437546+0000","last_became_active":"2024-03-19T03:30:35.651419+0000","last_became_peered":"2024-03-19T03:30:35.651419+0000","last_unstale":"2024-03-19T03:31:39.437546+0000","last_undegraded":"2024-03-19T03:31:39.437546+0000","last_fullsized":"2024-03-19T03:31:39.437546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T15:17:55.022050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426700+0000","last_change":"2024-03-19T03:30:39.733228+0000","last_active":"2024-03-19T03:31:39.426700+0000","last_peered":"2024-03-19T03:31:39.426700+0000","last_clean":"2024-03-19T03:31:39.426700+0000","last_became_active":"2024-03-19T03:30:39.732819+0000","last_became_peered":"2024-03-19T03:30:39.732819+0000","last_unstale":"2024-03-19T03:31:39.426700+0000","last_undegraded":"2024-03-19T03:31:39.426700+0000","last_fullsized":"2024-03-19T03:31:39.426700+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:03:43.968209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'35","reported_seq":230,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:32:28.102314+0000","last_change":"2024-03-19T03:30:34.618323+0000","last_active":"2024-03-19T03:32:28.102314+0000","last_peered":"2024-03-19T03:32:28.102314+0000","last_clean":"2024-03-19T03:32:28.102314+0000","last_became_active":"2024-03-19T03:30:34.617975+0000","last_became_peered":"2024-03-19T03:30:34.617975+0000","last_unstale":"2024-03-19T03:32:28.102314+0000","last_undegraded":"2024-03-19T03:32:28.102314+0000","last_fullsized":"2024-03-19T03:32:28.102314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:37:15.261998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64206,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":168,"num_read_kb":147,"num_write":37,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425662+0000","last_change":"2024-03-19T03:30:35.652754+0000","last_active":"2024-03-19T03:31:39.425662+0000","last_peered":"2024-03-19T03:31:39.425662+0000","last_clean":"2024-03-19T03:31:39.425662+0000","last_became_active":"2024-03-19T03:30:35.652415+0000","last_became_peered":"2024-03-19T03:30:35.652415+0000","last_unstale":"2024-03-19T03:31:39.425662+0000","last_undegraded":"2024-03-19T03:31:39.425662+0000","last_fullsized":"2024-03-19T03:31:39.425662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:29:31.838302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.428363+0000","last_change":"2024-03-19T03:30:39.732885+0000","last_active":"2024-03-19T03:31:39.428363+0000","last_peered":"2024-03-19T03:31:39.428363+0000","last_clean":"2024-03-19T03:31:39.428363+0000","last_became_active":"2024-03-19T03:30:39.732449+0000","last_became_peered":"2024-03-19T03:30:39.732449+0000","last_unstale":"2024-03-19T03:31:39.428363+0000","last_undegraded":"2024-03-19T03:31:39.428363+0000","last_fullsized":"2024-03-19T03:31:39.428363+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:08:01.304805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421344+0000","last_change":"2024-03-19T03:30:34.617560+0000","last_active":"2024-03-19T03:31:39.421344+0000","last_peered":"2024-03-19T03:31:39.421344+0000","last_clean":"2024-03-19T03:31:39.421344+0000","last_became_active":"2024-03-19T03:30:34.617234+0000","last_became_peered":"2024-03-19T03:30:34.617234+0000","last_unstale":"2024-03-19T03:31:39.421344+0000","last_undegraded":"2024-03-19T03:31:39.421344+0000","last_fullsized":"2024-03-19T03:31:39.421344+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:00:46.206589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425703+0000","last_change":"2024-03-19T03:30:35.672723+0000","last_active":"2024-03-19T03:31:39.425703+0000","last_peered":"2024-03-19T03:31:39.425703+0000","last_clean":"2024-03-19T03:31:39.425703+0000","last_became_active":"2024-03-19T03:30:35.670828+0000","last_became_peered":"2024-03-19T03:30:35.670828+0000","last_unstale":"2024-03-19T03:31:39.425703+0000","last_undegraded":"2024-03-19T03:31:39.425703+0000","last_fullsized":"2024-03-19T03:31:39.425703+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:22:19.254329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426674+0000","last_change":"2024-03-19T03:30:39.738887+0000","last_active":"2024-03-19T03:31:39.426674+0000","last_peered":"2024-03-19T03:31:39.426674+0000","last_clean":"2024-03-19T03:31:39.426674+0000","last_became_active":"2024-03-19T03:30:39.738450+0000","last_became_peered":"2024-03-19T03:30:39.738450+0000","last_unstale":"2024-03-19T03:31:39.426674+0000","last_undegraded":"2024-03-19T03:31:39.426674+0000","last_fullsized":"2024-03-19T03:31:39.426674+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:23:57.550464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426018+0000","last_change":"2024-03-19T03:30:34.634810+0000","last_active":"2024-03-19T03:31:39.426018+0000","last_peered":"2024-03-19T03:31:39.426018+0000","last_clean":"2024-03-19T03:31:39.426018+0000","last_became_active":"2024-03-19T03:30:34.634404+0000","last_became_peered":"2024-03-19T03:30:34.634404+0000","last_unstale":"2024-03-19T03:31:39.426018+0000","last_undegraded":"2024-03-19T03:31:39.426018+0000","last_fullsized":"2024-03-19T03:31:39.426018+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:19:09.231175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426504+0000","last_change":"2024-03-19T03:30:35.664808+0000","last_active":"2024-03-19T03:31:39.426504+0000","last_peered":"2024-03-19T03:31:39.426504+0000","last_clean":"2024-03-19T03:31:39.426504+0000","last_became_active":"2024-03-19T03:30:35.664431+0000","last_became_peered":"2024-03-19T03:30:35.664431+0000","last_unstale":"2024-03-19T03:31:39.426504+0000","last_undegraded":"2024-03-19T03:31:39.426504+0000","last_fullsized":"2024-03-19T03:31:39.426504+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:18:47.623528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.441795+0000","last_change":"2024-03-19T03:31:39.441384+0000","last_active":"2024-03-19T03:31:39.441795+0000","last_peered":"2024-03-19T03:31:39.441795+0000","last_clean":"2024-03-19T03:31:39.441795+0000","last_became_active":"2024-03-19T03:31:39.440947+0000","last_became_peered":"2024-03-19T03:31:39.440947+0000","last_unstale":"2024-03-19T03:31:39.441795+0000","last_undegraded":"2024-03-19T03:31:39.441795+0000","last_fullsized":"2024-03-19T03:31:39.441795+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:09:33.041934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":113,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:32:25.101683+0000","last_change":"2024-03-19T03:30:34.619890+0000","last_active":"2024-03-19T03:32:25.101683+0000","last_peered":"2024-03-19T03:32:25.101683+0000","last_clean":"2024-03-19T03:32:25.101683+0000","last_became_active":"2024-03-19T03:30:34.619518+0000","last_became_peered":"2024-03-19T03:30:34.619518+0000","last_unstale":"2024-03-19T03:32:25.101683+0000","last_undegraded":"2024-03-19T03:32:25.101683+0000","last_fullsized":"2024-03-19T03:32:25.101683+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T15:19:41.314321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":81,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426265+0000","last_change":"2024-03-19T03:30:35.649241+0000","last_active":"2024-03-19T03:31:39.426265+0000","last_peered":"2024-03-19T03:31:39.426265+0000","last_clean":"2024-03-19T03:31:39.426265+0000","last_became_active":"2024-03-19T03:30:35.648883+0000","last_became_peered":"2024-03-19T03:30:35.648883+0000","last_unstale":"2024-03-19T03:31:39.426265+0000","last_undegraded":"2024-03-19T03:31:39.426265+0000","last_fullsized":"2024-03-19T03:31:39.426265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:43:44.219322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.436929+0000","last_change":"2024-03-19T03:31:39.436929+0000","last_active":"2024-03-19T03:31:39.436929+0000","last_peered":"2024-03-19T03:31:39.436929+0000","last_clean":"2024-03-19T03:31:39.436929+0000","last_became_active":"2024-03-19T03:31:39.436418+0000","last_became_peered":"2024-03-19T03:31:39.436418+0000","last_unstale":"2024-03-19T03:31:39.436929+0000","last_undegraded":"2024-03-19T03:31:39.436929+0000","last_fullsized":"2024-03-19T03:31:39.436929+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:49:49.767632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426114+0000","last_change":"2024-03-19T03:30:34.635076+0000","last_active":"2024-03-19T03:31:39.426114+0000","last_peered":"2024-03-19T03:31:39.426114+0000","last_clean":"2024-03-19T03:31:39.426114+0000","last_became_active":"2024-03-19T03:30:34.634680+0000","last_became_peered":"2024-03-19T03:30:34.634680+0000","last_unstale":"2024-03-19T03:31:39.426114+0000","last_undegraded":"2024-03-19T03:31:39.426114+0000","last_fullsized":"2024-03-19T03:31:39.426114+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:16:38.874086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425553+0000","last_change":"2024-03-19T03:30:35.651912+0000","last_active":"2024-03-19T03:31:39.425553+0000","last_peered":"2024-03-19T03:31:39.425553+0000","last_clean":"2024-03-19T03:31:39.425553+0000","last_became_active":"2024-03-19T03:30:35.651563+0000","last_became_peered":"2024-03-19T03:30:35.651563+0000","last_unstale":"2024-03-19T03:31:39.425553+0000","last_undegraded":"2024-03-19T03:31:39.425553+0000","last_fullsized":"2024-03-19T03:31:39.425553+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:41:05.677230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425439+0000","last_change":"2024-03-19T03:30:39.737466+0000","last_active":"2024-03-19T03:31:39.425439+0000","last_peered":"2024-03-19T03:31:39.425439+0000","last_clean":"2024-03-19T03:31:39.425439+0000","last_became_active":"2024-03-19T03:30:39.737126+0000","last_became_peered":"2024-03-19T03:30:39.737126+0000","last_unstale":"2024-03-19T03:31:39.425439+0000","last_undegraded":"2024-03-19T03:31:39.425439+0000","last_fullsized":"2024-03-19T03:31:39.425439+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:52:25.030107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426740+0000","last_change":"2024-03-19T03:30:34.622729+0000","last_active":"2024-03-19T03:31:39.426740+0000","last_peered":"2024-03-19T03:31:39.426740+0000","last_clean":"2024-03-19T03:31:39.426740+0000","last_became_active":"2024-03-19T03:30:34.622349+0000","last_became_peered":"2024-03-19T03:30:34.622349+0000","last_unstale":"2024-03-19T03:31:39.426740+0000","last_undegraded":"2024-03-19T03:31:39.426740+0000","last_fullsized":"2024-03-19T03:31:39.426740+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:28.893702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.438224+0000","last_change":"2024-03-19T03:31:39.438224+0000","last_active":"2024-03-19T03:31:39.438224+0000","last_peered":"2024-03-19T03:31:39.438224+0000","last_clean":"2024-03-19T03:31:39.438224+0000","last_became_active":"2024-03-19T03:31:39.437764+0000","last_became_peered":"2024-03-19T03:31:39.437764+0000","last_unstale":"2024-03-19T03:31:39.438224+0000","last_undegraded":"2024-03-19T03:31:39.438224+0000","last_fullsized":"2024-03-19T03:31:39.438224+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:55.306306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425199+0000","last_change":"2024-03-19T03:30:39.728505+0000","last_active":"2024-03-19T03:31:39.425199+0000","last_peered":"2024-03-19T03:31:39.425199+0000","last_clean":"2024-03-19T03:31:39.425199+0000","last_became_active":"2024-03-19T03:30:39.728095+0000","last_became_peered":"2024-03-19T03:30:39.728095+0000","last_unstale":"2024-03-19T03:31:39.425199+0000","last_undegraded":"2024-03-19T03:31:39.425199+0000","last_fullsized":"2024-03-19T03:31:39.425199+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:59:44.189992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.427419+0000","last_change":"2024-03-19T03:30:34.642738+0000","last_active":"2024-03-19T03:31:39.427419+0000","last_peered":"2024-03-19T03:31:39.427419+0000","last_clean":"2024-03-19T03:31:39.427419+0000","last_became_active":"2024-03-19T03:30:34.642173+0000","last_became_peered":"2024-03-19T03:30:34.642173+0000","last_unstale":"2024-03-19T03:31:39.427419+0000","last_undegraded":"2024-03-19T03:31:39.427419+0000","last_fullsized":"2024-03-19T03:31:39.427419+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:17:51.633284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.424564+0000","last_change":"2024-03-19T03:30:35.663676+0000","last_active":"2024-03-19T03:31:39.424564+0000","last_peered":"2024-03-19T03:31:39.424564+0000","last_clean":"2024-03-19T03:31:39.424564+0000","last_became_active":"2024-03-19T03:30:35.663313+0000","last_became_peered":"2024-03-19T03:30:35.663313+0000","last_unstale":"2024-03-19T03:31:39.424564+0000","last_undegraded":"2024-03-19T03:31:39.424564+0000","last_fullsized":"2024-03-19T03:31:39.424564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:02:25.933784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425394+0000","last_change":"2024-03-19T03:30:39.744967+0000","last_active":"2024-03-19T03:31:39.425394+0000","last_peered":"2024-03-19T03:31:39.425394+0000","last_clean":"2024-03-19T03:31:39.425394+0000","last_became_active":"2024-03-19T03:30:39.744542+0000","last_became_peered":"2024-03-19T03:30:39.744542+0000","last_unstale":"2024-03-19T03:31:39.425394+0000","last_undegraded":"2024-03-19T03:31:39.425394+0000","last_fullsized":"2024-03-19T03:31:39.425394+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:56:54.699749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":109,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:32:24.134170+0000","last_change":"2024-03-19T03:30:34.622225+0000","last_active":"2024-03-19T03:32:24.134170+0000","last_peered":"2024-03-19T03:32:24.134170+0000","last_clean":"2024-03-19T03:32:24.134170+0000","last_became_active":"2024-03-19T03:30:34.621836+0000","last_became_peered":"2024-03-19T03:30:34.621836+0000","last_unstale":"2024-03-19T03:32:24.134170+0000","last_undegraded":"2024-03-19T03:32:24.134170+0000","last_fullsized":"2024-03-19T03:32:24.134170+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:20:35.854632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.423796+0000","last_change":"2024-03-19T03:30:35.675961+0000","last_active":"2024-03-19T03:31:39.423796+0000","last_peered":"2024-03-19T03:31:39.423796+0000","last_clean":"2024-03-19T03:31:39.423796+0000","last_became_active":"2024-03-19T03:30:35.675603+0000","last_became_peered":"2024-03-19T03:30:35.675603+0000","last_unstale":"2024-03-19T03:31:39.423796+0000","last_undegraded":"2024-03-19T03:31:39.423796+0000","last_fullsized":"2024-03-19T03:31:39.423796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:39:47.429027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.433889+0000","last_change":"2024-03-19T03:30:39.746747+0000","last_active":"2024-03-19T03:31:39.433889+0000","last_peered":"2024-03-19T03:31:39.433889+0000","last_clean":"2024-03-19T03:31:39.433889+0000","last_became_active":"2024-03-19T03:30:39.745553+0000","last_became_peered":"2024-03-19T03:30:39.745553+0000","last_unstale":"2024-03-19T03:31:39.433889+0000","last_undegraded":"2024-03-19T03:31:39.433889+0000","last_fullsized":"2024-03-19T03:31:39.433889+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:14:56.211061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426189+0000","last_change":"2024-03-19T03:30:35.649795+0000","last_active":"2024-03-19T03:31:39.426189+0000","last_peered":"2024-03-19T03:31:39.426189+0000","last_clean":"2024-03-19T03:31:39.426189+0000","last_became_active":"2024-03-19T03:30:35.649417+0000","last_became_peered":"2024-03-19T03:30:35.649417+0000","last_unstale":"2024-03-19T03:31:39.426189+0000","last_undegraded":"2024-03-19T03:31:39.426189+0000","last_fullsized":"2024-03-19T03:31:39.426189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:24.065817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":110,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:32:24.100097+0000","last_change":"2024-03-19T03:30:34.637446+0000","last_active":"2024-03-19T03:32:24.100097+0000","last_peered":"2024-03-19T03:32:24.100097+0000","last_clean":"2024-03-19T03:32:24.100097+0000","last_became_active":"2024-03-19T03:30:34.637083+0000","last_became_peered":"2024-03-19T03:30:34.637083+0000","last_unstale":"2024-03-19T03:32:24.100097+0000","last_undegraded":"2024-03-19T03:32:24.100097+0000","last_fullsized":"2024-03-19T03:32:24.100097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:51:02.090890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":78,"num_read_kb":78,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.438484+0000","last_change":"2024-03-19T03:31:39.438484+0000","last_active":"2024-03-19T03:31:39.438484+0000","last_peered":"2024-03-19T03:31:39.438484+0000","last_clean":"2024-03-19T03:31:39.438484+0000","last_became_active":"2024-03-19T03:31:39.433824+0000","last_became_peered":"2024-03-19T03:31:39.433824+0000","last_unstale":"2024-03-19T03:31:39.438484+0000","last_undegraded":"2024-03-19T03:31:39.438484+0000","last_fullsized":"2024-03-19T03:31:39.438484+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:58:52.691327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421595+0000","last_change":"2024-03-19T03:30:35.670002+0000","last_active":"2024-03-19T03:31:39.421595+0000","last_peered":"2024-03-19T03:31:39.421595+0000","last_clean":"2024-03-19T03:31:39.421595+0000","last_became_active":"2024-03-19T03:30:35.653208+0000","last_became_peered":"2024-03-19T03:30:35.653208+0000","last_unstale":"2024-03-19T03:31:39.421595+0000","last_undegraded":"2024-03-19T03:31:39.421595+0000","last_fullsized":"2024-03-19T03:31:39.421595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:37:15.246977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.422878+0000","last_change":"2024-03-19T03:30:34.625107+0000","last_active":"2024-03-19T03:31:39.422878+0000","last_peered":"2024-03-19T03:31:39.422878+0000","last_clean":"2024-03-19T03:31:39.422878+0000","last_became_active":"2024-03-19T03:30:34.624711+0000","last_became_peered":"2024-03-19T03:30:34.624711+0000","last_unstale":"2024-03-19T03:31:39.422878+0000","last_undegraded":"2024-03-19T03:31:39.422878+0000","last_fullsized":"2024-03-19T03:31:39.422878+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:39:33.880100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":351,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426188+0000","last_change":"2024-03-19T03:29:11.374136+0000","last_active":"2024-03-19T03:31:39.426188+0000","last_peered":"2024-03-19T03:31:39.426188+0000","last_clean":"2024-03-19T03:31:39.426188+0000","last_became_active":"2024-03-19T03:29:11.373691+0000","last_became_peered":"2024-03-19T03:29:11.373691+0000","last_unstale":"2024-03-19T03:31:39.426188+0000","last_undegraded":"2024-03-19T03:31:39.426188+0000","last_fullsized":"2024-03-19T03:31:39.426188+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:25:19.274380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:25:19.274380+0000","last_clean_scrub_stamp":"2024-03-19T03:25:19.274380+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:29:59.731772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.428371+0000","last_change":"2024-03-19T03:30:39.744391+0000","last_active":"2024-03-19T03:31:39.428371+0000","last_peered":"2024-03-19T03:31:39.428371+0000","last_clean":"2024-03-19T03:31:39.428371+0000","last_became_active":"2024-03-19T03:30:39.743995+0000","last_became_peered":"2024-03-19T03:30:39.743995+0000","last_unstale":"2024-03-19T03:31:39.428371+0000","last_undegraded":"2024-03-19T03:31:39.428371+0000","last_fullsized":"2024-03-19T03:31:39.428371+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:41:02.714132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425174+0000","last_change":"2024-03-19T03:30:35.662528+0000","last_active":"2024-03-19T03:31:39.425174+0000","last_peered":"2024-03-19T03:31:39.425174+0000","last_clean":"2024-03-19T03:31:39.425174+0000","last_became_active":"2024-03-19T03:30:35.659782+0000","last_became_peered":"2024-03-19T03:30:35.659782+0000","last_unstale":"2024-03-19T03:31:39.425174+0000","last_undegraded":"2024-03-19T03:31:39.425174+0000","last_fullsized":"2024-03-19T03:31:39.425174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:18:12.838234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426195+0000","last_change":"2024-03-19T03:30:34.620988+0000","last_active":"2024-03-19T03:31:39.426195+0000","last_peered":"2024-03-19T03:31:39.426195+0000","last_clean":"2024-03-19T03:31:39.426195+0000","last_became_active":"2024-03-19T03:30:34.620411+0000","last_became_peered":"2024-03-19T03:30:34.620411+0000","last_unstale":"2024-03-19T03:31:39.426195+0000","last_undegraded":"2024-03-19T03:31:39.426195+0000","last_fullsized":"2024-03-19T03:31:39.426195+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:37:13.275121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426353+0000","last_change":"2024-03-19T03:30:39.733325+0000","last_active":"2024-03-19T03:31:39.426353+0000","last_peered":"2024-03-19T03:31:39.426353+0000","last_clean":"2024-03-19T03:31:39.426353+0000","last_became_active":"2024-03-19T03:30:39.732553+0000","last_became_peered":"2024-03-19T03:30:39.732553+0000","last_unstale":"2024-03-19T03:31:39.426353+0000","last_undegraded":"2024-03-19T03:31:39.426353+0000","last_fullsized":"2024-03-19T03:31:39.426353+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:52:47.403673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425596+0000","last_change":"2024-03-19T03:30:34.637100+0000","last_active":"2024-03-19T03:31:39.425596+0000","last_peered":"2024-03-19T03:31:39.425596+0000","last_clean":"2024-03-19T03:31:39.425596+0000","last_became_active":"2024-03-19T03:30:34.636715+0000","last_became_peered":"2024-03-19T03:30:34.636715+0000","last_unstale":"2024-03-19T03:31:39.425596+0000","last_undegraded":"2024-03-19T03:31:39.425596+0000","last_fullsized":"2024-03-19T03:31:39.425596+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:48:16.212158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426337+0000","last_change":"2024-03-19T03:30:35.668405+0000","last_active":"2024-03-19T03:31:39.426337+0000","last_peered":"2024-03-19T03:31:39.426337+0000","last_clean":"2024-03-19T03:31:39.426337+0000","last_became_active":"2024-03-19T03:30:35.667972+0000","last_became_peered":"2024-03-19T03:30:35.667972+0000","last_unstale":"2024-03-19T03:31:39.426337+0000","last_undegraded":"2024-03-19T03:31:39.426337+0000","last_fullsized":"2024-03-19T03:31:39.426337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:01:52.816110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.420806+0000","last_change":"2024-03-19T03:30:39.747535+0000","last_active":"2024-03-19T03:31:39.420806+0000","last_peered":"2024-03-19T03:31:39.420806+0000","last_clean":"2024-03-19T03:31:39.420806+0000","last_became_active":"2024-03-19T03:30:39.747122+0000","last_became_peered":"2024-03-19T03:30:39.747122+0000","last_unstale":"2024-03-19T03:31:39.420806+0000","last_undegraded":"2024-03-19T03:31:39.420806+0000","last_fullsized":"2024-03-19T03:31:39.420806+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:07:35.642809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'13","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.428383+0000","last_change":"2024-03-19T03:30:34.637768+0000","last_active":"2024-03-19T03:31:39.428383+0000","last_peered":"2024-03-19T03:31:39.428383+0000","last_clean":"2024-03-19T03:31:39.428383+0000","last_became_active":"2024-03-19T03:30:34.637431+0000","last_became_peered":"2024-03-19T03:30:34.637431+0000","last_unstale":"2024-03-19T03:31:39.428383+0000","last_undegraded":"2024-03-19T03:31:39.428383+0000","last_fullsized":"2024-03-19T03:31:39.428383+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:22:50.487209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.423453+0000","last_change":"2024-03-19T03:30:35.661352+0000","last_active":"2024-03-19T03:31:39.423453+0000","last_peered":"2024-03-19T03:31:39.423453+0000","last_clean":"2024-03-19T03:31:39.423453+0000","last_became_active":"2024-03-19T03:30:35.658656+0000","last_became_peered":"2024-03-19T03:30:35.658656+0000","last_unstale":"2024-03-19T03:31:39.423453+0000","last_undegraded":"2024-03-19T03:31:39.423453+0000","last_fullsized":"2024-03-19T03:31:39.423453+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:36.532748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.420837+0000","last_change":"2024-03-19T03:30:39.724637+0000","last_active":"2024-03-19T03:31:39.420837+0000","last_peered":"2024-03-19T03:31:39.420837+0000","last_clean":"2024-03-19T03:31:39.420837+0000","last_became_active":"2024-03-19T03:30:39.723842+0000","last_became_peered":"2024-03-19T03:30:39.723842+0000","last_unstale":"2024-03-19T03:31:39.420837+0000","last_undegraded":"2024-03-19T03:31:39.420837+0000","last_fullsized":"2024-03-19T03:31:39.420837+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:04:02.446185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.423197+0000","last_change":"2024-03-19T03:30:34.620404+0000","last_active":"2024-03-19T03:31:39.423197+0000","last_peered":"2024-03-19T03:31:39.423197+0000","last_clean":"2024-03-19T03:31:39.423197+0000","last_became_active":"2024-03-19T03:30:34.620056+0000","last_became_peered":"2024-03-19T03:30:34.620056+0000","last_unstale":"2024-03-19T03:31:39.423197+0000","last_undegraded":"2024-03-19T03:31:39.423197+0000","last_fullsized":"2024-03-19T03:31:39.423197+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:54:42.203912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.435485+0000","last_change":"2024-03-19T03:30:35.686618+0000","last_active":"2024-03-19T03:31:39.435485+0000","last_peered":"2024-03-19T03:31:39.435485+0000","last_clean":"2024-03-19T03:31:39.435485+0000","last_became_active":"2024-03-19T03:30:35.685814+0000","last_became_peered":"2024-03-19T03:30:35.685814+0000","last_unstale":"2024-03-19T03:31:39.435485+0000","last_undegraded":"2024-03-19T03:31:39.435485+0000","last_fullsized":"2024-03-19T03:31:39.435485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:08.401307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.423269+0000","last_change":"2024-03-19T03:30:39.744039+0000","last_active":"2024-03-19T03:31:39.423269+0000","last_peered":"2024-03-19T03:31:39.423269+0000","last_clean":"2024-03-19T03:31:39.423269+0000","last_became_active":"2024-03-19T03:30:39.743637+0000","last_became_peered":"2024-03-19T03:30:39.743637+0000","last_unstale":"2024-03-19T03:31:39.423269+0000","last_undegraded":"2024-03-19T03:31:39.423269+0000","last_fullsized":"2024-03-19T03:31:39.423269+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:02:22.498333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.437167+0000","last_change":"2024-03-19T03:30:34.629059+0000","last_active":"2024-03-19T03:31:39.437167+0000","last_peered":"2024-03-19T03:31:39.437167+0000","last_clean":"2024-03-19T03:31:39.437167+0000","last_became_active":"2024-03-19T03:30:34.628617+0000","last_became_peered":"2024-03-19T03:30:34.628617+0000","last_unstale":"2024-03-19T03:31:39.437167+0000","last_undegraded":"2024-03-19T03:31:39.437167+0000","last_fullsized":"2024-03-19T03:31:39.437167+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:47:09.064502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425034+0000","last_change":"2024-03-19T03:30:35.650256+0000","last_active":"2024-03-19T03:31:39.425034+0000","last_peered":"2024-03-19T03:31:39.425034+0000","last_clean":"2024-03-19T03:31:39.425034+0000","last_became_active":"2024-03-19T03:30:35.649086+0000","last_became_peered":"2024-03-19T03:30:35.649086+0000","last_unstale":"2024-03-19T03:31:39.425034+0000","last_undegraded":"2024-03-19T03:31:39.425034+0000","last_fullsized":"2024-03-19T03:31:39.425034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:00:49.204307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.429819+0000","last_change":"2024-03-19T03:31:39.429381+0000","last_active":"2024-03-19T03:31:39.429819+0000","last_peered":"2024-03-19T03:31:39.429819+0000","last_clean":"2024-03-19T03:31:39.429819+0000","last_became_active":"2024-03-19T03:31:39.428915+0000","last_became_peered":"2024-03-19T03:31:39.428915+0000","last_unstale":"2024-03-19T03:31:39.429819+0000","last_undegraded":"2024-03-19T03:31:39.429819+0000","last_fullsized":"2024-03-19T03:31:39.429819+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:20.121002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,5],"acting":[10,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421146+0000","last_change":"2024-03-19T03:30:34.634935+0000","last_active":"2024-03-19T03:31:39.421146+0000","last_peered":"2024-03-19T03:31:39.421146+0000","last_clean":"2024-03-19T03:31:39.421146+0000","last_became_active":"2024-03-19T03:30:34.634602+0000","last_became_peered":"2024-03-19T03:30:34.634602+0000","last_unstale":"2024-03-19T03:31:39.421146+0000","last_undegraded":"2024-03-19T03:31:39.421146+0000","last_fullsized":"2024-03-19T03:31:39.421146+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:16:57.158105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425439+0000","last_change":"2024-03-19T03:30:35.675650+0000","last_active":"2024-03-19T03:31:39.425439+0000","last_peered":"2024-03-19T03:31:39.425439+0000","last_clean":"2024-03-19T03:31:39.425439+0000","last_became_active":"2024-03-19T03:30:35.675255+0000","last_became_peered":"2024-03-19T03:30:35.675255+0000","last_unstale":"2024-03-19T03:31:39.425439+0000","last_undegraded":"2024-03-19T03:31:39.425439+0000","last_fullsized":"2024-03-19T03:31:39.425439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:55:03.201526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.422982+0000","last_change":"2024-03-19T03:30:39.733413+0000","last_active":"2024-03-19T03:31:39.422982+0000","last_peered":"2024-03-19T03:31:39.422982+0000","last_clean":"2024-03-19T03:31:39.422982+0000","last_became_active":"2024-03-19T03:30:39.732998+0000","last_became_peered":"2024-03-19T03:30:39.732998+0000","last_unstale":"2024-03-19T03:31:39.422982+0000","last_undegraded":"2024-03-19T03:31:39.422982+0000","last_fullsized":"2024-03-19T03:31:39.422982+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:07:39.512355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421028+0000","last_change":"2024-03-19T03:30:34.623749+0000","last_active":"2024-03-19T03:31:39.421028+0000","last_peered":"2024-03-19T03:31:39.421028+0000","last_clean":"2024-03-19T03:31:39.421028+0000","last_became_active":"2024-03-19T03:30:34.623378+0000","last_became_peered":"2024-03-19T03:30:34.623378+0000","last_unstale":"2024-03-19T03:31:39.421028+0000","last_undegraded":"2024-03-19T03:31:39.421028+0000","last_fullsized":"2024-03-19T03:31:39.421028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:06:01.939685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421693+0000","last_change":"2024-03-19T03:30:35.685702+0000","last_active":"2024-03-19T03:31:39.421693+0000","last_peered":"2024-03-19T03:31:39.421693+0000","last_clean":"2024-03-19T03:31:39.421693+0000","last_became_active":"2024-03-19T03:30:35.683340+0000","last_became_peered":"2024-03-19T03:30:35.683340+0000","last_unstale":"2024-03-19T03:31:39.421693+0000","last_undegraded":"2024-03-19T03:31:39.421693+0000","last_fullsized":"2024-03-19T03:31:39.421693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:12.246255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.422983+0000","last_change":"2024-03-19T03:30:39.726747+0000","last_active":"2024-03-19T03:31:39.422983+0000","last_peered":"2024-03-19T03:31:39.422983+0000","last_clean":"2024-03-19T03:31:39.422983+0000","last_became_active":"2024-03-19T03:30:39.726236+0000","last_became_peered":"2024-03-19T03:30:39.726236+0000","last_unstale":"2024-03-19T03:31:39.422983+0000","last_undegraded":"2024-03-19T03:31:39.422983+0000","last_fullsized":"2024-03-19T03:31:39.422983+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:02.333544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425294+0000","last_change":"2024-03-19T03:30:34.627828+0000","last_active":"2024-03-19T03:31:39.425294+0000","last_peered":"2024-03-19T03:31:39.425294+0000","last_clean":"2024-03-19T03:31:39.425294+0000","last_became_active":"2024-03-19T03:30:34.627290+0000","last_became_peered":"2024-03-19T03:30:34.627290+0000","last_unstale":"2024-03-19T03:31:39.425294+0000","last_undegraded":"2024-03-19T03:31:39.425294+0000","last_fullsized":"2024-03-19T03:31:39.425294+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:07:50.140087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426829+0000","last_change":"2024-03-19T03:30:35.661180+0000","last_active":"2024-03-19T03:31:39.426829+0000","last_peered":"2024-03-19T03:31:39.426829+0000","last_clean":"2024-03-19T03:31:39.426829+0000","last_became_active":"2024-03-19T03:30:35.660797+0000","last_became_peered":"2024-03-19T03:30:35.660797+0000","last_unstale":"2024-03-19T03:31:39.426829+0000","last_undegraded":"2024-03-19T03:31:39.426829+0000","last_fullsized":"2024-03-19T03:31:39.426829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:54:36.034281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.437278+0000","last_change":"2024-03-19T03:30:39.732832+0000","last_active":"2024-03-19T03:31:39.437278+0000","last_peered":"2024-03-19T03:31:39.437278+0000","last_clean":"2024-03-19T03:31:39.437278+0000","last_became_active":"2024-03-19T03:30:39.732434+0000","last_became_peered":"2024-03-19T03:30:39.732434+0000","last_unstale":"2024-03-19T03:31:39.437278+0000","last_undegraded":"2024-03-19T03:31:39.437278+0000","last_fullsized":"2024-03-19T03:31:39.437278+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:49.737361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":115,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:32:28.102987+0000","last_change":"2024-03-19T03:30:34.625655+0000","last_active":"2024-03-19T03:32:28.102987+0000","last_peered":"2024-03-19T03:32:28.102987+0000","last_clean":"2024-03-19T03:32:28.102987+0000","last_became_active":"2024-03-19T03:30:34.625251+0000","last_became_peered":"2024-03-19T03:30:34.625251+0000","last_unstale":"2024-03-19T03:32:28.102987+0000","last_undegraded":"2024-03-19T03:32:28.102987+0000","last_fullsized":"2024-03-19T03:32:28.102987+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:25.592670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":83,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426972+0000","last_change":"2024-03-19T03:30:35.651147+0000","last_active":"2024-03-19T03:31:39.426972+0000","last_peered":"2024-03-19T03:31:39.426972+0000","last_clean":"2024-03-19T03:31:39.426972+0000","last_became_active":"2024-03-19T03:30:35.650817+0000","last_became_peered":"2024-03-19T03:30:35.650817+0000","last_unstale":"2024-03-19T03:31:39.426972+0000","last_undegraded":"2024-03-19T03:31:39.426972+0000","last_fullsized":"2024-03-19T03:31:39.426972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:35.529226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.434246+0000","last_change":"2024-03-19T03:30:39.749895+0000","last_active":"2024-03-19T03:31:39.434246+0000","last_peered":"2024-03-19T03:31:39.434246+0000","last_clean":"2024-03-19T03:31:39.434246+0000","last_became_active":"2024-03-19T03:30:39.749555+0000","last_became_peered":"2024-03-19T03:30:39.749555+0000","last_unstale":"2024-03-19T03:31:39.434246+0000","last_undegraded":"2024-03-19T03:31:39.434246+0000","last_fullsized":"2024-03-19T03:31:39.434246+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:10:14.383271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.427995+0000","last_change":"2024-03-19T03:30:34.635456+0000","last_active":"2024-03-19T03:31:39.427995+0000","last_peered":"2024-03-19T03:31:39.427995+0000","last_clean":"2024-03-19T03:31:39.427995+0000","last_became_active":"2024-03-19T03:30:34.635099+0000","last_became_peered":"2024-03-19T03:30:34.635099+0000","last_unstale":"2024-03-19T03:31:39.427995+0000","last_undegraded":"2024-03-19T03:31:39.427995+0000","last_fullsized":"2024-03-19T03:31:39.427995+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:39:30.083678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426720+0000","last_change":"2024-03-19T03:30:35.659490+0000","last_active":"2024-03-19T03:31:39.426720+0000","last_peered":"2024-03-19T03:31:39.426720+0000","last_clean":"2024-03-19T03:31:39.426720+0000","last_became_active":"2024-03-19T03:30:35.659208+0000","last_became_peered":"2024-03-19T03:30:35.659208+0000","last_unstale":"2024-03-19T03:31:39.426720+0000","last_undegraded":"2024-03-19T03:31:39.426720+0000","last_fullsized":"2024-03-19T03:31:39.426720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:18.788981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.424978+0000","last_change":"2024-03-19T03:30:39.725193+0000","last_active":"2024-03-19T03:31:39.424978+0000","last_peered":"2024-03-19T03:31:39.424978+0000","last_clean":"2024-03-19T03:31:39.424978+0000","last_became_active":"2024-03-19T03:30:39.724745+0000","last_became_peered":"2024-03-19T03:30:39.724745+0000","last_unstale":"2024-03-19T03:31:39.424978+0000","last_undegraded":"2024-03-19T03:31:39.424978+0000","last_fullsized":"2024-03-19T03:31:39.424978+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:30:07.621852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.436049+0000","last_change":"2024-03-19T03:30:34.636532+0000","last_active":"2024-03-19T03:31:39.436049+0000","last_peered":"2024-03-19T03:31:39.436049+0000","last_clean":"2024-03-19T03:31:39.436049+0000","last_became_active":"2024-03-19T03:30:34.634104+0000","last_became_peered":"2024-03-19T03:30:34.634104+0000","last_unstale":"2024-03-19T03:31:39.436049+0000","last_undegraded":"2024-03-19T03:31:39.436049+0000","last_fullsized":"2024-03-19T03:31:39.436049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:41:18.308628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426815+0000","last_change":"2024-03-19T03:30:35.653036+0000","last_active":"2024-03-19T03:31:39.426815+0000","last_peered":"2024-03-19T03:31:39.426815+0000","last_clean":"2024-03-19T03:31:39.426815+0000","last_became_active":"2024-03-19T03:30:35.652564+0000","last_became_peered":"2024-03-19T03:30:35.652564+0000","last_unstale":"2024-03-19T03:31:39.426815+0000","last_undegraded":"2024-03-19T03:31:39.426815+0000","last_fullsized":"2024-03-19T03:31:39.426815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:01.812525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425554+0000","last_change":"2024-03-19T03:30:35.650724+0000","last_active":"2024-03-19T03:31:39.425554+0000","last_peered":"2024-03-19T03:31:39.425554+0000","last_clean":"2024-03-19T03:31:39.425554+0000","last_became_active":"2024-03-19T03:30:35.650204+0000","last_became_peered":"2024-03-19T03:30:35.650204+0000","last_unstale":"2024-03-19T03:31:39.425554+0000","last_undegraded":"2024-03-19T03:31:39.425554+0000","last_fullsized":"2024-03-19T03:31:39.425554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:06:56.159133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421190+0000","last_change":"2024-03-19T03:30:35.650241+0000","last_active":"2024-03-19T03:31:39.421190+0000","last_peered":"2024-03-19T03:31:39.421190+0000","last_clean":"2024-03-19T03:31:39.421190+0000","last_became_active":"2024-03-19T03:30:35.649901+0000","last_became_peered":"2024-03-19T03:30:35.649901+0000","last_unstale":"2024-03-19T03:31:39.421190+0000","last_undegraded":"2024-03-19T03:31:39.421190+0000","last_fullsized":"2024-03-19T03:31:39.421190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:35:53.333403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421107+0000","last_change":"2024-03-19T03:30:35.679267+0000","last_active":"2024-03-19T03:31:39.421107+0000","last_peered":"2024-03-19T03:31:39.421107+0000","last_clean":"2024-03-19T03:31:39.421107+0000","last_became_active":"2024-03-19T03:30:35.678871+0000","last_became_peered":"2024-03-19T03:30:35.678871+0000","last_unstale":"2024-03-19T03:31:39.421107+0000","last_undegraded":"2024-03-19T03:31:39.421107+0000","last_fullsized":"2024-03-19T03:31:39.421107+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:23:17.681854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.434732+0000","last_change":"2024-03-19T03:30:35.683060+0000","last_active":"2024-03-19T03:31:39.434732+0000","last_peered":"2024-03-19T03:31:39.434732+0000","last_clean":"2024-03-19T03:31:39.434732+0000","last_became_active":"2024-03-19T03:30:35.676872+0000","last_became_peered":"2024-03-19T03:30:35.676872+0000","last_unstale":"2024-03-19T03:31:39.434732+0000","last_undegraded":"2024-03-19T03:31:39.434732+0000","last_fullsized":"2024-03-19T03:31:39.434732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T15:13:51.530914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426459+0000","last_change":"2024-03-19T03:30:35.682685+0000","last_active":"2024-03-19T03:31:39.426459+0000","last_peered":"2024-03-19T03:31:39.426459+0000","last_clean":"2024-03-19T03:31:39.426459+0000","last_became_active":"2024-03-19T03:30:35.680309+0000","last_became_peered":"2024-03-19T03:30:35.680309+0000","last_unstale":"2024-03-19T03:31:39.426459+0000","last_undegraded":"2024-03-19T03:31:39.426459+0000","last_fullsized":"2024-03-19T03:31:39.426459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:31:59.925521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426932+0000","last_change":"2024-03-19T03:30:35.652656+0000","last_active":"2024-03-19T03:31:39.426932+0000","last_peered":"2024-03-19T03:31:39.426932+0000","last_clean":"2024-03-19T03:31:39.426932+0000","last_became_active":"2024-03-19T03:30:35.650744+0000","last_became_peered":"2024-03-19T03:30:35.650744+0000","last_unstale":"2024-03-19T03:31:39.426932+0000","last_undegraded":"2024-03-19T03:31:39.426932+0000","last_fullsized":"2024-03-19T03:31:39.426932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:49:01.450719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421665+0000","last_change":"2024-03-19T03:30:35.650092+0000","last_active":"2024-03-19T03:31:39.421665+0000","last_peered":"2024-03-19T03:31:39.421665+0000","last_clean":"2024-03-19T03:31:39.421665+0000","last_became_active":"2024-03-19T03:30:35.649727+0000","last_became_peered":"2024-03-19T03:30:35.649727+0000","last_unstale":"2024-03-19T03:31:39.421665+0000","last_undegraded":"2024-03-19T03:31:39.421665+0000","last_fullsized":"2024-03-19T03:31:39.421665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:22:13.968226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.434943+0000","last_change":"2024-03-19T03:31:39.431670+0000","last_active":"2024-03-19T03:31:39.434943+0000","last_peered":"2024-03-19T03:31:39.434943+0000","last_clean":"2024-03-19T03:31:39.434943+0000","last_became_active":"2024-03-19T03:31:39.431299+0000","last_became_peered":"2024-03-19T03:31:39.431299+0000","last_unstale":"2024-03-19T03:31:39.434943+0000","last_undegraded":"2024-03-19T03:31:39.434943+0000","last_fullsized":"2024-03-19T03:31:39.434943+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:04:51.530636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426030+0000","last_change":"2024-03-19T03:30:35.660289+0000","last_active":"2024-03-19T03:31:39.426030+0000","last_peered":"2024-03-19T03:31:39.426030+0000","last_clean":"2024-03-19T03:31:39.426030+0000","last_became_active":"2024-03-19T03:30:35.660005+0000","last_became_peered":"2024-03-19T03:30:35.660005+0000","last_unstale":"2024-03-19T03:31:39.426030+0000","last_undegraded":"2024-03-19T03:31:39.426030+0000","last_fullsized":"2024-03-19T03:31:39.426030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:36:16.033629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421015+0000","last_change":"2024-03-19T03:30:35.660461+0000","last_active":"2024-03-19T03:31:39.421015+0000","last_peered":"2024-03-19T03:31:39.421015+0000","last_clean":"2024-03-19T03:31:39.421015+0000","last_became_active":"2024-03-19T03:30:35.660058+0000","last_became_peered":"2024-03-19T03:30:35.660058+0000","last_unstale":"2024-03-19T03:31:39.421015+0000","last_undegraded":"2024-03-19T03:31:39.421015+0000","last_fullsized":"2024-03-19T03:31:39.421015+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:08:40.326314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.437395+0000","last_change":"2024-03-19T03:30:35.665634+0000","last_active":"2024-03-19T03:31:39.437395+0000","last_peered":"2024-03-19T03:31:39.437395+0000","last_clean":"2024-03-19T03:31:39.437395+0000","last_became_active":"2024-03-19T03:30:35.665264+0000","last_became_peered":"2024-03-19T03:30:35.665264+0000","last_unstale":"2024-03-19T03:31:39.437395+0000","last_undegraded":"2024-03-19T03:31:39.437395+0000","last_fullsized":"2024-03-19T03:31:39.437395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:33:56.931412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425568+0000","last_change":"2024-03-19T03:30:35.681887+0000","last_active":"2024-03-19T03:31:39.425568+0000","last_peered":"2024-03-19T03:31:39.425568+0000","last_clean":"2024-03-19T03:31:39.425568+0000","last_became_active":"2024-03-19T03:30:35.680332+0000","last_became_peered":"2024-03-19T03:30:35.680332+0000","last_unstale":"2024-03-19T03:31:39.425568+0000","last_undegraded":"2024-03-19T03:31:39.425568+0000","last_fullsized":"2024-03-19T03:31:39.425568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:50:24.243641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426932+0000","last_change":"2024-03-19T03:30:35.652988+0000","last_active":"2024-03-19T03:31:39.426932+0000","last_peered":"2024-03-19T03:31:39.426932+0000","last_clean":"2024-03-19T03:31:39.426932+0000","last_became_active":"2024-03-19T03:30:35.652608+0000","last_became_peered":"2024-03-19T03:30:35.652608+0000","last_unstale":"2024-03-19T03:31:39.426932+0000","last_undegraded":"2024-03-19T03:31:39.426932+0000","last_fullsized":"2024-03-19T03:31:39.426932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:44:35.619094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283260,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71410,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":823,"num_read_kb":490,"num_write":244,"num_write_kb":142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":214230,"data_compressed":5478,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":212,"ondisk_log_size":212,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383719,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1003520,"data_stored":501378,"data_compressed":20673,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645355,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28148,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717004,"statfs":{"total":95995035648,"available":95966212096,"internally_reserved":0,"allocated":1150976,"data_stored":1152791,"data_compressed":58523,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808881,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":505404,"data_compressed":20673,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070518,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":505782,"data_compressed":20673,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332155,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1077248,"data_stored":1098574,"data_compressed":56697,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495682,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1003520,"data_stored":496812,"data_compressed":20673,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1069056,"data_stored":562048,"data_compressed":22499,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":498268,"data_compressed":20673,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1089536,"data_stored":562659,"data_compressed":22499,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":496520,"data_compressed":20673,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920846,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1011712,"data_stored":501277,"data_compressed":20673,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757319,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1081344,"data_stored":1089669,"data_compressed":56697,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34380,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34380,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34380,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36024,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36024,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36024,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36024,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36024,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":65958,"data_compressed":1826,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":66569,"data_compressed":1826,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":66333,"data_compressed":1826,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T03:32:32.098 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-19T03:32:32.670 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:32:32.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:32 smithi119 ceph-mon[66861]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T03:32:32.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:32 smithi117 ceph-mon[31838]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T03:32:32.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:32 smithi178 ceph-mon[67578]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T03:32:33.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:33 smithi119 ceph-mon[66861]: pgmap v378: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T03:32:33.743 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:33 smithi117 ceph-mon[31838]: pgmap v378: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T03:32:33.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:33 smithi178 ceph-mon[67578]: pgmap v378: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T03:32:34.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.868+0000 7f3049e00700 1 -- 172.21.15.117:0/348780579 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3044102550 msgr2=0x7f3044102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:34.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.868+0000 7f3049e00700 1 --2- 172.21.15.117:0/348780579 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3044102550 0x7f3044102930 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3034009990 tx=0x7f3034009ca0 comp rx=0 tx=0).stop 2024-03-19T03:32:34.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.868+0000 7f3049e00700 1 -- 172.21.15.117:0/348780579 shutdown_connections 2024-03-19T03:32:34.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.868+0000 7f3049e00700 1 --2- 172.21.15.117:0/348780579 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3044107630 0x7f304410f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:34.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.868+0000 7f3049e00700 1 --2- 172.21.15.117:0/348780579 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3044102550 0x7f3044102930 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:34.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.868+0000 7f3049e00700 1 --2- 172.21.15.117:0/348780579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3044102f00 0x7f3044106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:34.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.868+0000 7f3049e00700 1 -- 172.21.15.117:0/348780579 >> 172.21.15.117:0/348780579 conn(0x7f30440fddc0 msgr2=0x7f30441001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:34.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.868+0000 7f3049e00700 1 -- 172.21.15.117:0/348780579 shutdown_connections 2024-03-19T03:32:34.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.868+0000 7f3049e00700 1 -- 172.21.15.117:0/348780579 wait complete. 2024-03-19T03:32:34.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.869+0000 7f3049e00700 1 Processor -- start 2024-03-19T03:32:34.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.869+0000 7f3049e00700 1 -- start start 2024-03-19T03:32:34.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.870+0000 7f3049e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3044102550 0x7f304419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:34.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.870+0000 7f3049e00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3044102f00 0x7f304419ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:34.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.870+0000 7f3049e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3044107630 0x7f3044198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:34.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.870+0000 7f3049e00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3044112610 con 0x7f3044107630 2024-03-19T03:32:34.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.870+0000 7f3049e00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3044112490 con 0x7f3044102550 2024-03-19T03:32:34.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.870+0000 7f3049e00700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f3044112790 con 0x7f3044102f00 2024-03-19T03:32:34.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.870+0000 7f3042a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3044102550 0x7f304419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:34.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.870+0000 7f3042a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3044102550 0x7f304419e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:48834/0 (socket says 172.21.15.117:48834) 2024-03-19T03:32:34.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.870+0000 7f3042a00700 1 -- 172.21.15.117:0/167730333 learned_addr learned my addr 172.21.15.117:0/167730333 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:34.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.870+0000 7f3042000700 1 --2- 172.21.15.117:0/167730333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3044102f00 0x7f304419ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:34.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.870+0000 7f3043400700 1 --2- 172.21.15.117:0/167730333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3044107630 0x7f3044198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:34.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.870+0000 7f3042a00700 1 -- 172.21.15.117:0/167730333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3044102f00 msgr2=0x7f304419ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:34.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.870+0000 7f3042a00700 1 --2- 172.21.15.117:0/167730333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3044102f00 0x7f304419ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:34.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.870+0000 7f3042a00700 1 -- 172.21.15.117:0/167730333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3044107630 msgr2=0x7f3044198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:34.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.870+0000 7f3042a00700 1 --2- 172.21.15.117:0/167730333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3044107630 0x7f3044198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:34.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.870+0000 7f3042a00700 1 -- 172.21.15.117:0/167730333 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3044198ff0 con 0x7f3044102550 2024-03-19T03:32:34.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.870+0000 7f3043400700 1 --2- 172.21.15.117:0/167730333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3044107630 0x7f3044198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T03:32:34.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.871+0000 7f3042a00700 1 --2- 172.21.15.117:0/167730333 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3044102550 0x7f304419e6d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3034000c00 tx=0x7f30340047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:34.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.871+0000 7f3048a00700 1 -- 172.21.15.117:0/167730333 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3034027070 con 0x7f3044102550 2024-03-19T03:32:34.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.871+0000 7f3048a00700 1 -- 172.21.15.117:0/167730333 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3034004050 con 0x7f3044102550 2024-03-19T03:32:34.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.871+0000 7f3049e00700 1 -- 172.21.15.117:0/167730333 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3044199280 con 0x7f3044102550 2024-03-19T03:32:34.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.871+0000 7f3048a00700 1 -- 172.21.15.117:0/167730333 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303400f460 con 0x7f3044102550 2024-03-19T03:32:34.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.871+0000 7f3049e00700 1 -- 172.21.15.117:0/167730333 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30441a55a0 con 0x7f3044102550 2024-03-19T03:32:34.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.872+0000 7f3049e00700 1 -- 172.21.15.117:0/167730333 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3044068fd0 con 0x7f3044102550 2024-03-19T03:32:34.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.874+0000 7f3048a00700 1 -- 172.21.15.117:0/167730333 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f303402a070 con 0x7f3044102550 2024-03-19T03:32:34.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.876+0000 7f3048a00700 1 --2- 172.21.15.117:0/167730333 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f30280a29a0 0x7f30280a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:34.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.876+0000 7f3048a00700 1 -- 172.21.15.117:0/167730333 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f30340cfec0 con 0x7f3044102550 2024-03-19T03:32:34.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.876+0000 7f3042000700 1 --2- 172.21.15.117:0/167730333 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f30280a29a0 0x7f30280a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:34.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.877+0000 7f3042000700 1 --2- 172.21.15.117:0/167730333 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f30280a29a0 0x7f30280a4e60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3030005fd0 tx=0x7f3030005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:34.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:34.884+0000 7f3048a00700 1 -- 172.21.15.117:0/167730333 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f303405e740 con 0x7f3044102550 2024-03-19T03:32:35.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:35.124+0000 7f3049e00700 1 -- 172.21.15.117:0/167730333 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3044062750 con 0x7f30280a29a0 2024-03-19T03:32:35.136 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:35.136+0000 7f3048a00700 1 -- 172.21.15.117:0/167730333 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340712 (secure 0 0 0) 0x7f3044062750 con 0x7f30280a29a0 2024-03-19T03:32:35.137 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:32:35.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:35.142+0000 7f3049e00700 1 -- 172.21.15.117:0/167730333 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f30280a29a0 msgr2=0x7f30280a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:35.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:35.142+0000 7f3049e00700 1 --2- 172.21.15.117:0/167730333 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f30280a29a0 0x7f30280a4e60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3030005fd0 tx=0x7f3030005f60 comp rx=0 tx=0).stop 2024-03-19T03:32:35.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:35.142+0000 7f3049e00700 1 -- 172.21.15.117:0/167730333 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3044102550 msgr2=0x7f304419e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:35.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:35.142+0000 7f3049e00700 1 --2- 172.21.15.117:0/167730333 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3044102550 0x7f304419e6d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3034000c00 tx=0x7f30340047b0 comp rx=0 tx=0).stop 2024-03-19T03:32:35.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:35.142+0000 7f3049e00700 1 -- 172.21.15.117:0/167730333 shutdown_connections 2024-03-19T03:32:35.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:35.142+0000 7f3049e00700 1 --2- 172.21.15.117:0/167730333 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f30280a29a0 0x7f30280a4e60 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:35.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:35.142+0000 7f3049e00700 1 --2- 172.21.15.117:0/167730333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3044107630 0x7f3044198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:35.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:35.142+0000 7f3049e00700 1 --2- 172.21.15.117:0/167730333 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3044102550 0x7f304419e6d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:35.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:35.142+0000 7f3049e00700 1 --2- 172.21.15.117:0/167730333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3044102f00 0x7f304419ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:35.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:35.142+0000 7f3049e00700 1 -- 172.21.15.117:0/167730333 >> 172.21.15.117:0/167730333 conn(0x7f30440fddc0 msgr2=0x7f30440ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:35.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:35.143+0000 7f3049e00700 1 -- 172.21.15.117:0/167730333 shutdown_connections 2024-03-19T03:32:35.145 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:35.143+0000 7f3049e00700 1 -- 172.21.15.117:0/167730333 wait complete. 2024-03-19T03:32:35.146 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-03-19T03:32:35.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:35 smithi119 ceph-mon[66861]: pgmap v379: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:32:35.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:35 smithi117 ceph-mon[31838]: pgmap v379: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:32:35.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:35 smithi178 ceph-mon[67578]: pgmap v379: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:32:35.975 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2024-03-19T03:32:34.326037+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662012,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":999,"num_read_kb":624,"num_write":427,"num_write_kb":3082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1180853,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":351,"ondisk_log_size":351,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361528,"kb_used_data":12296,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580296,"statfs":{"total":1151940427776,"available":1151570223104,"internally_reserved":0,"allocated":12591104,"data_stored":7971182,"data_compressed":361626,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19138,"internal_metadata":357414206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":65,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005585"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.427095+0000","last_change":"2024-03-19T03:30:35.676095+0000","last_active":"2024-03-19T03:31:39.427095+0000","last_peered":"2024-03-19T03:31:39.427095+0000","last_clean":"2024-03-19T03:31:39.427095+0000","last_became_active":"2024-03-19T03:30:35.675772+0000","last_became_peered":"2024-03-19T03:30:35.675772+0000","last_unstale":"2024-03-19T03:31:39.427095+0000","last_undegraded":"2024-03-19T03:31:39.427095+0000","last_fullsized":"2024-03-19T03:31:39.427095+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:12:31.902482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.435407+0000","last_change":"2024-03-19T03:30:35.672451+0000","last_active":"2024-03-19T03:31:39.435407+0000","last_peered":"2024-03-19T03:31:39.435407+0000","last_clean":"2024-03-19T03:31:39.435407+0000","last_became_active":"2024-03-19T03:30:35.671516+0000","last_became_peered":"2024-03-19T03:30:35.671516+0000","last_unstale":"2024-03-19T03:31:39.435407+0000","last_undegraded":"2024-03-19T03:31:39.435407+0000","last_fullsized":"2024-03-19T03:31:39.435407+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:14:07.021704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.427269+0000","last_change":"2024-03-19T03:30:35.674425+0000","last_active":"2024-03-19T03:31:39.427269+0000","last_peered":"2024-03-19T03:31:39.427269+0000","last_clean":"2024-03-19T03:31:39.427269+0000","last_became_active":"2024-03-19T03:30:35.674012+0000","last_became_peered":"2024-03-19T03:30:35.674012+0000","last_unstale":"2024-03-19T03:31:39.427269+0000","last_undegraded":"2024-03-19T03:31:39.427269+0000","last_fullsized":"2024-03-19T03:31:39.427269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:51:44.850847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.423258+0000","last_change":"2024-03-19T03:30:35.649218+0000","last_active":"2024-03-19T03:31:39.423258+0000","last_peered":"2024-03-19T03:31:39.423258+0000","last_clean":"2024-03-19T03:31:39.423258+0000","last_became_active":"2024-03-19T03:30:35.648848+0000","last_became_peered":"2024-03-19T03:30:35.648848+0000","last_unstale":"2024-03-19T03:31:39.423258+0000","last_undegraded":"2024-03-19T03:31:39.423258+0000","last_fullsized":"2024-03-19T03:31:39.423258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:24.204126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425132+0000","last_change":"2024-03-19T03:30:35.670208+0000","last_active":"2024-03-19T03:31:39.425132+0000","last_peered":"2024-03-19T03:31:39.425132+0000","last_clean":"2024-03-19T03:31:39.425132+0000","last_became_active":"2024-03-19T03:30:35.669939+0000","last_became_peered":"2024-03-19T03:30:35.669939+0000","last_unstale":"2024-03-19T03:31:39.425132+0000","last_undegraded":"2024-03-19T03:31:39.425132+0000","last_fullsized":"2024-03-19T03:31:39.425132+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:57:59.687262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.423789+0000","last_change":"2024-03-19T03:30:35.662329+0000","last_active":"2024-03-19T03:31:39.423789+0000","last_peered":"2024-03-19T03:31:39.423789+0000","last_clean":"2024-03-19T03:31:39.423789+0000","last_became_active":"2024-03-19T03:30:35.661969+0000","last_became_peered":"2024-03-19T03:30:35.661969+0000","last_unstale":"2024-03-19T03:31:39.423789+0000","last_undegraded":"2024-03-19T03:31:39.423789+0000","last_fullsized":"2024-03-19T03:31:39.423789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:29.265017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426342+0000","last_change":"2024-03-19T03:30:35.662586+0000","last_active":"2024-03-19T03:31:39.426342+0000","last_peered":"2024-03-19T03:31:39.426342+0000","last_clean":"2024-03-19T03:31:39.426342+0000","last_became_active":"2024-03-19T03:30:35.661774+0000","last_became_peered":"2024-03-19T03:30:35.661774+0000","last_unstale":"2024-03-19T03:31:39.426342+0000","last_undegraded":"2024-03-19T03:31:39.426342+0000","last_fullsized":"2024-03-19T03:31:39.426342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:04:49.258178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.420906+0000","last_change":"2024-03-19T03:30:35.638512+0000","last_active":"2024-03-19T03:31:39.420906+0000","last_peered":"2024-03-19T03:31:39.420906+0000","last_clean":"2024-03-19T03:31:39.420906+0000","last_became_active":"2024-03-19T03:30:35.637205+0000","last_became_peered":"2024-03-19T03:30:35.637205+0000","last_unstale":"2024-03-19T03:31:39.420906+0000","last_undegraded":"2024-03-19T03:31:39.420906+0000","last_fullsized":"2024-03-19T03:31:39.420906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:29:34.578520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.437823+0000","last_change":"2024-03-19T03:30:35.650681+0000","last_active":"2024-03-19T03:31:39.437823+0000","last_peered":"2024-03-19T03:31:39.437823+0000","last_clean":"2024-03-19T03:31:39.437823+0000","last_became_active":"2024-03-19T03:30:35.650324+0000","last_became_peered":"2024-03-19T03:30:35.650324+0000","last_unstale":"2024-03-19T03:31:39.437823+0000","last_undegraded":"2024-03-19T03:31:39.437823+0000","last_fullsized":"2024-03-19T03:31:39.437823+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:48:23.413530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.424504+0000","last_change":"2024-03-19T03:30:35.650240+0000","last_active":"2024-03-19T03:31:39.424504+0000","last_peered":"2024-03-19T03:31:39.424504+0000","last_clean":"2024-03-19T03:31:39.424504+0000","last_became_active":"2024-03-19T03:30:35.649888+0000","last_became_peered":"2024-03-19T03:30:35.649888+0000","last_unstale":"2024-03-19T03:31:39.424504+0000","last_undegraded":"2024-03-19T03:31:39.424504+0000","last_fullsized":"2024-03-19T03:31:39.424504+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:55:30.110289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426913+0000","last_change":"2024-03-19T03:30:35.657058+0000","last_active":"2024-03-19T03:31:39.426913+0000","last_peered":"2024-03-19T03:31:39.426913+0000","last_clean":"2024-03-19T03:31:39.426913+0000","last_became_active":"2024-03-19T03:30:35.656678+0000","last_became_peered":"2024-03-19T03:30:35.656678+0000","last_unstale":"2024-03-19T03:31:39.426913+0000","last_undegraded":"2024-03-19T03:31:39.426913+0000","last_fullsized":"2024-03-19T03:31:39.426913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:10.211852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.423141+0000","last_change":"2024-03-19T03:30:35.651543+0000","last_active":"2024-03-19T03:31:39.423141+0000","last_peered":"2024-03-19T03:31:39.423141+0000","last_clean":"2024-03-19T03:31:39.423141+0000","last_became_active":"2024-03-19T03:30:35.649514+0000","last_became_peered":"2024-03-19T03:30:35.649514+0000","last_unstale":"2024-03-19T03:31:39.423141+0000","last_undegraded":"2024-03-19T03:31:39.423141+0000","last_fullsized":"2024-03-19T03:31:39.423141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:54:37.607873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425739+0000","last_change":"2024-03-19T03:30:35.662494+0000","last_active":"2024-03-19T03:31:39.425739+0000","last_peered":"2024-03-19T03:31:39.425739+0000","last_clean":"2024-03-19T03:31:39.425739+0000","last_became_active":"2024-03-19T03:30:35.660596+0000","last_became_peered":"2024-03-19T03:30:35.660596+0000","last_unstale":"2024-03-19T03:31:39.425739+0000","last_undegraded":"2024-03-19T03:31:39.425739+0000","last_fullsized":"2024-03-19T03:31:39.425739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:25:59.184025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.433898+0000","last_change":"2024-03-19T03:30:35.675751+0000","last_active":"2024-03-19T03:31:39.433898+0000","last_peered":"2024-03-19T03:31:39.433898+0000","last_clean":"2024-03-19T03:31:39.433898+0000","last_became_active":"2024-03-19T03:30:35.674313+0000","last_became_peered":"2024-03-19T03:30:35.674313+0000","last_unstale":"2024-03-19T03:31:39.433898+0000","last_undegraded":"2024-03-19T03:31:39.433898+0000","last_fullsized":"2024-03-19T03:31:39.433898+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:26:26.433958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.428301+0000","last_change":"2024-03-19T03:30:35.672628+0000","last_active":"2024-03-19T03:31:39.428301+0000","last_peered":"2024-03-19T03:31:39.428301+0000","last_clean":"2024-03-19T03:31:39.428301+0000","last_became_active":"2024-03-19T03:30:35.672246+0000","last_became_peered":"2024-03-19T03:30:35.672246+0000","last_unstale":"2024-03-19T03:31:39.428301+0000","last_undegraded":"2024-03-19T03:31:39.428301+0000","last_fullsized":"2024-03-19T03:31:39.428301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:07:59.891108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.437160+0000","last_change":"2024-03-19T03:30:35.678718+0000","last_active":"2024-03-19T03:31:39.437160+0000","last_peered":"2024-03-19T03:31:39.437160+0000","last_clean":"2024-03-19T03:31:39.437160+0000","last_became_active":"2024-03-19T03:30:35.678461+0000","last_became_peered":"2024-03-19T03:30:35.678461+0000","last_unstale":"2024-03-19T03:31:39.437160+0000","last_undegraded":"2024-03-19T03:31:39.437160+0000","last_fullsized":"2024-03-19T03:31:39.437160+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:19:39.079048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426980+0000","last_change":"2024-03-19T03:30:35.675847+0000","last_active":"2024-03-19T03:31:39.426980+0000","last_peered":"2024-03-19T03:31:39.426980+0000","last_clean":"2024-03-19T03:31:39.426980+0000","last_became_active":"2024-03-19T03:30:35.675204+0000","last_became_peered":"2024-03-19T03:30:35.675204+0000","last_unstale":"2024-03-19T03:31:39.426980+0000","last_undegraded":"2024-03-19T03:31:39.426980+0000","last_fullsized":"2024-03-19T03:31:39.426980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:08.732536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.428252+0000","last_change":"2024-03-19T03:30:35.663110+0000","last_active":"2024-03-19T03:31:39.428252+0000","last_peered":"2024-03-19T03:31:39.428252+0000","last_clean":"2024-03-19T03:31:39.428252+0000","last_became_active":"2024-03-19T03:30:35.658376+0000","last_became_peered":"2024-03-19T03:30:35.658376+0000","last_unstale":"2024-03-19T03:31:39.428252+0000","last_undegraded":"2024-03-19T03:31:39.428252+0000","last_fullsized":"2024-03-19T03:31:39.428252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:29:44.658681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421561+0000","last_change":"2024-03-19T03:30:35.661154+0000","last_active":"2024-03-19T03:31:39.421561+0000","last_peered":"2024-03-19T03:31:39.421561+0000","last_clean":"2024-03-19T03:31:39.421561+0000","last_became_active":"2024-03-19T03:30:35.660794+0000","last_became_peered":"2024-03-19T03:30:35.660794+0000","last_unstale":"2024-03-19T03:31:39.421561+0000","last_undegraded":"2024-03-19T03:31:39.421561+0000","last_fullsized":"2024-03-19T03:31:39.421561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:26:29.803827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.420977+0000","last_change":"2024-03-19T03:30:39.735971+0000","last_active":"2024-03-19T03:31:39.420977+0000","last_peered":"2024-03-19T03:31:39.420977+0000","last_clean":"2024-03-19T03:31:39.420977+0000","last_became_active":"2024-03-19T03:30:39.735585+0000","last_became_peered":"2024-03-19T03:30:39.735585+0000","last_unstale":"2024-03-19T03:31:39.420977+0000","last_undegraded":"2024-03-19T03:31:39.420977+0000","last_fullsized":"2024-03-19T03:31:39.420977+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:30:15.969148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":122,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:32:29.142868+0000","last_change":"2024-03-19T03:30:34.632177+0000","last_active":"2024-03-19T03:32:29.142868+0000","last_peered":"2024-03-19T03:32:29.142868+0000","last_clean":"2024-03-19T03:32:29.142868+0000","last_became_active":"2024-03-19T03:30:34.631836+0000","last_became_peered":"2024-03-19T03:30:34.631836+0000","last_unstale":"2024-03-19T03:32:29.142868+0000","last_undegraded":"2024-03-19T03:32:29.142868+0000","last_fullsized":"2024-03-19T03:32:29.142868+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:15:42.883089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":82,"num_read_kb":82,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.435242+0000","last_change":"2024-03-19T03:31:39.433509+0000","last_active":"2024-03-19T03:31:39.435242+0000","last_peered":"2024-03-19T03:31:39.435242+0000","last_clean":"2024-03-19T03:31:39.435242+0000","last_became_active":"2024-03-19T03:31:39.431889+0000","last_became_peered":"2024-03-19T03:31:39.431889+0000","last_unstale":"2024-03-19T03:31:39.435242+0000","last_undegraded":"2024-03-19T03:31:39.435242+0000","last_fullsized":"2024-03-19T03:31:39.435242+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:51:14.868898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.422448+0000","last_change":"2024-03-19T03:30:39.733744+0000","last_active":"2024-03-19T03:31:39.422448+0000","last_peered":"2024-03-19T03:31:39.422448+0000","last_clean":"2024-03-19T03:31:39.422448+0000","last_became_active":"2024-03-19T03:30:39.733373+0000","last_became_peered":"2024-03-19T03:30:39.733373+0000","last_unstale":"2024-03-19T03:31:39.422448+0000","last_undegraded":"2024-03-19T03:31:39.422448+0000","last_fullsized":"2024-03-19T03:31:39.422448+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:48:48.186995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.422844+0000","last_change":"2024-03-19T03:30:34.632369+0000","last_active":"2024-03-19T03:31:39.422844+0000","last_peered":"2024-03-19T03:31:39.422844+0000","last_clean":"2024-03-19T03:31:39.422844+0000","last_became_active":"2024-03-19T03:30:34.632027+0000","last_became_peered":"2024-03-19T03:30:34.632027+0000","last_unstale":"2024-03-19T03:31:39.422844+0000","last_undegraded":"2024-03-19T03:31:39.422844+0000","last_fullsized":"2024-03-19T03:31:39.422844+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:04:22.575864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.437652+0000","last_change":"2024-03-19T03:30:35.677250+0000","last_active":"2024-03-19T03:31:39.437652+0000","last_peered":"2024-03-19T03:31:39.437652+0000","last_clean":"2024-03-19T03:31:39.437652+0000","last_became_active":"2024-03-19T03:30:35.676741+0000","last_became_peered":"2024-03-19T03:30:35.676741+0000","last_unstale":"2024-03-19T03:31:39.437652+0000","last_undegraded":"2024-03-19T03:31:39.437652+0000","last_fullsized":"2024-03-19T03:31:39.437652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:52:06.095738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.422804+0000","last_change":"2024-03-19T03:30:39.733501+0000","last_active":"2024-03-19T03:31:39.422804+0000","last_peered":"2024-03-19T03:31:39.422804+0000","last_clean":"2024-03-19T03:31:39.422804+0000","last_became_active":"2024-03-19T03:30:39.733102+0000","last_became_peered":"2024-03-19T03:30:39.733102+0000","last_unstale":"2024-03-19T03:31:39.422804+0000","last_undegraded":"2024-03-19T03:31:39.422804+0000","last_fullsized":"2024-03-19T03:31:39.422804+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:35:06.004381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425441+0000","last_change":"2024-03-19T03:30:34.627192+0000","last_active":"2024-03-19T03:31:39.425441+0000","last_peered":"2024-03-19T03:31:39.425441+0000","last_clean":"2024-03-19T03:31:39.425441+0000","last_became_active":"2024-03-19T03:30:34.626730+0000","last_became_peered":"2024-03-19T03:30:34.626730+0000","last_unstale":"2024-03-19T03:31:39.425441+0000","last_undegraded":"2024-03-19T03:31:39.425441+0000","last_fullsized":"2024-03-19T03:31:39.425441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:20:25.816500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.437424+0000","last_change":"2024-03-19T03:30:35.663783+0000","last_active":"2024-03-19T03:31:39.437424+0000","last_peered":"2024-03-19T03:31:39.437424+0000","last_clean":"2024-03-19T03:31:39.437424+0000","last_became_active":"2024-03-19T03:30:35.663472+0000","last_became_peered":"2024-03-19T03:30:35.663472+0000","last_unstale":"2024-03-19T03:31:39.437424+0000","last_undegraded":"2024-03-19T03:31:39.437424+0000","last_fullsized":"2024-03-19T03:31:39.437424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:51:15.211113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.422588+0000","last_change":"2024-03-19T03:30:39.731492+0000","last_active":"2024-03-19T03:31:39.422588+0000","last_peered":"2024-03-19T03:31:39.422588+0000","last_clean":"2024-03-19T03:31:39.422588+0000","last_became_active":"2024-03-19T03:30:39.731168+0000","last_became_peered":"2024-03-19T03:30:39.731168+0000","last_unstale":"2024-03-19T03:31:39.422588+0000","last_undegraded":"2024-03-19T03:31:39.422588+0000","last_fullsized":"2024-03-19T03:31:39.422588+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:11:48.281607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425834+0000","last_change":"2024-03-19T03:30:34.627589+0000","last_active":"2024-03-19T03:31:39.425834+0000","last_peered":"2024-03-19T03:31:39.425834+0000","last_clean":"2024-03-19T03:31:39.425834+0000","last_became_active":"2024-03-19T03:30:34.627229+0000","last_became_peered":"2024-03-19T03:30:34.627229+0000","last_unstale":"2024-03-19T03:31:39.425834+0000","last_undegraded":"2024-03-19T03:31:39.425834+0000","last_fullsized":"2024-03-19T03:31:39.425834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:53:23.128932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.438748+0000","last_change":"2024-03-19T03:31:39.438748+0000","last_active":"2024-03-19T03:31:39.438748+0000","last_peered":"2024-03-19T03:31:39.438748+0000","last_clean":"2024-03-19T03:31:39.438748+0000","last_became_active":"2024-03-19T03:31:39.438163+0000","last_became_peered":"2024-03-19T03:31:39.438163+0000","last_unstale":"2024-03-19T03:31:39.438748+0000","last_undegraded":"2024-03-19T03:31:39.438748+0000","last_fullsized":"2024-03-19T03:31:39.438748+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:01.537442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421281+0000","last_change":"2024-03-19T03:30:39.727351+0000","last_active":"2024-03-19T03:31:39.421281+0000","last_peered":"2024-03-19T03:31:39.421281+0000","last_clean":"2024-03-19T03:31:39.421281+0000","last_became_active":"2024-03-19T03:30:39.726868+0000","last_became_peered":"2024-03-19T03:30:39.726868+0000","last_unstale":"2024-03-19T03:31:39.421281+0000","last_undegraded":"2024-03-19T03:31:39.421281+0000","last_fullsized":"2024-03-19T03:31:39.421281+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:40:59.003064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":116,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:32:29.104559+0000","last_change":"2024-03-19T03:30:34.623911+0000","last_active":"2024-03-19T03:32:29.104559+0000","last_peered":"2024-03-19T03:32:29.104559+0000","last_clean":"2024-03-19T03:32:29.104559+0000","last_became_active":"2024-03-19T03:30:34.623548+0000","last_became_peered":"2024-03-19T03:30:34.623548+0000","last_unstale":"2024-03-19T03:32:29.104559+0000","last_undegraded":"2024-03-19T03:32:29.104559+0000","last_fullsized":"2024-03-19T03:32:29.104559+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:29:56.616353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425315+0000","last_change":"2024-03-19T03:30:35.661874+0000","last_active":"2024-03-19T03:31:39.425315+0000","last_peered":"2024-03-19T03:31:39.425315+0000","last_clean":"2024-03-19T03:31:39.425315+0000","last_became_active":"2024-03-19T03:30:35.661521+0000","last_became_peered":"2024-03-19T03:30:35.661521+0000","last_unstale":"2024-03-19T03:31:39.425315+0000","last_undegraded":"2024-03-19T03:31:39.425315+0000","last_fullsized":"2024-03-19T03:31:39.425315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:19:35.017020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421192+0000","last_change":"2024-03-19T03:30:39.748197+0000","last_active":"2024-03-19T03:31:39.421192+0000","last_peered":"2024-03-19T03:31:39.421192+0000","last_clean":"2024-03-19T03:31:39.421192+0000","last_became_active":"2024-03-19T03:30:39.747833+0000","last_became_peered":"2024-03-19T03:30:39.747833+0000","last_unstale":"2024-03-19T03:31:39.421192+0000","last_undegraded":"2024-03-19T03:31:39.421192+0000","last_fullsized":"2024-03-19T03:31:39.421192+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:29:19.213403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.423728+0000","last_change":"2024-03-19T03:30:34.623491+0000","last_active":"2024-03-19T03:31:39.423728+0000","last_peered":"2024-03-19T03:31:39.423728+0000","last_clean":"2024-03-19T03:31:39.423728+0000","last_became_active":"2024-03-19T03:30:34.623152+0000","last_became_peered":"2024-03-19T03:30:34.623152+0000","last_unstale":"2024-03-19T03:31:39.423728+0000","last_undegraded":"2024-03-19T03:31:39.423728+0000","last_fullsized":"2024-03-19T03:31:39.423728+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:00:25.530926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.427403+0000","last_change":"2024-03-19T03:30:35.651528+0000","last_active":"2024-03-19T03:31:39.427403+0000","last_peered":"2024-03-19T03:31:39.427403+0000","last_clean":"2024-03-19T03:31:39.427403+0000","last_became_active":"2024-03-19T03:30:35.650240+0000","last_became_peered":"2024-03-19T03:30:35.650240+0000","last_unstale":"2024-03-19T03:31:39.427403+0000","last_undegraded":"2024-03-19T03:31:39.427403+0000","last_fullsized":"2024-03-19T03:31:39.427403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:25.123570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426848+0000","last_change":"2024-03-19T03:30:39.740630+0000","last_active":"2024-03-19T03:31:39.426848+0000","last_peered":"2024-03-19T03:31:39.426848+0000","last_clean":"2024-03-19T03:31:39.426848+0000","last_became_active":"2024-03-19T03:30:39.740170+0000","last_became_peered":"2024-03-19T03:30:39.740170+0000","last_unstale":"2024-03-19T03:31:39.426848+0000","last_undegraded":"2024-03-19T03:31:39.426848+0000","last_fullsized":"2024-03-19T03:31:39.426848+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:23:49.415718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425434+0000","last_change":"2024-03-19T03:30:34.635173+0000","last_active":"2024-03-19T03:31:39.425434+0000","last_peered":"2024-03-19T03:31:39.425434+0000","last_clean":"2024-03-19T03:31:39.425434+0000","last_became_active":"2024-03-19T03:30:34.633809+0000","last_became_peered":"2024-03-19T03:30:34.633809+0000","last_unstale":"2024-03-19T03:31:39.425434+0000","last_undegraded":"2024-03-19T03:31:39.425434+0000","last_fullsized":"2024-03-19T03:31:39.425434+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:03:57.409707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.420821+0000","last_change":"2024-03-19T03:30:35.665660+0000","last_active":"2024-03-19T03:31:39.420821+0000","last_peered":"2024-03-19T03:31:39.420821+0000","last_clean":"2024-03-19T03:31:39.420821+0000","last_became_active":"2024-03-19T03:30:35.665290+0000","last_became_peered":"2024-03-19T03:30:35.665290+0000","last_unstale":"2024-03-19T03:31:39.420821+0000","last_undegraded":"2024-03-19T03:31:39.420821+0000","last_fullsized":"2024-03-19T03:31:39.420821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:20.333009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.435292+0000","last_change":"2024-03-19T03:30:39.744391+0000","last_active":"2024-03-19T03:31:39.435292+0000","last_peered":"2024-03-19T03:31:39.435292+0000","last_clean":"2024-03-19T03:31:39.435292+0000","last_became_active":"2024-03-19T03:30:39.744000+0000","last_became_peered":"2024-03-19T03:30:39.744000+0000","last_unstale":"2024-03-19T03:31:39.435292+0000","last_undegraded":"2024-03-19T03:31:39.435292+0000","last_fullsized":"2024-03-19T03:31:39.435292+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:17:06.124286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425903+0000","last_change":"2024-03-19T03:30:34.621811+0000","last_active":"2024-03-19T03:31:39.425903+0000","last_peered":"2024-03-19T03:31:39.425903+0000","last_clean":"2024-03-19T03:31:39.425903+0000","last_became_active":"2024-03-19T03:30:34.621308+0000","last_became_peered":"2024-03-19T03:30:34.621308+0000","last_unstale":"2024-03-19T03:31:39.425903+0000","last_undegraded":"2024-03-19T03:31:39.425903+0000","last_fullsized":"2024-03-19T03:31:39.425903+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:36:45.895720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.429510+0000","last_change":"2024-03-19T03:30:35.673683+0000","last_active":"2024-03-19T03:31:39.429510+0000","last_peered":"2024-03-19T03:31:39.429510+0000","last_clean":"2024-03-19T03:31:39.429510+0000","last_became_active":"2024-03-19T03:30:35.670489+0000","last_became_peered":"2024-03-19T03:30:35.670489+0000","last_unstale":"2024-03-19T03:31:39.429510+0000","last_undegraded":"2024-03-19T03:31:39.429510+0000","last_fullsized":"2024-03-19T03:31:39.429510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:48:30.521454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421010+0000","last_change":"2024-03-19T03:30:39.726358+0000","last_active":"2024-03-19T03:31:39.421010+0000","last_peered":"2024-03-19T03:31:39.421010+0000","last_clean":"2024-03-19T03:31:39.421010+0000","last_became_active":"2024-03-19T03:30:39.725720+0000","last_became_peered":"2024-03-19T03:30:39.725720+0000","last_unstale":"2024-03-19T03:31:39.421010+0000","last_undegraded":"2024-03-19T03:31:39.421010+0000","last_fullsized":"2024-03-19T03:31:39.421010+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:01:08.759794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":201,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:32:33.109304+0000","last_change":"2024-03-19T03:30:34.626336+0000","last_active":"2024-03-19T03:32:33.109304+0000","last_peered":"2024-03-19T03:32:33.109304+0000","last_clean":"2024-03-19T03:32:33.109304+0000","last_became_active":"2024-03-19T03:30:34.625991+0000","last_became_peered":"2024-03-19T03:30:34.625991+0000","last_unstale":"2024-03-19T03:32:33.109304+0000","last_undegraded":"2024-03-19T03:32:33.109304+0000","last_fullsized":"2024-03-19T03:32:33.109304+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:40:58.791759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":170,"num_read_kb":170,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.437787+0000","last_change":"2024-03-19T03:30:35.667001+0000","last_active":"2024-03-19T03:31:39.437787+0000","last_peered":"2024-03-19T03:31:39.437787+0000","last_clean":"2024-03-19T03:31:39.437787+0000","last_became_active":"2024-03-19T03:30:35.666586+0000","last_became_peered":"2024-03-19T03:30:35.666586+0000","last_unstale":"2024-03-19T03:31:39.437787+0000","last_undegraded":"2024-03-19T03:31:39.437787+0000","last_fullsized":"2024-03-19T03:31:39.437787+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:40:30.100860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.428857+0000","last_change":"2024-03-19T03:30:39.745267+0000","last_active":"2024-03-19T03:31:39.428857+0000","last_peered":"2024-03-19T03:31:39.428857+0000","last_clean":"2024-03-19T03:31:39.428857+0000","last_became_active":"2024-03-19T03:30:39.743434+0000","last_became_peered":"2024-03-19T03:30:39.743434+0000","last_unstale":"2024-03-19T03:31:39.428857+0000","last_undegraded":"2024-03-19T03:31:39.428857+0000","last_fullsized":"2024-03-19T03:31:39.428857+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:53:37.583886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426670+0000","last_change":"2024-03-19T03:30:34.617769+0000","last_active":"2024-03-19T03:31:39.426670+0000","last_peered":"2024-03-19T03:31:39.426670+0000","last_clean":"2024-03-19T03:31:39.426670+0000","last_became_active":"2024-03-19T03:30:34.617428+0000","last_became_peered":"2024-03-19T03:30:34.617428+0000","last_unstale":"2024-03-19T03:31:39.426670+0000","last_undegraded":"2024-03-19T03:31:39.426670+0000","last_fullsized":"2024-03-19T03:31:39.426670+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:58:09.181320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.440189+0000","last_change":"2024-03-19T03:31:39.440189+0000","last_active":"2024-03-19T03:31:39.440189+0000","last_peered":"2024-03-19T03:31:39.440189+0000","last_clean":"2024-03-19T03:31:39.440189+0000","last_became_active":"2024-03-19T03:31:39.439752+0000","last_became_peered":"2024-03-19T03:31:39.439752+0000","last_unstale":"2024-03-19T03:31:39.440189+0000","last_undegraded":"2024-03-19T03:31:39.440189+0000","last_fullsized":"2024-03-19T03:31:39.440189+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:34:45.028679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.422928+0000","last_change":"2024-03-19T03:30:39.733273+0000","last_active":"2024-03-19T03:31:39.422928+0000","last_peered":"2024-03-19T03:31:39.422928+0000","last_clean":"2024-03-19T03:31:39.422928+0000","last_became_active":"2024-03-19T03:30:39.732885+0000","last_became_peered":"2024-03-19T03:30:39.732885+0000","last_unstale":"2024-03-19T03:31:39.422928+0000","last_undegraded":"2024-03-19T03:31:39.422928+0000","last_fullsized":"2024-03-19T03:31:39.422928+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:48:10.446577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.427841+0000","last_change":"2024-03-19T03:30:34.637076+0000","last_active":"2024-03-19T03:31:39.427841+0000","last_peered":"2024-03-19T03:31:39.427841+0000","last_clean":"2024-03-19T03:31:39.427841+0000","last_became_active":"2024-03-19T03:30:34.636642+0000","last_became_peered":"2024-03-19T03:30:34.636642+0000","last_unstale":"2024-03-19T03:31:39.427841+0000","last_undegraded":"2024-03-19T03:31:39.427841+0000","last_fullsized":"2024-03-19T03:31:39.427841+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:26:17.494081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.437546+0000","last_change":"2024-03-19T03:30:35.651739+0000","last_active":"2024-03-19T03:31:39.437546+0000","last_peered":"2024-03-19T03:31:39.437546+0000","last_clean":"2024-03-19T03:31:39.437546+0000","last_became_active":"2024-03-19T03:30:35.651419+0000","last_became_peered":"2024-03-19T03:30:35.651419+0000","last_unstale":"2024-03-19T03:31:39.437546+0000","last_undegraded":"2024-03-19T03:31:39.437546+0000","last_fullsized":"2024-03-19T03:31:39.437546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T15:17:55.022050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426700+0000","last_change":"2024-03-19T03:30:39.733228+0000","last_active":"2024-03-19T03:31:39.426700+0000","last_peered":"2024-03-19T03:31:39.426700+0000","last_clean":"2024-03-19T03:31:39.426700+0000","last_became_active":"2024-03-19T03:30:39.732819+0000","last_became_peered":"2024-03-19T03:30:39.732819+0000","last_unstale":"2024-03-19T03:31:39.426700+0000","last_undegraded":"2024-03-19T03:31:39.426700+0000","last_fullsized":"2024-03-19T03:31:39.426700+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:03:43.968209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'35","reported_seq":240,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:32:33.109294+0000","last_change":"2024-03-19T03:30:34.618323+0000","last_active":"2024-03-19T03:32:33.109294+0000","last_peered":"2024-03-19T03:32:33.109294+0000","last_clean":"2024-03-19T03:32:33.109294+0000","last_became_active":"2024-03-19T03:30:34.617975+0000","last_became_peered":"2024-03-19T03:30:34.617975+0000","last_unstale":"2024-03-19T03:32:33.109294+0000","last_undegraded":"2024-03-19T03:32:33.109294+0000","last_fullsized":"2024-03-19T03:32:33.109294+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:37:15.261998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64206,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":178,"num_read_kb":152,"num_write":37,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425662+0000","last_change":"2024-03-19T03:30:35.652754+0000","last_active":"2024-03-19T03:31:39.425662+0000","last_peered":"2024-03-19T03:31:39.425662+0000","last_clean":"2024-03-19T03:31:39.425662+0000","last_became_active":"2024-03-19T03:30:35.652415+0000","last_became_peered":"2024-03-19T03:30:35.652415+0000","last_unstale":"2024-03-19T03:31:39.425662+0000","last_undegraded":"2024-03-19T03:31:39.425662+0000","last_fullsized":"2024-03-19T03:31:39.425662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:29:31.838302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.428363+0000","last_change":"2024-03-19T03:30:39.732885+0000","last_active":"2024-03-19T03:31:39.428363+0000","last_peered":"2024-03-19T03:31:39.428363+0000","last_clean":"2024-03-19T03:31:39.428363+0000","last_became_active":"2024-03-19T03:30:39.732449+0000","last_became_peered":"2024-03-19T03:30:39.732449+0000","last_unstale":"2024-03-19T03:31:39.428363+0000","last_undegraded":"2024-03-19T03:31:39.428363+0000","last_fullsized":"2024-03-19T03:31:39.428363+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:08:01.304805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421344+0000","last_change":"2024-03-19T03:30:34.617560+0000","last_active":"2024-03-19T03:31:39.421344+0000","last_peered":"2024-03-19T03:31:39.421344+0000","last_clean":"2024-03-19T03:31:39.421344+0000","last_became_active":"2024-03-19T03:30:34.617234+0000","last_became_peered":"2024-03-19T03:30:34.617234+0000","last_unstale":"2024-03-19T03:31:39.421344+0000","last_undegraded":"2024-03-19T03:31:39.421344+0000","last_fullsized":"2024-03-19T03:31:39.421344+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:00:46.206589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425703+0000","last_change":"2024-03-19T03:30:35.672723+0000","last_active":"2024-03-19T03:31:39.425703+0000","last_peered":"2024-03-19T03:31:39.425703+0000","last_clean":"2024-03-19T03:31:39.425703+0000","last_became_active":"2024-03-19T03:30:35.670828+0000","last_became_peered":"2024-03-19T03:30:35.670828+0000","last_unstale":"2024-03-19T03:31:39.425703+0000","last_undegraded":"2024-03-19T03:31:39.425703+0000","last_fullsized":"2024-03-19T03:31:39.425703+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:22:19.254329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426674+0000","last_change":"2024-03-19T03:30:39.738887+0000","last_active":"2024-03-19T03:31:39.426674+0000","last_peered":"2024-03-19T03:31:39.426674+0000","last_clean":"2024-03-19T03:31:39.426674+0000","last_became_active":"2024-03-19T03:30:39.738450+0000","last_became_peered":"2024-03-19T03:30:39.738450+0000","last_unstale":"2024-03-19T03:31:39.426674+0000","last_undegraded":"2024-03-19T03:31:39.426674+0000","last_fullsized":"2024-03-19T03:31:39.426674+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:23:57.550464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426018+0000","last_change":"2024-03-19T03:30:34.634810+0000","last_active":"2024-03-19T03:31:39.426018+0000","last_peered":"2024-03-19T03:31:39.426018+0000","last_clean":"2024-03-19T03:31:39.426018+0000","last_became_active":"2024-03-19T03:30:34.634404+0000","last_became_peered":"2024-03-19T03:30:34.634404+0000","last_unstale":"2024-03-19T03:31:39.426018+0000","last_undegraded":"2024-03-19T03:31:39.426018+0000","last_fullsized":"2024-03-19T03:31:39.426018+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:19:09.231175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426504+0000","last_change":"2024-03-19T03:30:35.664808+0000","last_active":"2024-03-19T03:31:39.426504+0000","last_peered":"2024-03-19T03:31:39.426504+0000","last_clean":"2024-03-19T03:31:39.426504+0000","last_became_active":"2024-03-19T03:30:35.664431+0000","last_became_peered":"2024-03-19T03:30:35.664431+0000","last_unstale":"2024-03-19T03:31:39.426504+0000","last_undegraded":"2024-03-19T03:31:39.426504+0000","last_fullsized":"2024-03-19T03:31:39.426504+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:18:47.623528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.441795+0000","last_change":"2024-03-19T03:31:39.441384+0000","last_active":"2024-03-19T03:31:39.441795+0000","last_peered":"2024-03-19T03:31:39.441795+0000","last_clean":"2024-03-19T03:31:39.441795+0000","last_became_active":"2024-03-19T03:31:39.440947+0000","last_became_peered":"2024-03-19T03:31:39.440947+0000","last_unstale":"2024-03-19T03:31:39.441795+0000","last_undegraded":"2024-03-19T03:31:39.441795+0000","last_fullsized":"2024-03-19T03:31:39.441795+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:09:33.041934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":118,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:32:30.107826+0000","last_change":"2024-03-19T03:30:34.619890+0000","last_active":"2024-03-19T03:32:30.107826+0000","last_peered":"2024-03-19T03:32:30.107826+0000","last_clean":"2024-03-19T03:32:30.107826+0000","last_became_active":"2024-03-19T03:30:34.619518+0000","last_became_peered":"2024-03-19T03:30:34.619518+0000","last_unstale":"2024-03-19T03:32:30.107826+0000","last_undegraded":"2024-03-19T03:32:30.107826+0000","last_fullsized":"2024-03-19T03:32:30.107826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T15:19:41.314321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":86,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426265+0000","last_change":"2024-03-19T03:30:35.649241+0000","last_active":"2024-03-19T03:31:39.426265+0000","last_peered":"2024-03-19T03:31:39.426265+0000","last_clean":"2024-03-19T03:31:39.426265+0000","last_became_active":"2024-03-19T03:30:35.648883+0000","last_became_peered":"2024-03-19T03:30:35.648883+0000","last_unstale":"2024-03-19T03:31:39.426265+0000","last_undegraded":"2024-03-19T03:31:39.426265+0000","last_fullsized":"2024-03-19T03:31:39.426265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:43:44.219322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.436929+0000","last_change":"2024-03-19T03:31:39.436929+0000","last_active":"2024-03-19T03:31:39.436929+0000","last_peered":"2024-03-19T03:31:39.436929+0000","last_clean":"2024-03-19T03:31:39.436929+0000","last_became_active":"2024-03-19T03:31:39.436418+0000","last_became_peered":"2024-03-19T03:31:39.436418+0000","last_unstale":"2024-03-19T03:31:39.436929+0000","last_undegraded":"2024-03-19T03:31:39.436929+0000","last_fullsized":"2024-03-19T03:31:39.436929+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:49:49.767632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426114+0000","last_change":"2024-03-19T03:30:34.635076+0000","last_active":"2024-03-19T03:31:39.426114+0000","last_peered":"2024-03-19T03:31:39.426114+0000","last_clean":"2024-03-19T03:31:39.426114+0000","last_became_active":"2024-03-19T03:30:34.634680+0000","last_became_peered":"2024-03-19T03:30:34.634680+0000","last_unstale":"2024-03-19T03:31:39.426114+0000","last_undegraded":"2024-03-19T03:31:39.426114+0000","last_fullsized":"2024-03-19T03:31:39.426114+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:16:38.874086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425553+0000","last_change":"2024-03-19T03:30:35.651912+0000","last_active":"2024-03-19T03:31:39.425553+0000","last_peered":"2024-03-19T03:31:39.425553+0000","last_clean":"2024-03-19T03:31:39.425553+0000","last_became_active":"2024-03-19T03:30:35.651563+0000","last_became_peered":"2024-03-19T03:30:35.651563+0000","last_unstale":"2024-03-19T03:31:39.425553+0000","last_undegraded":"2024-03-19T03:31:39.425553+0000","last_fullsized":"2024-03-19T03:31:39.425553+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:41:05.677230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425439+0000","last_change":"2024-03-19T03:30:39.737466+0000","last_active":"2024-03-19T03:31:39.425439+0000","last_peered":"2024-03-19T03:31:39.425439+0000","last_clean":"2024-03-19T03:31:39.425439+0000","last_became_active":"2024-03-19T03:30:39.737126+0000","last_became_peered":"2024-03-19T03:30:39.737126+0000","last_unstale":"2024-03-19T03:31:39.425439+0000","last_undegraded":"2024-03-19T03:31:39.425439+0000","last_fullsized":"2024-03-19T03:31:39.425439+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:52:25.030107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426740+0000","last_change":"2024-03-19T03:30:34.622729+0000","last_active":"2024-03-19T03:31:39.426740+0000","last_peered":"2024-03-19T03:31:39.426740+0000","last_clean":"2024-03-19T03:31:39.426740+0000","last_became_active":"2024-03-19T03:30:34.622349+0000","last_became_peered":"2024-03-19T03:30:34.622349+0000","last_unstale":"2024-03-19T03:31:39.426740+0000","last_undegraded":"2024-03-19T03:31:39.426740+0000","last_fullsized":"2024-03-19T03:31:39.426740+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:28.893702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.438224+0000","last_change":"2024-03-19T03:31:39.438224+0000","last_active":"2024-03-19T03:31:39.438224+0000","last_peered":"2024-03-19T03:31:39.438224+0000","last_clean":"2024-03-19T03:31:39.438224+0000","last_became_active":"2024-03-19T03:31:39.437764+0000","last_became_peered":"2024-03-19T03:31:39.437764+0000","last_unstale":"2024-03-19T03:31:39.438224+0000","last_undegraded":"2024-03-19T03:31:39.438224+0000","last_fullsized":"2024-03-19T03:31:39.438224+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:15:55.306306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425199+0000","last_change":"2024-03-19T03:30:39.728505+0000","last_active":"2024-03-19T03:31:39.425199+0000","last_peered":"2024-03-19T03:31:39.425199+0000","last_clean":"2024-03-19T03:31:39.425199+0000","last_became_active":"2024-03-19T03:30:39.728095+0000","last_became_peered":"2024-03-19T03:30:39.728095+0000","last_unstale":"2024-03-19T03:31:39.425199+0000","last_undegraded":"2024-03-19T03:31:39.425199+0000","last_fullsized":"2024-03-19T03:31:39.425199+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:59:44.189992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.427419+0000","last_change":"2024-03-19T03:30:34.642738+0000","last_active":"2024-03-19T03:31:39.427419+0000","last_peered":"2024-03-19T03:31:39.427419+0000","last_clean":"2024-03-19T03:31:39.427419+0000","last_became_active":"2024-03-19T03:30:34.642173+0000","last_became_peered":"2024-03-19T03:30:34.642173+0000","last_unstale":"2024-03-19T03:31:39.427419+0000","last_undegraded":"2024-03-19T03:31:39.427419+0000","last_fullsized":"2024-03-19T03:31:39.427419+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:17:51.633284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.424564+0000","last_change":"2024-03-19T03:30:35.663676+0000","last_active":"2024-03-19T03:31:39.424564+0000","last_peered":"2024-03-19T03:31:39.424564+0000","last_clean":"2024-03-19T03:31:39.424564+0000","last_became_active":"2024-03-19T03:30:35.663313+0000","last_became_peered":"2024-03-19T03:30:35.663313+0000","last_unstale":"2024-03-19T03:31:39.424564+0000","last_undegraded":"2024-03-19T03:31:39.424564+0000","last_fullsized":"2024-03-19T03:31:39.424564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:02:25.933784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425394+0000","last_change":"2024-03-19T03:30:39.744967+0000","last_active":"2024-03-19T03:31:39.425394+0000","last_peered":"2024-03-19T03:31:39.425394+0000","last_clean":"2024-03-19T03:31:39.425394+0000","last_became_active":"2024-03-19T03:30:39.744542+0000","last_became_peered":"2024-03-19T03:30:39.744542+0000","last_unstale":"2024-03-19T03:31:39.425394+0000","last_undegraded":"2024-03-19T03:31:39.425394+0000","last_fullsized":"2024-03-19T03:31:39.425394+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:56:54.699749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":114,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:32:29.143460+0000","last_change":"2024-03-19T03:30:34.622225+0000","last_active":"2024-03-19T03:32:29.143460+0000","last_peered":"2024-03-19T03:32:29.143460+0000","last_clean":"2024-03-19T03:32:29.143460+0000","last_became_active":"2024-03-19T03:30:34.621836+0000","last_became_peered":"2024-03-19T03:30:34.621836+0000","last_unstale":"2024-03-19T03:32:29.143460+0000","last_undegraded":"2024-03-19T03:32:29.143460+0000","last_fullsized":"2024-03-19T03:32:29.143460+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:20:35.854632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.423796+0000","last_change":"2024-03-19T03:30:35.675961+0000","last_active":"2024-03-19T03:31:39.423796+0000","last_peered":"2024-03-19T03:31:39.423796+0000","last_clean":"2024-03-19T03:31:39.423796+0000","last_became_active":"2024-03-19T03:30:35.675603+0000","last_became_peered":"2024-03-19T03:30:35.675603+0000","last_unstale":"2024-03-19T03:31:39.423796+0000","last_undegraded":"2024-03-19T03:31:39.423796+0000","last_fullsized":"2024-03-19T03:31:39.423796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:39:47.429027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.433889+0000","last_change":"2024-03-19T03:30:39.746747+0000","last_active":"2024-03-19T03:31:39.433889+0000","last_peered":"2024-03-19T03:31:39.433889+0000","last_clean":"2024-03-19T03:31:39.433889+0000","last_became_active":"2024-03-19T03:30:39.745553+0000","last_became_peered":"2024-03-19T03:30:39.745553+0000","last_unstale":"2024-03-19T03:31:39.433889+0000","last_undegraded":"2024-03-19T03:31:39.433889+0000","last_fullsized":"2024-03-19T03:31:39.433889+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:14:56.211061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426189+0000","last_change":"2024-03-19T03:30:35.649795+0000","last_active":"2024-03-19T03:31:39.426189+0000","last_peered":"2024-03-19T03:31:39.426189+0000","last_clean":"2024-03-19T03:31:39.426189+0000","last_became_active":"2024-03-19T03:30:35.649417+0000","last_became_peered":"2024-03-19T03:30:35.649417+0000","last_unstale":"2024-03-19T03:31:39.426189+0000","last_undegraded":"2024-03-19T03:31:39.426189+0000","last_fullsized":"2024-03-19T03:31:39.426189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:37:24.065817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":115,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:32:29.106289+0000","last_change":"2024-03-19T03:30:34.637446+0000","last_active":"2024-03-19T03:32:29.106289+0000","last_peered":"2024-03-19T03:32:29.106289+0000","last_clean":"2024-03-19T03:32:29.106289+0000","last_became_active":"2024-03-19T03:30:34.637083+0000","last_became_peered":"2024-03-19T03:30:34.637083+0000","last_unstale":"2024-03-19T03:32:29.106289+0000","last_undegraded":"2024-03-19T03:32:29.106289+0000","last_fullsized":"2024-03-19T03:32:29.106289+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:51:02.090890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":83,"num_read_kb":83,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.438484+0000","last_change":"2024-03-19T03:31:39.438484+0000","last_active":"2024-03-19T03:31:39.438484+0000","last_peered":"2024-03-19T03:31:39.438484+0000","last_clean":"2024-03-19T03:31:39.438484+0000","last_became_active":"2024-03-19T03:31:39.433824+0000","last_became_peered":"2024-03-19T03:31:39.433824+0000","last_unstale":"2024-03-19T03:31:39.438484+0000","last_undegraded":"2024-03-19T03:31:39.438484+0000","last_fullsized":"2024-03-19T03:31:39.438484+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:58:52.691327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421595+0000","last_change":"2024-03-19T03:30:35.670002+0000","last_active":"2024-03-19T03:31:39.421595+0000","last_peered":"2024-03-19T03:31:39.421595+0000","last_clean":"2024-03-19T03:31:39.421595+0000","last_became_active":"2024-03-19T03:30:35.653208+0000","last_became_peered":"2024-03-19T03:30:35.653208+0000","last_unstale":"2024-03-19T03:31:39.421595+0000","last_undegraded":"2024-03-19T03:31:39.421595+0000","last_fullsized":"2024-03-19T03:31:39.421595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:37:15.246977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.422878+0000","last_change":"2024-03-19T03:30:34.625107+0000","last_active":"2024-03-19T03:31:39.422878+0000","last_peered":"2024-03-19T03:31:39.422878+0000","last_clean":"2024-03-19T03:31:39.422878+0000","last_became_active":"2024-03-19T03:30:34.624711+0000","last_became_peered":"2024-03-19T03:30:34.624711+0000","last_unstale":"2024-03-19T03:31:39.422878+0000","last_undegraded":"2024-03-19T03:31:39.422878+0000","last_fullsized":"2024-03-19T03:31:39.422878+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:39:33.880100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":351,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426188+0000","last_change":"2024-03-19T03:29:11.374136+0000","last_active":"2024-03-19T03:31:39.426188+0000","last_peered":"2024-03-19T03:31:39.426188+0000","last_clean":"2024-03-19T03:31:39.426188+0000","last_became_active":"2024-03-19T03:29:11.373691+0000","last_became_peered":"2024-03-19T03:29:11.373691+0000","last_unstale":"2024-03-19T03:31:39.426188+0000","last_undegraded":"2024-03-19T03:31:39.426188+0000","last_fullsized":"2024-03-19T03:31:39.426188+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:25:19.274380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:25:19.274380+0000","last_clean_scrub_stamp":"2024-03-19T03:25:19.274380+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:29:59.731772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.428371+0000","last_change":"2024-03-19T03:30:39.744391+0000","last_active":"2024-03-19T03:31:39.428371+0000","last_peered":"2024-03-19T03:31:39.428371+0000","last_clean":"2024-03-19T03:31:39.428371+0000","last_became_active":"2024-03-19T03:30:39.743995+0000","last_became_peered":"2024-03-19T03:30:39.743995+0000","last_unstale":"2024-03-19T03:31:39.428371+0000","last_undegraded":"2024-03-19T03:31:39.428371+0000","last_fullsized":"2024-03-19T03:31:39.428371+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T13:41:02.714132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425174+0000","last_change":"2024-03-19T03:30:35.662528+0000","last_active":"2024-03-19T03:31:39.425174+0000","last_peered":"2024-03-19T03:31:39.425174+0000","last_clean":"2024-03-19T03:31:39.425174+0000","last_became_active":"2024-03-19T03:30:35.659782+0000","last_became_peered":"2024-03-19T03:30:35.659782+0000","last_unstale":"2024-03-19T03:31:39.425174+0000","last_undegraded":"2024-03-19T03:31:39.425174+0000","last_fullsized":"2024-03-19T03:31:39.425174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:18:12.838234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426195+0000","last_change":"2024-03-19T03:30:34.620988+0000","last_active":"2024-03-19T03:31:39.426195+0000","last_peered":"2024-03-19T03:31:39.426195+0000","last_clean":"2024-03-19T03:31:39.426195+0000","last_became_active":"2024-03-19T03:30:34.620411+0000","last_became_peered":"2024-03-19T03:30:34.620411+0000","last_unstale":"2024-03-19T03:31:39.426195+0000","last_undegraded":"2024-03-19T03:31:39.426195+0000","last_fullsized":"2024-03-19T03:31:39.426195+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:37:13.275121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426353+0000","last_change":"2024-03-19T03:30:39.733325+0000","last_active":"2024-03-19T03:31:39.426353+0000","last_peered":"2024-03-19T03:31:39.426353+0000","last_clean":"2024-03-19T03:31:39.426353+0000","last_became_active":"2024-03-19T03:30:39.732553+0000","last_became_peered":"2024-03-19T03:30:39.732553+0000","last_unstale":"2024-03-19T03:31:39.426353+0000","last_undegraded":"2024-03-19T03:31:39.426353+0000","last_fullsized":"2024-03-19T03:31:39.426353+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:52:47.403673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425596+0000","last_change":"2024-03-19T03:30:34.637100+0000","last_active":"2024-03-19T03:31:39.425596+0000","last_peered":"2024-03-19T03:31:39.425596+0000","last_clean":"2024-03-19T03:31:39.425596+0000","last_became_active":"2024-03-19T03:30:34.636715+0000","last_became_peered":"2024-03-19T03:30:34.636715+0000","last_unstale":"2024-03-19T03:31:39.425596+0000","last_undegraded":"2024-03-19T03:31:39.425596+0000","last_fullsized":"2024-03-19T03:31:39.425596+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:48:16.212158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426337+0000","last_change":"2024-03-19T03:30:35.668405+0000","last_active":"2024-03-19T03:31:39.426337+0000","last_peered":"2024-03-19T03:31:39.426337+0000","last_clean":"2024-03-19T03:31:39.426337+0000","last_became_active":"2024-03-19T03:30:35.667972+0000","last_became_peered":"2024-03-19T03:30:35.667972+0000","last_unstale":"2024-03-19T03:31:39.426337+0000","last_undegraded":"2024-03-19T03:31:39.426337+0000","last_fullsized":"2024-03-19T03:31:39.426337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:01:52.816110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.420806+0000","last_change":"2024-03-19T03:30:39.747535+0000","last_active":"2024-03-19T03:31:39.420806+0000","last_peered":"2024-03-19T03:31:39.420806+0000","last_clean":"2024-03-19T03:31:39.420806+0000","last_became_active":"2024-03-19T03:30:39.747122+0000","last_became_peered":"2024-03-19T03:30:39.747122+0000","last_unstale":"2024-03-19T03:31:39.420806+0000","last_undegraded":"2024-03-19T03:31:39.420806+0000","last_fullsized":"2024-03-19T03:31:39.420806+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:07:35.642809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'13","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.428383+0000","last_change":"2024-03-19T03:30:34.637768+0000","last_active":"2024-03-19T03:31:39.428383+0000","last_peered":"2024-03-19T03:31:39.428383+0000","last_clean":"2024-03-19T03:31:39.428383+0000","last_became_active":"2024-03-19T03:30:34.637431+0000","last_became_peered":"2024-03-19T03:30:34.637431+0000","last_unstale":"2024-03-19T03:31:39.428383+0000","last_undegraded":"2024-03-19T03:31:39.428383+0000","last_fullsized":"2024-03-19T03:31:39.428383+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:22:50.487209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.423453+0000","last_change":"2024-03-19T03:30:35.661352+0000","last_active":"2024-03-19T03:31:39.423453+0000","last_peered":"2024-03-19T03:31:39.423453+0000","last_clean":"2024-03-19T03:31:39.423453+0000","last_became_active":"2024-03-19T03:30:35.658656+0000","last_became_peered":"2024-03-19T03:30:35.658656+0000","last_unstale":"2024-03-19T03:31:39.423453+0000","last_undegraded":"2024-03-19T03:31:39.423453+0000","last_fullsized":"2024-03-19T03:31:39.423453+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:51:36.532748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.420837+0000","last_change":"2024-03-19T03:30:39.724637+0000","last_active":"2024-03-19T03:31:39.420837+0000","last_peered":"2024-03-19T03:31:39.420837+0000","last_clean":"2024-03-19T03:31:39.420837+0000","last_became_active":"2024-03-19T03:30:39.723842+0000","last_became_peered":"2024-03-19T03:30:39.723842+0000","last_unstale":"2024-03-19T03:31:39.420837+0000","last_undegraded":"2024-03-19T03:31:39.420837+0000","last_fullsized":"2024-03-19T03:31:39.420837+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:04:02.446185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.423197+0000","last_change":"2024-03-19T03:30:34.620404+0000","last_active":"2024-03-19T03:31:39.423197+0000","last_peered":"2024-03-19T03:31:39.423197+0000","last_clean":"2024-03-19T03:31:39.423197+0000","last_became_active":"2024-03-19T03:30:34.620056+0000","last_became_peered":"2024-03-19T03:30:34.620056+0000","last_unstale":"2024-03-19T03:31:39.423197+0000","last_undegraded":"2024-03-19T03:31:39.423197+0000","last_fullsized":"2024-03-19T03:31:39.423197+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:54:42.203912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.435485+0000","last_change":"2024-03-19T03:30:35.686618+0000","last_active":"2024-03-19T03:31:39.435485+0000","last_peered":"2024-03-19T03:31:39.435485+0000","last_clean":"2024-03-19T03:31:39.435485+0000","last_became_active":"2024-03-19T03:30:35.685814+0000","last_became_peered":"2024-03-19T03:30:35.685814+0000","last_unstale":"2024-03-19T03:31:39.435485+0000","last_undegraded":"2024-03-19T03:31:39.435485+0000","last_fullsized":"2024-03-19T03:31:39.435485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:08.401307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.423269+0000","last_change":"2024-03-19T03:30:39.744039+0000","last_active":"2024-03-19T03:31:39.423269+0000","last_peered":"2024-03-19T03:31:39.423269+0000","last_clean":"2024-03-19T03:31:39.423269+0000","last_became_active":"2024-03-19T03:30:39.743637+0000","last_became_peered":"2024-03-19T03:30:39.743637+0000","last_unstale":"2024-03-19T03:31:39.423269+0000","last_undegraded":"2024-03-19T03:31:39.423269+0000","last_fullsized":"2024-03-19T03:31:39.423269+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:02:22.498333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.437167+0000","last_change":"2024-03-19T03:30:34.629059+0000","last_active":"2024-03-19T03:31:39.437167+0000","last_peered":"2024-03-19T03:31:39.437167+0000","last_clean":"2024-03-19T03:31:39.437167+0000","last_became_active":"2024-03-19T03:30:34.628617+0000","last_became_peered":"2024-03-19T03:30:34.628617+0000","last_unstale":"2024-03-19T03:31:39.437167+0000","last_undegraded":"2024-03-19T03:31:39.437167+0000","last_fullsized":"2024-03-19T03:31:39.437167+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:47:09.064502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425034+0000","last_change":"2024-03-19T03:30:35.650256+0000","last_active":"2024-03-19T03:31:39.425034+0000","last_peered":"2024-03-19T03:31:39.425034+0000","last_clean":"2024-03-19T03:31:39.425034+0000","last_became_active":"2024-03-19T03:30:35.649086+0000","last_became_peered":"2024-03-19T03:30:35.649086+0000","last_unstale":"2024-03-19T03:31:39.425034+0000","last_undegraded":"2024-03-19T03:31:39.425034+0000","last_fullsized":"2024-03-19T03:31:39.425034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:00:49.204307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.429819+0000","last_change":"2024-03-19T03:31:39.429381+0000","last_active":"2024-03-19T03:31:39.429819+0000","last_peered":"2024-03-19T03:31:39.429819+0000","last_clean":"2024-03-19T03:31:39.429819+0000","last_became_active":"2024-03-19T03:31:39.428915+0000","last_became_peered":"2024-03-19T03:31:39.428915+0000","last_unstale":"2024-03-19T03:31:39.429819+0000","last_undegraded":"2024-03-19T03:31:39.429819+0000","last_fullsized":"2024-03-19T03:31:39.429819+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:20.121002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,5],"acting":[10,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421146+0000","last_change":"2024-03-19T03:30:34.634935+0000","last_active":"2024-03-19T03:31:39.421146+0000","last_peered":"2024-03-19T03:31:39.421146+0000","last_clean":"2024-03-19T03:31:39.421146+0000","last_became_active":"2024-03-19T03:30:34.634602+0000","last_became_peered":"2024-03-19T03:30:34.634602+0000","last_unstale":"2024-03-19T03:31:39.421146+0000","last_undegraded":"2024-03-19T03:31:39.421146+0000","last_fullsized":"2024-03-19T03:31:39.421146+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:16:57.158105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425439+0000","last_change":"2024-03-19T03:30:35.675650+0000","last_active":"2024-03-19T03:31:39.425439+0000","last_peered":"2024-03-19T03:31:39.425439+0000","last_clean":"2024-03-19T03:31:39.425439+0000","last_became_active":"2024-03-19T03:30:35.675255+0000","last_became_peered":"2024-03-19T03:30:35.675255+0000","last_unstale":"2024-03-19T03:31:39.425439+0000","last_undegraded":"2024-03-19T03:31:39.425439+0000","last_fullsized":"2024-03-19T03:31:39.425439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:55:03.201526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.422982+0000","last_change":"2024-03-19T03:30:39.733413+0000","last_active":"2024-03-19T03:31:39.422982+0000","last_peered":"2024-03-19T03:31:39.422982+0000","last_clean":"2024-03-19T03:31:39.422982+0000","last_became_active":"2024-03-19T03:30:39.732998+0000","last_became_peered":"2024-03-19T03:30:39.732998+0000","last_unstale":"2024-03-19T03:31:39.422982+0000","last_undegraded":"2024-03-19T03:31:39.422982+0000","last_fullsized":"2024-03-19T03:31:39.422982+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:07:39.512355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421028+0000","last_change":"2024-03-19T03:30:34.623749+0000","last_active":"2024-03-19T03:31:39.421028+0000","last_peered":"2024-03-19T03:31:39.421028+0000","last_clean":"2024-03-19T03:31:39.421028+0000","last_became_active":"2024-03-19T03:30:34.623378+0000","last_became_peered":"2024-03-19T03:30:34.623378+0000","last_unstale":"2024-03-19T03:31:39.421028+0000","last_undegraded":"2024-03-19T03:31:39.421028+0000","last_fullsized":"2024-03-19T03:31:39.421028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:06:01.939685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421693+0000","last_change":"2024-03-19T03:30:35.685702+0000","last_active":"2024-03-19T03:31:39.421693+0000","last_peered":"2024-03-19T03:31:39.421693+0000","last_clean":"2024-03-19T03:31:39.421693+0000","last_became_active":"2024-03-19T03:30:35.683340+0000","last_became_peered":"2024-03-19T03:30:35.683340+0000","last_unstale":"2024-03-19T03:31:39.421693+0000","last_undegraded":"2024-03-19T03:31:39.421693+0000","last_fullsized":"2024-03-19T03:31:39.421693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:47:12.246255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.422983+0000","last_change":"2024-03-19T03:30:39.726747+0000","last_active":"2024-03-19T03:31:39.422983+0000","last_peered":"2024-03-19T03:31:39.422983+0000","last_clean":"2024-03-19T03:31:39.422983+0000","last_became_active":"2024-03-19T03:30:39.726236+0000","last_became_peered":"2024-03-19T03:30:39.726236+0000","last_unstale":"2024-03-19T03:31:39.422983+0000","last_undegraded":"2024-03-19T03:31:39.422983+0000","last_fullsized":"2024-03-19T03:31:39.422983+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:30:02.333544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425294+0000","last_change":"2024-03-19T03:30:34.627828+0000","last_active":"2024-03-19T03:31:39.425294+0000","last_peered":"2024-03-19T03:31:39.425294+0000","last_clean":"2024-03-19T03:31:39.425294+0000","last_became_active":"2024-03-19T03:30:34.627290+0000","last_became_peered":"2024-03-19T03:30:34.627290+0000","last_unstale":"2024-03-19T03:31:39.425294+0000","last_undegraded":"2024-03-19T03:31:39.425294+0000","last_fullsized":"2024-03-19T03:31:39.425294+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:07:50.140087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426829+0000","last_change":"2024-03-19T03:30:35.661180+0000","last_active":"2024-03-19T03:31:39.426829+0000","last_peered":"2024-03-19T03:31:39.426829+0000","last_clean":"2024-03-19T03:31:39.426829+0000","last_became_active":"2024-03-19T03:30:35.660797+0000","last_became_peered":"2024-03-19T03:30:35.660797+0000","last_unstale":"2024-03-19T03:31:39.426829+0000","last_undegraded":"2024-03-19T03:31:39.426829+0000","last_fullsized":"2024-03-19T03:31:39.426829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:54:36.034281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.437278+0000","last_change":"2024-03-19T03:30:39.732832+0000","last_active":"2024-03-19T03:31:39.437278+0000","last_peered":"2024-03-19T03:31:39.437278+0000","last_clean":"2024-03-19T03:31:39.437278+0000","last_became_active":"2024-03-19T03:30:39.732434+0000","last_became_peered":"2024-03-19T03:30:39.732434+0000","last_unstale":"2024-03-19T03:31:39.437278+0000","last_undegraded":"2024-03-19T03:31:39.437278+0000","last_fullsized":"2024-03-19T03:31:39.437278+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:45:49.737361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":120,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:32:33.109911+0000","last_change":"2024-03-19T03:30:34.625655+0000","last_active":"2024-03-19T03:32:33.109911+0000","last_peered":"2024-03-19T03:32:33.109911+0000","last_clean":"2024-03-19T03:32:33.109911+0000","last_became_active":"2024-03-19T03:30:34.625251+0000","last_became_peered":"2024-03-19T03:30:34.625251+0000","last_unstale":"2024-03-19T03:32:33.109911+0000","last_undegraded":"2024-03-19T03:32:33.109911+0000","last_fullsized":"2024-03-19T03:32:33.109911+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:30:25.592670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426972+0000","last_change":"2024-03-19T03:30:35.651147+0000","last_active":"2024-03-19T03:31:39.426972+0000","last_peered":"2024-03-19T03:31:39.426972+0000","last_clean":"2024-03-19T03:31:39.426972+0000","last_became_active":"2024-03-19T03:30:35.650817+0000","last_became_peered":"2024-03-19T03:30:35.650817+0000","last_unstale":"2024-03-19T03:31:39.426972+0000","last_undegraded":"2024-03-19T03:31:39.426972+0000","last_fullsized":"2024-03-19T03:31:39.426972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:28:35.529226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.434246+0000","last_change":"2024-03-19T03:30:39.749895+0000","last_active":"2024-03-19T03:31:39.434246+0000","last_peered":"2024-03-19T03:31:39.434246+0000","last_clean":"2024-03-19T03:31:39.434246+0000","last_became_active":"2024-03-19T03:30:39.749555+0000","last_became_peered":"2024-03-19T03:30:39.749555+0000","last_unstale":"2024-03-19T03:31:39.434246+0000","last_undegraded":"2024-03-19T03:31:39.434246+0000","last_fullsized":"2024-03-19T03:31:39.434246+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:10:14.383271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.427995+0000","last_change":"2024-03-19T03:30:34.635456+0000","last_active":"2024-03-19T03:31:39.427995+0000","last_peered":"2024-03-19T03:31:39.427995+0000","last_clean":"2024-03-19T03:31:39.427995+0000","last_became_active":"2024-03-19T03:30:34.635099+0000","last_became_peered":"2024-03-19T03:30:34.635099+0000","last_unstale":"2024-03-19T03:31:39.427995+0000","last_undegraded":"2024-03-19T03:31:39.427995+0000","last_fullsized":"2024-03-19T03:31:39.427995+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:39:30.083678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426720+0000","last_change":"2024-03-19T03:30:35.659490+0000","last_active":"2024-03-19T03:31:39.426720+0000","last_peered":"2024-03-19T03:31:39.426720+0000","last_clean":"2024-03-19T03:31:39.426720+0000","last_became_active":"2024-03-19T03:30:35.659208+0000","last_became_peered":"2024-03-19T03:30:35.659208+0000","last_unstale":"2024-03-19T03:31:39.426720+0000","last_undegraded":"2024-03-19T03:31:39.426720+0000","last_fullsized":"2024-03-19T03:31:39.426720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:45:18.788981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.424978+0000","last_change":"2024-03-19T03:30:39.725193+0000","last_active":"2024-03-19T03:31:39.424978+0000","last_peered":"2024-03-19T03:31:39.424978+0000","last_clean":"2024-03-19T03:31:39.424978+0000","last_became_active":"2024-03-19T03:30:39.724745+0000","last_became_peered":"2024-03-19T03:30:39.724745+0000","last_unstale":"2024-03-19T03:31:39.424978+0000","last_undegraded":"2024-03-19T03:31:39.424978+0000","last_fullsized":"2024-03-19T03:31:39.424978+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:38.686031+0000","last_clean_scrub_stamp":"2024-03-19T03:30:38.686031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:30:07.621852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.436049+0000","last_change":"2024-03-19T03:30:34.636532+0000","last_active":"2024-03-19T03:31:39.436049+0000","last_peered":"2024-03-19T03:31:39.436049+0000","last_clean":"2024-03-19T03:31:39.436049+0000","last_became_active":"2024-03-19T03:30:34.634104+0000","last_became_peered":"2024-03-19T03:30:34.634104+0000","last_unstale":"2024-03-19T03:31:39.436049+0000","last_undegraded":"2024-03-19T03:31:39.436049+0000","last_fullsized":"2024-03-19T03:31:39.436049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:33.564938+0000","last_clean_scrub_stamp":"2024-03-19T03:30:33.564938+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:41:18.308628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426815+0000","last_change":"2024-03-19T03:30:35.653036+0000","last_active":"2024-03-19T03:31:39.426815+0000","last_peered":"2024-03-19T03:31:39.426815+0000","last_clean":"2024-03-19T03:31:39.426815+0000","last_became_active":"2024-03-19T03:30:35.652564+0000","last_became_peered":"2024-03-19T03:30:35.652564+0000","last_unstale":"2024-03-19T03:31:39.426815+0000","last_undegraded":"2024-03-19T03:31:39.426815+0000","last_fullsized":"2024-03-19T03:31:39.426815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:01.812525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425554+0000","last_change":"2024-03-19T03:30:35.650724+0000","last_active":"2024-03-19T03:31:39.425554+0000","last_peered":"2024-03-19T03:31:39.425554+0000","last_clean":"2024-03-19T03:31:39.425554+0000","last_became_active":"2024-03-19T03:30:35.650204+0000","last_became_peered":"2024-03-19T03:30:35.650204+0000","last_unstale":"2024-03-19T03:31:39.425554+0000","last_undegraded":"2024-03-19T03:31:39.425554+0000","last_fullsized":"2024-03-19T03:31:39.425554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:06:56.159133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421190+0000","last_change":"2024-03-19T03:30:35.650241+0000","last_active":"2024-03-19T03:31:39.421190+0000","last_peered":"2024-03-19T03:31:39.421190+0000","last_clean":"2024-03-19T03:31:39.421190+0000","last_became_active":"2024-03-19T03:30:35.649901+0000","last_became_peered":"2024-03-19T03:30:35.649901+0000","last_unstale":"2024-03-19T03:31:39.421190+0000","last_undegraded":"2024-03-19T03:31:39.421190+0000","last_fullsized":"2024-03-19T03:31:39.421190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:35:53.333403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421107+0000","last_change":"2024-03-19T03:30:35.679267+0000","last_active":"2024-03-19T03:31:39.421107+0000","last_peered":"2024-03-19T03:31:39.421107+0000","last_clean":"2024-03-19T03:31:39.421107+0000","last_became_active":"2024-03-19T03:30:35.678871+0000","last_became_peered":"2024-03-19T03:30:35.678871+0000","last_unstale":"2024-03-19T03:31:39.421107+0000","last_undegraded":"2024-03-19T03:31:39.421107+0000","last_fullsized":"2024-03-19T03:31:39.421107+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:23:17.681854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.434732+0000","last_change":"2024-03-19T03:30:35.683060+0000","last_active":"2024-03-19T03:31:39.434732+0000","last_peered":"2024-03-19T03:31:39.434732+0000","last_clean":"2024-03-19T03:31:39.434732+0000","last_became_active":"2024-03-19T03:30:35.676872+0000","last_became_peered":"2024-03-19T03:30:35.676872+0000","last_unstale":"2024-03-19T03:31:39.434732+0000","last_undegraded":"2024-03-19T03:31:39.434732+0000","last_fullsized":"2024-03-19T03:31:39.434732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T15:13:51.530914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426459+0000","last_change":"2024-03-19T03:30:35.682685+0000","last_active":"2024-03-19T03:31:39.426459+0000","last_peered":"2024-03-19T03:31:39.426459+0000","last_clean":"2024-03-19T03:31:39.426459+0000","last_became_active":"2024-03-19T03:30:35.680309+0000","last_became_peered":"2024-03-19T03:30:35.680309+0000","last_unstale":"2024-03-19T03:31:39.426459+0000","last_undegraded":"2024-03-19T03:31:39.426459+0000","last_fullsized":"2024-03-19T03:31:39.426459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:31:59.925521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426932+0000","last_change":"2024-03-19T03:30:35.652656+0000","last_active":"2024-03-19T03:31:39.426932+0000","last_peered":"2024-03-19T03:31:39.426932+0000","last_clean":"2024-03-19T03:31:39.426932+0000","last_became_active":"2024-03-19T03:30:35.650744+0000","last_became_peered":"2024-03-19T03:30:35.650744+0000","last_unstale":"2024-03-19T03:31:39.426932+0000","last_undegraded":"2024-03-19T03:31:39.426932+0000","last_fullsized":"2024-03-19T03:31:39.426932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:49:01.450719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421665+0000","last_change":"2024-03-19T03:30:35.650092+0000","last_active":"2024-03-19T03:31:39.421665+0000","last_peered":"2024-03-19T03:31:39.421665+0000","last_clean":"2024-03-19T03:31:39.421665+0000","last_became_active":"2024-03-19T03:30:35.649727+0000","last_became_peered":"2024-03-19T03:30:35.649727+0000","last_unstale":"2024-03-19T03:31:39.421665+0000","last_undegraded":"2024-03-19T03:31:39.421665+0000","last_fullsized":"2024-03-19T03:31:39.421665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:22:13.968226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.434943+0000","last_change":"2024-03-19T03:31:39.431670+0000","last_active":"2024-03-19T03:31:39.434943+0000","last_peered":"2024-03-19T03:31:39.434943+0000","last_clean":"2024-03-19T03:31:39.434943+0000","last_became_active":"2024-03-19T03:31:39.431299+0000","last_became_peered":"2024-03-19T03:31:39.431299+0000","last_unstale":"2024-03-19T03:31:39.434943+0000","last_undegraded":"2024-03-19T03:31:39.434943+0000","last_fullsized":"2024-03-19T03:31:39.434943+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:04:51.530636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426030+0000","last_change":"2024-03-19T03:30:35.660289+0000","last_active":"2024-03-19T03:31:39.426030+0000","last_peered":"2024-03-19T03:31:39.426030+0000","last_clean":"2024-03-19T03:31:39.426030+0000","last_became_active":"2024-03-19T03:30:35.660005+0000","last_became_peered":"2024-03-19T03:30:35.660005+0000","last_unstale":"2024-03-19T03:31:39.426030+0000","last_undegraded":"2024-03-19T03:31:39.426030+0000","last_fullsized":"2024-03-19T03:31:39.426030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:36:16.033629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.421015+0000","last_change":"2024-03-19T03:30:35.660461+0000","last_active":"2024-03-19T03:31:39.421015+0000","last_peered":"2024-03-19T03:31:39.421015+0000","last_clean":"2024-03-19T03:31:39.421015+0000","last_became_active":"2024-03-19T03:30:35.660058+0000","last_became_peered":"2024-03-19T03:30:35.660058+0000","last_unstale":"2024-03-19T03:31:39.421015+0000","last_undegraded":"2024-03-19T03:31:39.421015+0000","last_fullsized":"2024-03-19T03:31:39.421015+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:08:40.326314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.437395+0000","last_change":"2024-03-19T03:30:35.665634+0000","last_active":"2024-03-19T03:31:39.437395+0000","last_peered":"2024-03-19T03:31:39.437395+0000","last_clean":"2024-03-19T03:31:39.437395+0000","last_became_active":"2024-03-19T03:30:35.665264+0000","last_became_peered":"2024-03-19T03:30:35.665264+0000","last_unstale":"2024-03-19T03:31:39.437395+0000","last_undegraded":"2024-03-19T03:31:39.437395+0000","last_fullsized":"2024-03-19T03:31:39.437395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:33:56.931412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.425568+0000","last_change":"2024-03-19T03:30:35.681887+0000","last_active":"2024-03-19T03:31:39.425568+0000","last_peered":"2024-03-19T03:31:39.425568+0000","last_clean":"2024-03-19T03:31:39.425568+0000","last_became_active":"2024-03-19T03:30:35.680332+0000","last_became_peered":"2024-03-19T03:30:35.680332+0000","last_unstale":"2024-03-19T03:31:39.425568+0000","last_undegraded":"2024-03-19T03:31:39.425568+0000","last_fullsized":"2024-03-19T03:31:39.425568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:50:24.243641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-19T03:31:39.426932+0000","last_change":"2024-03-19T03:30:35.652988+0000","last_active":"2024-03-19T03:31:39.426932+0000","last_peered":"2024-03-19T03:31:39.426932+0000","last_clean":"2024-03-19T03:31:39.426932+0000","last_became_active":"2024-03-19T03:30:35.652608+0000","last_became_peered":"2024-03-19T03:30:35.652608+0000","last_unstale":"2024-03-19T03:31:39.426932+0000","last_undegraded":"2024-03-19T03:31:39.426932+0000","last_fullsized":"2024-03-19T03:31:39.426932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T03:30:34.599101+0000","last_clean_scrub_stamp":"2024-03-19T03:30:34.599101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T14:44:35.619094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283260,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71410,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":873,"num_read_kb":515,"num_write":244,"num_write_kb":142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":214230,"data_compressed":5478,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":212,"ondisk_log_size":212,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383720,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1003520,"data_stored":501378,"data_compressed":20673,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645356,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28148,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717004,"statfs":{"total":95995035648,"available":95966212096,"internally_reserved":0,"allocated":1150976,"data_stored":1152791,"data_compressed":58523,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808882,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":505404,"data_compressed":20673,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070519,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":505782,"data_compressed":20673,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332156,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1077248,"data_stored":1098574,"data_compressed":56697,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495683,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1003520,"data_stored":496812,"data_compressed":20673,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1069056,"data_stored":562048,"data_compressed":22499,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":498268,"data_compressed":20673,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1089536,"data_stored":562659,"data_compressed":22499,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":496520,"data_compressed":20673,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1011712,"data_stored":501277,"data_compressed":20673,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757320,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1081344,"data_stored":1089669,"data_compressed":56697,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34380,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34380,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34380,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36024,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36024,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36024,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36024,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36024,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":65958,"data_compressed":1826,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":66569,"data_compressed":1826,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":66333,"data_compressed":1826,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T03:32:35.981 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-19T03:32:35.981 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-19T03:32:35.981 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-19T03:32:35.981 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-19T03:32:36.458 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:32:36.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:36 smithi119 ceph-mon[66861]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T03:32:36.769 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:36 smithi117 ceph-mon[31838]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T03:32:36.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:36 smithi178 ceph-mon[67578]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T03:32:37.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:37 smithi119 ceph-mon[66861]: pgmap v380: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:32:37.763 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:37 smithi117 ceph-mon[31838]: pgmap v380: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:32:37.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:37 smithi178 ceph-mon[67578]: pgmap v380: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:32:38.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.619+0000 7f55cb200700 1 -- 172.21.15.117:0/3329340125 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55c4103410 msgr2=0x7f55c41037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:38.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.619+0000 7f55cb200700 1 --2- 172.21.15.117:0/3329340125 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55c4103410 0x7f55c41037f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f55b8009990 tx=0x7f55b8009ca0 comp rx=0 tx=0).stop 2024-03-19T03:32:38.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.619+0000 7f55cb200700 1 -- 172.21.15.117:0/3329340125 shutdown_connections 2024-03-19T03:32:38.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.619+0000 7f55cb200700 1 --2- 172.21.15.117:0/3329340125 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55c410c7f0 0x7f55c410ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:38.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.619+0000 7f55cb200700 1 --2- 172.21.15.117:0/3329340125 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55c4103410 0x7f55c41037f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:38.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.619+0000 7f55cb200700 1 --2- 172.21.15.117:0/3329340125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55c4103d30 0x7f55c410c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:38.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.619+0000 7f55cb200700 1 -- 172.21.15.117:0/3329340125 >> 172.21.15.117:0/3329340125 conn(0x7f55c40fcbe0 msgr2=0x7f55c40ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:38.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.619+0000 7f55cb200700 1 -- 172.21.15.117:0/3329340125 shutdown_connections 2024-03-19T03:32:38.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.619+0000 7f55cb200700 1 -- 172.21.15.117:0/3329340125 wait complete. 2024-03-19T03:32:38.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.621+0000 7f55cb200700 1 Processor -- start 2024-03-19T03:32:38.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.621+0000 7f55cb200700 1 -- start start 2024-03-19T03:32:38.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.621+0000 7f55cb200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55c4103410 0x7f55c410b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:38.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.621+0000 7f55cb200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55c4103d30 0x7f55c410baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:38.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.621+0000 7f55cb200700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55c410c7f0 0x7f55c41a38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:38.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.621+0000 7f55cb200700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f55c4112580 con 0x7f55c4103410 2024-03-19T03:32:38.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.621+0000 7f55cb200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f55c4112400 con 0x7f55c4103d30 2024-03-19T03:32:38.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.621+0000 7f55cb200700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f55c4112700 con 0x7f55c410c7f0 2024-03-19T03:32:38.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.622+0000 7f55c9400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55c4103d30 0x7f55c410baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:38.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.622+0000 7f55c9400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55c4103d30 0x7f55c410baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:34344/0 (socket says 172.21.15.117:34344) 2024-03-19T03:32:38.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.622+0000 7f55c9400700 1 -- 172.21.15.117:0/3249996110 learned_addr learned my addr 172.21.15.117:0/3249996110 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:38.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.622+0000 7f55c9e00700 1 --2- 172.21.15.117:0/3249996110 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55c4103410 0x7f55c410b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:38.625 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.622+0000 7f55ca800700 1 --2- 172.21.15.117:0/3249996110 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55c410c7f0 0x7f55c41a38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:38.625 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.622+0000 7f55c9e00700 1 -- 172.21.15.117:0/3249996110 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55c410c7f0 msgr2=0x7f55c41a38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:38.625 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.622+0000 7f55c9e00700 1 --2- 172.21.15.117:0/3249996110 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55c410c7f0 0x7f55c41a38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:38.625 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.622+0000 7f55c9e00700 1 -- 172.21.15.117:0/3249996110 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55c4103d30 msgr2=0x7f55c410baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:38.625 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.622+0000 7f55c9e00700 1 --2- 172.21.15.117:0/3249996110 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55c4103d30 0x7f55c410baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:38.625 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.622+0000 7f55c9e00700 1 -- 172.21.15.117:0/3249996110 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55c41a3ff0 con 0x7f55c4103410 2024-03-19T03:32:38.625 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.623+0000 7f55c9e00700 1 --2- 172.21.15.117:0/3249996110 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55c4103410 0x7f55c410b560 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f55b8000c00 tx=0x7f55b800bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:38.625 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.623+0000 7f55bea00700 1 -- 172.21.15.117:0/3249996110 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b8026070 con 0x7f55c4103410 2024-03-19T03:32:38.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.623+0000 7f55cb200700 1 -- 172.21.15.117:0/3249996110 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55c41a4280 con 0x7f55c4103410 2024-03-19T03:32:38.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.623+0000 7f55bea00700 1 -- 172.21.15.117:0/3249996110 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55b8003d70 con 0x7f55c4103410 2024-03-19T03:32:38.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.623+0000 7f55bea00700 1 -- 172.21.15.117:0/3249996110 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b8021410 con 0x7f55c4103410 2024-03-19T03:32:38.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.623+0000 7f55cb200700 1 -- 172.21.15.117:0/3249996110 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55c41abb90 con 0x7f55c4103410 2024-03-19T03:32:38.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.625+0000 7f55b3e00700 1 -- 172.21.15.117:0/3249996110 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55a0005330 con 0x7f55c4103410 2024-03-19T03:32:38.632 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.627+0000 7f55bea00700 1 -- 172.21.15.117:0/3249996110 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f55b8029070 con 0x7f55c4103410 2024-03-19T03:32:38.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.629+0000 7f55bea00700 1 --2- 172.21.15.117:0/3249996110 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f55ac0a29a0 0x7f55ac0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:38.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.630+0000 7f55bea00700 1 -- 172.21.15.117:0/3249996110 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f55b80cfc80 con 0x7f55c4103410 2024-03-19T03:32:38.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.630+0000 7f55c9400700 1 --2- 172.21.15.117:0/3249996110 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f55ac0a29a0 0x7f55ac0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:38.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.630+0000 7f55c9400700 1 --2- 172.21.15.117:0/3249996110 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f55ac0a29a0 0x7f55ac0a4e60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f55c410e090 tx=0x7f55b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:38.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.633+0000 7f55bea00700 1 -- 172.21.15.117:0/3249996110 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f55b805d740 con 0x7f55c4103410 2024-03-19T03:32:38.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.912+0000 7f55b3e00700 1 -- 172.21.15.117:0/3249996110 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f55a00058f0 con 0x7f55c4103410 2024-03-19T03:32:38.913 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:32:38.913 INFO:teuthology.orchestra.run.smithi117.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-19T03:32:38.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.913+0000 7f55bea00700 1 -- 172.21.15.117:0/3249996110 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f55b8013070 con 0x7f55c4103410 2024-03-19T03:32:38.915 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.916+0000 7f55b3e00700 1 -- 172.21.15.117:0/3249996110 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f55ac0a29a0 msgr2=0x7f55ac0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:38.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.916+0000 7f55b3e00700 1 --2- 172.21.15.117:0/3249996110 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f55ac0a29a0 0x7f55ac0a4e60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f55c410e090 tx=0x7f55b4009260 comp rx=0 tx=0).stop 2024-03-19T03:32:38.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.916+0000 7f55b3e00700 1 -- 172.21.15.117:0/3249996110 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55c4103410 msgr2=0x7f55c410b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:38.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.916+0000 7f55b3e00700 1 --2- 172.21.15.117:0/3249996110 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55c4103410 0x7f55c410b560 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f55b8000c00 tx=0x7f55b800bab0 comp rx=0 tx=0).stop 2024-03-19T03:32:38.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.916+0000 7f55b3e00700 1 -- 172.21.15.117:0/3249996110 shutdown_connections 2024-03-19T03:32:38.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.916+0000 7f55b3e00700 1 --2- 172.21.15.117:0/3249996110 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f55ac0a29a0 0x7f55ac0a4e60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:38.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.916+0000 7f55b3e00700 1 --2- 172.21.15.117:0/3249996110 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55c410c7f0 0x7f55c41a38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:38.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.916+0000 7f55b3e00700 1 --2- 172.21.15.117:0/3249996110 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55c4103410 0x7f55c410b560 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:38.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.916+0000 7f55b3e00700 1 --2- 172.21.15.117:0/3249996110 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55c4103d30 0x7f55c410baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:38.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.916+0000 7f55b3e00700 1 -- 172.21.15.117:0/3249996110 >> 172.21.15.117:0/3249996110 conn(0x7f55c40fcbe0 msgr2=0x7f55c410ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:38.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.916+0000 7f55b3e00700 1 -- 172.21.15.117:0/3249996110 shutdown_connections 2024-03-19T03:32:38.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:38.916+0000 7f55b3e00700 1 -- 172.21.15.117:0/3249996110 wait complete. 2024-03-19T03:32:39.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:39 smithi117 ceph-mon[31838]: pgmap v381: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:32:39.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-19T03:32:39.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-03-19T03:32:39.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-03-19T03:32:39.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-19T03:32:39.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [1, 9]}]: dispatch 2024-03-19T03:32:39.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]: dispatch 2024-03-19T03:32:39.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:39 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3249996110' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-19T03:32:39.646 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-19T03:32:39.646 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:32:39.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:39 smithi119 ceph-mon[66861]: pgmap v381: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:32:39.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-19T03:32:39.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-03-19T03:32:39.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-03-19T03:32:39.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-19T03:32:39.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [1, 9]}]: dispatch 2024-03-19T03:32:39.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]: dispatch 2024-03-19T03:32:39.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:39 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3249996110' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-19T03:32:39.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.843+0000 7f968a000640 1 -- 172.21.15.117:0/867811704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9684101690 msgr2=0x7f9684101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:39.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.843+0000 7f968a000640 1 --2- 172.21.15.117:0/867811704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9684101690 0x7f9684101ab0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9678009900 tx=0x7f967802bc30 comp rx=0 tx=0).stop 2024-03-19T03:32:39.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.843+0000 7f968a000640 1 -- 172.21.15.117:0/867811704 shutdown_connections 2024-03-19T03:32:39.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.843+0000 7f968a000640 1 --2- 172.21.15.117:0/867811704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9684103240 0x7f9684109ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:39.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.843+0000 7f968a000640 1 --2- 172.21.15.117:0/867811704 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9684102880 0x7f9684102d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:39.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.843+0000 7f968a000640 1 --2- 172.21.15.117:0/867811704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9684101690 0x7f9684101ab0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:39.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.843+0000 7f968a000640 1 -- 172.21.15.117:0/867811704 >> 172.21.15.117:0/867811704 conn(0x7f96840fcc00 msgr2=0x7f96840ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:39.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.843+0000 7f968a000640 1 -- 172.21.15.117:0/867811704 shutdown_connections 2024-03-19T03:32:39.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.843+0000 7f968a000640 1 -- 172.21.15.117:0/867811704 wait complete. 2024-03-19T03:32:39.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:39 smithi178 ceph-mon[67578]: pgmap v381: 129 pgs: 129 active+clean; 646 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:32:39.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-19T03:32:39.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-03-19T03:32:39.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-03-19T03:32:39.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-19T03:32:39.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [1, 9]}]: dispatch 2024-03-19T03:32:39.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]: dispatch 2024-03-19T03:32:39.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:39 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3249996110' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-19T03:32:39.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.844+0000 7f968a000640 1 Processor -- start 2024-03-19T03:32:39.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.844+0000 7f968a000640 1 -- start start 2024-03-19T03:32:39.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.844+0000 7f968a000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9684101690 0x7f96840755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:39.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.844+0000 7f968a000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9684102880 0x7f9684073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:39.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.844+0000 7f968a000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9684103240 0x7f9684074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:39.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.844+0000 7f968a000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f968410d2e0 con 0x7f9684102880 2024-03-19T03:32:39.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.844+0000 7f968a000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f968410d160 con 0x7f9684103240 2024-03-19T03:32:39.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.844+0000 7f968a000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f968410d460 con 0x7f9684101690 2024-03-19T03:32:39.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.845+0000 7f9689600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9684103240 0x7f9684074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:39.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.845+0000 7f9683e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9684102880 0x7f9684073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:39.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.845+0000 7f9688c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9684101690 0x7f96840755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:39.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.845+0000 7f9689600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9684103240 0x7f9684074150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:34354/0 (socket says 172.21.15.117:34354) 2024-03-19T03:32:39.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.845+0000 7f9689600640 1 -- 172.21.15.117:0/2746312766 learned_addr learned my addr 172.21.15.117:0/2746312766 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:39.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.845+0000 7f9689600640 1 -- 172.21.15.117:0/2746312766 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9684101690 msgr2=0x7f96840755d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:39.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.845+0000 7f9689600640 1 --2- 172.21.15.117:0/2746312766 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9684101690 0x7f96840755d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:39.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.845+0000 7f9689600640 1 -- 172.21.15.117:0/2746312766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9684102880 msgr2=0x7f9684073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:39.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.845+0000 7f9689600640 1 --2- 172.21.15.117:0/2746312766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9684102880 0x7f9684073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:39.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.845+0000 7f9689600640 1 -- 172.21.15.117:0/2746312766 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9684074690 con 0x7f9684103240 2024-03-19T03:32:39.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.845+0000 7f9689600640 1 --2- 172.21.15.117:0/2746312766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9684103240 0x7f9684074150 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f967c00eb10 tx=0x7f967c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:39.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.846+0000 7f9681600640 1 -- 172.21.15.117:0/2746312766 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f967c004b20 con 0x7f9684103240 2024-03-19T03:32:39.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.846+0000 7f968a000640 1 -- 172.21.15.117:0/2746312766 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9684074980 con 0x7f9684103240 2024-03-19T03:32:39.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.846+0000 7f9681600640 1 -- 172.21.15.117:0/2746312766 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f967c004cc0 con 0x7f9684103240 2024-03-19T03:32:39.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.846+0000 7f9681600640 1 -- 172.21.15.117:0/2746312766 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f967c010690 con 0x7f9684103240 2024-03-19T03:32:39.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.846+0000 7f968a000640 1 -- 172.21.15.117:0/2746312766 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96841a6490 con 0x7f9684103240 2024-03-19T03:32:39.849 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.847+0000 7f968a000640 1 -- 172.21.15.117:0/2746312766 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9684108a30 con 0x7f9684103240 2024-03-19T03:32:39.850 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.849+0000 7f9681600640 1 -- 172.21.15.117:0/2746312766 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f967c005000 con 0x7f9684103240 2024-03-19T03:32:39.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.850+0000 7f9681600640 1 --2- 172.21.15.117:0/2746312766 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f966c0a2880 0x7f966c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:39.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.850+0000 7f9681600640 1 -- 172.21.15.117:0/2746312766 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9245+0+0 (secure 0 0 0) 0x7f967c0cf3e0 con 0x7f9684103240 2024-03-19T03:32:39.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.850+0000 7f9688c00640 1 --2- 172.21.15.117:0/2746312766 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f966c0a2880 0x7f966c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:39.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.851+0000 7f9688c00640 1 --2- 172.21.15.117:0/2746312766 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f966c0a2880 0x7f966c0a4d40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9678000c00 tx=0x7f9678005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:39.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:39.858+0000 7f9681600640 1 -- 172.21.15.117:0/2746312766 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f967c09c030 con 0x7f9684103240 2024-03-19T03:32:40.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.108+0000 7f968a000640 1 -- 172.21.15.117:0/2746312766 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9684062420 con 0x7f9684103240 2024-03-19T03:32:40.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.111+0000 7f9681600640 1 -- 172.21.15.117:0/2746312766 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24989 (secure 0 0 0) 0x7f967c09b930 con 0x7f9684103240 2024-03-19T03:32:40.112 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:32:40.112 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":92,"fsid":"a3a82456-e59f-11ee-95c9-87774f69a715","created":"2024-03-19T03:21:30.427136+0000","modified":"2024-03-19T03:32:39.437577+0000","last_up_change":"2024-03-19T03:29:32.967505+0000","last_in_change":"2024-03-19T03:29:12.848740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:25:18.824270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T03:30:33.137170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T03:30:34.598838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T03:30:38.230023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b32eb0ee-9d4b-4748-87ec-eda02c907415","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6803","nonce":3014934900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6805","nonce":3014934900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6809","nonce":3014934900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6807","nonce":3014934900}]},"public_addr":"172.21.15.117:6803/3014934900","cluster_addr":"172.21.15.117:6805/3014934900","heartbeat_back_addr":"172.21.15.117:6809/3014934900","heartbeat_front_addr":"172.21.15.117:6807/3014934900","state":["exists","up"]},{"osd":1,"uuid":"e07585be-c016-4466-b07c-abbc22a58de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6801","nonce":2425927341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6803","nonce":2425927341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6807","nonce":2425927341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6805","nonce":2425927341}]},"public_addr":"172.21.15.119:6801/2425927341","cluster_addr":"172.21.15.119:6803/2425927341","heartbeat_back_addr":"172.21.15.119:6807/2425927341","heartbeat_front_addr":"172.21.15.119:6805/2425927341","state":["exists","up"]},{"osd":2,"uuid":"ede9edda-3b9f-41ca-ad04-5dcb738ab757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6801","nonce":3733812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6803","nonce":3733812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6807","nonce":3733812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6805","nonce":3733812989}]},"public_addr":"172.21.15.178:6801/3733812989","cluster_addr":"172.21.15.178:6803/3733812989","heartbeat_back_addr":"172.21.15.178:6807/3733812989","heartbeat_front_addr":"172.21.15.178:6805/3733812989","state":["exists","up"]},{"osd":3,"uuid":"f81c41e1-3a69-40db-a90f-58151286b081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6811","nonce":2178310135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6813","nonce":2178310135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6817","nonce":2178310135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6815","nonce":2178310135}]},"public_addr":"172.21.15.117:6811/2178310135","cluster_addr":"172.21.15.117:6813/2178310135","heartbeat_back_addr":"172.21.15.117:6817/2178310135","heartbeat_front_addr":"172.21.15.117:6815/2178310135","state":["exists","up"]},{"osd":4,"uuid":"caf7ce86-caf0-4eaf-bbab-bd2d93e63c33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6809","nonce":3455398086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6811","nonce":3455398086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6815","nonce":3455398086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6813","nonce":3455398086}]},"public_addr":"172.21.15.119:6809/3455398086","cluster_addr":"172.21.15.119:6811/3455398086","heartbeat_back_addr":"172.21.15.119:6815/3455398086","heartbeat_front_addr":"172.21.15.119:6813/3455398086","state":["exists","up"]},{"osd":5,"uuid":"20ba0b8a-2e0f-4ea7-b435-430e44766200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6809","nonce":1259631912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6811","nonce":1259631912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6815","nonce":1259631912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6813","nonce":1259631912}]},"public_addr":"172.21.15.178:6809/1259631912","cluster_addr":"172.21.15.178:6811/1259631912","heartbeat_back_addr":"172.21.15.178:6815/1259631912","heartbeat_front_addr":"172.21.15.178:6813/1259631912","state":["exists","up"]},{"osd":6,"uuid":"75c2ddd7-f837-42e0-ac47-8cfbcf97a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6819","nonce":864146757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6821","nonce":864146757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6825","nonce":864146757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6823","nonce":864146757}]},"public_addr":"172.21.15.117:6819/864146757","cluster_addr":"172.21.15.117:6821/864146757","heartbeat_back_addr":"172.21.15.117:6825/864146757","heartbeat_front_addr":"172.21.15.117:6823/864146757","state":["exists","up"]},{"osd":7,"uuid":"929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6817","nonce":1273365545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6819","nonce":1273365545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6823","nonce":1273365545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6821","nonce":1273365545}]},"public_addr":"172.21.15.119:6817/1273365545","cluster_addr":"172.21.15.119:6819/1273365545","heartbeat_back_addr":"172.21.15.119:6823/1273365545","heartbeat_front_addr":"172.21.15.119:6821/1273365545","state":["exists","up"]},{"osd":8,"uuid":"fd371461-a538-4781-92e1-5eead4ca3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6817","nonce":1059587501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6819","nonce":1059587501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6823","nonce":1059587501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6821","nonce":1059587501}]},"public_addr":"172.21.15.178:6817/1059587501","cluster_addr":"172.21.15.178:6819/1059587501","heartbeat_back_addr":"172.21.15.178:6823/1059587501","heartbeat_front_addr":"172.21.15.178:6821/1059587501","state":["exists","up"]},{"osd":9,"uuid":"0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6827","nonce":3653875600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6829","nonce":3653875600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6833","nonce":3653875600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6831","nonce":3653875600}]},"public_addr":"172.21.15.117:6827/3653875600","cluster_addr":"172.21.15.117:6829/3653875600","heartbeat_back_addr":"172.21.15.117:6833/3653875600","heartbeat_front_addr":"172.21.15.117:6831/3653875600","state":["exists","up"]},{"osd":10,"uuid":"de2049da-d19e-4386-afbf-fd6b5fddbcba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6825","nonce":1050976194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6827","nonce":1050976194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6831","nonce":1050976194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6829","nonce":1050976194}]},"public_addr":"172.21.15.119:6825/1050976194","cluster_addr":"172.21.15.119:6827/1050976194","heartbeat_back_addr":"172.21.15.119:6831/1050976194","heartbeat_front_addr":"172.21.15.119:6829/1050976194","state":["exists","up"]},{"osd":11,"uuid":"3cccc0f9-d273-4b6b-9b8e-c926371472dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6825","nonce":2915080368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6827","nonce":2915080368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6831","nonce":2915080368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6829","nonce":2915080368}]},"public_addr":"172.21.15.178:6825/2915080368","cluster_addr":"172.21.15.178:6827/2915080368","heartbeat_back_addr":"172.21.15.178:6831/2915080368","heartbeat_front_addr":"172.21.15.178:6829/2915080368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:27.876274+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:53.094706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:16.598813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:48.160419+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:14.048925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:38.814304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:12.542328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:38.942747+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:04.488963+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:38.196053+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:05.254728+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:30.874915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7},{"from":5,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.117:0/185684447":"2024-03-20T03:22:38.134578+0000","172.21.15.117:0/361367101":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/2788067603":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3506200675":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6800/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3283471541":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3750920169":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/2780813298":"2024-03-20T03:22:03.905461+0000","172.21.15.117:6800/2788067603":"2024-03-20T03:22:38.134578+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:32:40.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.113+0000 7f968a000640 1 -- 172.21.15.117:0/2746312766 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f966c0a2880 msgr2=0x7f966c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:40.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.114+0000 7f968a000640 1 --2- 172.21.15.117:0/2746312766 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f966c0a2880 0x7f966c0a4d40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9678000c00 tx=0x7f9678005fd0 comp rx=0 tx=0).stop 2024-03-19T03:32:40.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.114+0000 7f968a000640 1 -- 172.21.15.117:0/2746312766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9684103240 msgr2=0x7f9684074150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:40.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.114+0000 7f968a000640 1 --2- 172.21.15.117:0/2746312766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9684103240 0x7f9684074150 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f967c00eb10 tx=0x7f967c00c2d0 comp rx=0 tx=0).stop 2024-03-19T03:32:40.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.114+0000 7f968a000640 1 -- 172.21.15.117:0/2746312766 shutdown_connections 2024-03-19T03:32:40.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.114+0000 7f968a000640 1 --2- 172.21.15.117:0/2746312766 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f966c0a2880 0x7f966c0a4d40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:40.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.114+0000 7f968a000640 1 --2- 172.21.15.117:0/2746312766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9684103240 0x7f9684074150 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:40.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.114+0000 7f968a000640 1 --2- 172.21.15.117:0/2746312766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9684102880 0x7f9684073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:40.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.114+0000 7f968a000640 1 --2- 172.21.15.117:0/2746312766 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9684101690 0x7f96840755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:40.116 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.114+0000 7f968a000640 1 -- 172.21.15.117:0/2746312766 >> 172.21.15.117:0/2746312766 conn(0x7f96840fcc00 msgr2=0x7f9684105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:40.116 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.114+0000 7f968a000640 1 -- 172.21.15.117:0/2746312766 shutdown_connections 2024-03-19T03:32:40.116 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.114+0000 7f968a000640 1 -- 172.21.15.117:0/2746312766 wait complete. 2024-03-19T03:32:40.136 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T03:25:18.824270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T03:30:33.137170+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T03:30:34.598838+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T03:30:38.230023+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T03:32:40.136 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T03:32:40.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.323+0000 7f0039a00640 1 -- 172.21.15.117:0/2668712963 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0034107d40 msgr2=0x7f0034108160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:40.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.323+0000 7f0039a00640 1 --2- 172.21.15.117:0/2668712963 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0034107d40 0x7f0034108160 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0024009900 tx=0x7f002402bc40 comp rx=0 tx=0).stop 2024-03-19T03:32:40.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.324+0000 7f0039a00640 1 -- 172.21.15.117:0/2668712963 shutdown_connections 2024-03-19T03:32:40.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.324+0000 7f0039a00640 1 --2- 172.21.15.117:0/2668712963 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00341098f0 0x7f0034110180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:40.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.324+0000 7f0039a00640 1 --2- 172.21.15.117:0/2668712963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0034108f30 0x7f00341093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:40.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.324+0000 7f0039a00640 1 --2- 172.21.15.117:0/2668712963 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0034107d40 0x7f0034108160 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:40.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.324+0000 7f0039a00640 1 -- 172.21.15.117:0/2668712963 >> 172.21.15.117:0/2668712963 conn(0x7f0034076cf0 msgr2=0x7f0034079110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:40.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.324+0000 7f0039a00640 1 -- 172.21.15.117:0/2668712963 shutdown_connections 2024-03-19T03:32:40.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.324+0000 7f0039a00640 1 -- 172.21.15.117:0/2668712963 wait complete. 2024-03-19T03:32:40.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.325+0000 7f0039a00640 1 Processor -- start 2024-03-19T03:32:40.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.325+0000 7f0039a00640 1 -- start start 2024-03-19T03:32:40.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.325+0000 7f0039a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0034107d40 0x7f003419e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:40.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.325+0000 7f0039a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0034108f30 0x7f003419e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:40.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.325+0000 7f0039a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00341098f0 0x7f00341a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:40.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.325+0000 7f0039a00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f00341139e0 con 0x7f00341098f0 2024-03-19T03:32:40.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.325+0000 7f0039a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0034113860 con 0x7f0034108f30 2024-03-19T03:32:40.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.325+0000 7f0039a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f0034113b60 con 0x7f0034107d40 2024-03-19T03:32:40.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.326+0000 7f0033400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0034108f30 0x7f003419e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:40.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.326+0000 7f0033400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0034108f30 0x7f003419e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:34360/0 (socket says 172.21.15.117:34360) 2024-03-19T03:32:40.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.326+0000 7f0033400640 1 -- 172.21.15.117:0/3603520736 learned_addr learned my addr 172.21.15.117:0/3603520736 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:40.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.326+0000 7f0039000640 1 --2- 172.21.15.117:0/3603520736 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00341098f0 0x7f00341a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:40.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.326+0000 7f0033e00640 1 --2- 172.21.15.117:0/3603520736 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0034107d40 0x7f003419e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:40.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.326+0000 7f0033400640 1 -- 172.21.15.117:0/3603520736 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0034107d40 msgr2=0x7f003419e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:40.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.326+0000 7f0033400640 1 --2- 172.21.15.117:0/3603520736 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0034107d40 0x7f003419e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:40.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.326+0000 7f0033400640 1 -- 172.21.15.117:0/3603520736 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00341098f0 msgr2=0x7f00341a55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:40.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.326+0000 7f0033400640 1 --2- 172.21.15.117:0/3603520736 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00341098f0 0x7f00341a55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:40.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.326+0000 7f0033400640 1 -- 172.21.15.117:0/3603520736 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00341a5c20 con 0x7f0034108f30 2024-03-19T03:32:40.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.326+0000 7f0033400640 1 --2- 172.21.15.117:0/3603520736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0034108f30 0x7f003419e540 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f002000d950 tx=0x7f002000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:40.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.326+0000 7f0030c00640 1 -- 172.21.15.117:0/3603520736 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020017070 con 0x7f0034108f30 2024-03-19T03:32:40.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.327+0000 7f0030c00640 1 -- 172.21.15.117:0/3603520736 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0020009c30 con 0x7f0034108f30 2024-03-19T03:32:40.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.327+0000 7f0030c00640 1 -- 172.21.15.117:0/3603520736 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002000f6b0 con 0x7f0034108f30 2024-03-19T03:32:40.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.327+0000 7f0039a00640 1 -- 172.21.15.117:0/3603520736 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00341a5f10 con 0x7f0034108f30 2024-03-19T03:32:40.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.327+0000 7f0039a00640 1 -- 172.21.15.117:0/3603520736 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00341a6380 con 0x7f0034108f30 2024-03-19T03:32:40.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.328+0000 7f0039a00640 1 -- 172.21.15.117:0/3603520736 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f003410f060 con 0x7f0034108f30 2024-03-19T03:32:40.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.331+0000 7f0030c00640 1 -- 172.21.15.117:0/3603520736 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f00200109d0 con 0x7f0034108f30 2024-03-19T03:32:40.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.332+0000 7f0030c00640 1 --2- 172.21.15.117:0/3603520736 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f00180a2880 0x7f00180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:40.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.332+0000 7f0030c00640 1 -- 172.21.15.117:0/3603520736 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9245+0+0 (secure 0 0 0) 0x7f00200ce2f0 con 0x7f0034108f30 2024-03-19T03:32:40.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.332+0000 7f0033e00640 1 --2- 172.21.15.117:0/3603520736 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f00180a2880 0x7f00180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:40.333 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.333+0000 7f0033e00640 1 --2- 172.21.15.117:0/3603520736 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f00180a2880 0x7f00180a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f002402c180 tx=0x7f002402cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:40.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.336+0000 7f0030c00640 1 -- 172.21.15.117:0/3603520736 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0020009de0 con 0x7f0034108f30 2024-03-19T03:32:40.468 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:40 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-19T03:32:40.468 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:40 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-03-19T03:32:40.468 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:40 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-03-19T03:32:40.468 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:40 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-19T03:32:40.468 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:40 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [1, 9]}]': finished 2024-03-19T03:32:40.469 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:40 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]': finished 2024-03-19T03:32:40.469 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:40 smithi117 ceph-mon[31838]: osdmap e92: 12 total, 12 up, 12 in 2024-03-19T03:32:40.469 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:40 smithi117 ceph-mon[31838]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-19T03:32:40.469 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:40 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2746312766' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:32:40.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.540+0000 7f0039a00640 1 -- 172.21.15.117:0/3603520736 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0034062420 con 0x7f0034108f30 2024-03-19T03:32:40.541 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.541+0000 7f0030c00640 1 -- 172.21.15.117:0/3603520736 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f002009a8f0 con 0x7f0034108f30 2024-03-19T03:32:40.542 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 1 2024-03-19T03:32:40.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.543+0000 7f0039a00640 1 -- 172.21.15.117:0/3603520736 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f00180a2880 msgr2=0x7f00180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:40.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.543+0000 7f0039a00640 1 --2- 172.21.15.117:0/3603520736 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f00180a2880 0x7f00180a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f002402c180 tx=0x7f002402cf90 comp rx=0 tx=0).stop 2024-03-19T03:32:40.544 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.543+0000 7f0039a00640 1 -- 172.21.15.117:0/3603520736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0034108f30 msgr2=0x7f003419e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:40.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.543+0000 7f0039a00640 1 --2- 172.21.15.117:0/3603520736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0034108f30 0x7f003419e540 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f002000d950 tx=0x7f002000de10 comp rx=0 tx=0).stop 2024-03-19T03:32:40.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.543+0000 7f0039a00640 1 -- 172.21.15.117:0/3603520736 shutdown_connections 2024-03-19T03:32:40.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.543+0000 7f0039a00640 1 --2- 172.21.15.117:0/3603520736 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00341098f0 0x7f00341a55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:40.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.543+0000 7f0039a00640 1 --2- 172.21.15.117:0/3603520736 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f00180a2880 0x7f00180a4d40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:40.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.543+0000 7f0039a00640 1 --2- 172.21.15.117:0/3603520736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0034108f30 0x7f003419e540 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:40.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.543+0000 7f0039a00640 1 --2- 172.21.15.117:0/3603520736 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0034107d40 0x7f003419e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:40.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.543+0000 7f0039a00640 1 -- 172.21.15.117:0/3603520736 >> 172.21.15.117:0/3603520736 conn(0x7f0034076cf0 msgr2=0x7f003410c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:40.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.543+0000 7f0039a00640 1 -- 172.21.15.117:0/3603520736 shutdown_connections 2024-03-19T03:32:40.545 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.543+0000 7f0039a00640 1 -- 172.21.15.117:0/3603520736 wait complete. 2024-03-19T03:32:40.562 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T03:32:40.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:40 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-19T03:32:40.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:40 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-03-19T03:32:40.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:40 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-03-19T03:32:40.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:40 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-19T03:32:40.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:40 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [1, 9]}]': finished 2024-03-19T03:32:40.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:40 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]': finished 2024-03-19T03:32:40.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:40 smithi119 ceph-mon[66861]: osdmap e92: 12 total, 12 up, 12 in 2024-03-19T03:32:40.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:40 smithi119 ceph-mon[66861]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-19T03:32:40.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:40 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2746312766' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:32:40.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.741+0000 7f0f7a200640 1 -- 172.21.15.117:0/2460970417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f74101720 msgr2=0x7f0f74101b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:40.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.741+0000 7f0f7a200640 1 --2- 172.21.15.117:0/2460970417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f74101720 0x7f0f74101b40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0f68009900 tx=0x7f0f6802bc40 comp rx=0 tx=0).stop 2024-03-19T03:32:40.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.741+0000 7f0f7a200640 1 -- 172.21.15.117:0/2460970417 shutdown_connections 2024-03-19T03:32:40.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.741+0000 7f0f7a200640 1 --2- 172.21.15.117:0/2460970417 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f741032d0 0x7f0f74109b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:40.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.741+0000 7f0f7a200640 1 --2- 172.21.15.117:0/2460970417 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f74102910 0x7f0f74102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:40.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.741+0000 7f0f7a200640 1 --2- 172.21.15.117:0/2460970417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f74101720 0x7f0f74101b40 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:40.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.741+0000 7f0f7a200640 1 -- 172.21.15.117:0/2460970417 >> 172.21.15.117:0/2460970417 conn(0x7f0f740fcc90 msgr2=0x7f0f740ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:40.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.741+0000 7f0f7a200640 1 -- 172.21.15.117:0/2460970417 shutdown_connections 2024-03-19T03:32:40.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.741+0000 7f0f7a200640 1 -- 172.21.15.117:0/2460970417 wait complete. 2024-03-19T03:32:40.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.742+0000 7f0f7a200640 1 Processor -- start 2024-03-19T03:32:40.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.742+0000 7f0f7a200640 1 -- start start 2024-03-19T03:32:40.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.743+0000 7f0f7a200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f74101720 0x7f0f74199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:40.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.743+0000 7f0f7a200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f74102910 0x7f0f7419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:40.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.743+0000 7f0f7a200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f741032d0 0x7f0f741a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:40.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.743+0000 7f0f7a200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7410d3c0 con 0x7f0f74101720 2024-03-19T03:32:40.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.743+0000 7f0f7a200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7410d240 con 0x7f0f741032d0 2024-03-19T03:32:40.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.743+0000 7f0f7a200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7410d540 con 0x7f0f74102910 2024-03-19T03:32:40.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.743+0000 7f0f79800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f741032d0 0x7f0f741a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:40.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.743+0000 7f0f79800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f741032d0 0x7f0f741a1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:34376/0 (socket says 172.21.15.117:34376) 2024-03-19T03:32:40.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.743+0000 7f0f79800640 1 -- 172.21.15.117:0/1155500252 learned_addr learned my addr 172.21.15.117:0/1155500252 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:40.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.743+0000 7f0f78e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f74101720 0x7f0f74199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:40.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.743+0000 7f0f73e00640 1 --2- 172.21.15.117:0/1155500252 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f74102910 0x7f0f7419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:40.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.743+0000 7f0f79800640 1 -- 172.21.15.117:0/1155500252 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f74102910 msgr2=0x7f0f7419a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:40.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.743+0000 7f0f79800640 1 --2- 172.21.15.117:0/1155500252 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f74102910 0x7f0f7419a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:40.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.743+0000 7f0f79800640 1 -- 172.21.15.117:0/1155500252 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f74101720 msgr2=0x7f0f74199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:40.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.743+0000 7f0f79800640 1 --2- 172.21.15.117:0/1155500252 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f74101720 0x7f0f74199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:40.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.743+0000 7f0f79800640 1 -- 172.21.15.117:0/1155500252 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f741a18b0 con 0x7f0f741032d0 2024-03-19T03:32:40.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.744+0000 7f0f79800640 1 --2- 172.21.15.117:0/1155500252 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f741032d0 0x7f0f741a1210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c009fd0 tx=0x7f0f6c00b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:40.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.744+0000 7f0f71600640 1 -- 172.21.15.117:0/1155500252 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c017070 con 0x7f0f741032d0 2024-03-19T03:32:40.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.744+0000 7f0f7a200640 1 -- 172.21.15.117:0/1155500252 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f741a1ba0 con 0x7f0f741032d0 2024-03-19T03:32:40.746 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.744+0000 7f0f71600640 1 -- 172.21.15.117:0/1155500252 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f6c003e50 con 0x7f0f741032d0 2024-03-19T03:32:40.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.744+0000 7f0f71600640 1 -- 172.21.15.117:0/1155500252 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c0106c0 con 0x7f0f741032d0 2024-03-19T03:32:40.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.744+0000 7f0f7a200640 1 -- 172.21.15.117:0/1155500252 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f741a2070 con 0x7f0f741032d0 2024-03-19T03:32:40.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.745+0000 7f0f7a200640 1 -- 172.21.15.117:0/1155500252 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f741a2240 con 0x7f0f741032d0 2024-03-19T03:32:40.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.747+0000 7f0f71600640 1 -- 172.21.15.117:0/1155500252 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f0f6c003970 con 0x7f0f741032d0 2024-03-19T03:32:40.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.748+0000 7f0f71600640 1 --2- 172.21.15.117:0/1155500252 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0f5c0a2800 0x7f0f5c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:40.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.749+0000 7f0f71600640 1 -- 172.21.15.117:0/1155500252 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f0f6c0cea90 con 0x7f0f741032d0 2024-03-19T03:32:40.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.749+0000 7f0f78e00640 1 --2- 172.21.15.117:0/1155500252 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0f5c0a2800 0x7f0f5c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:40.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.750+0000 7f0f78e00640 1 --2- 172.21.15.117:0/1155500252 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0f5c0a2800 0x7f0f5c0a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0f6802c180 tx=0x7f0f6802cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:40.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:40.753+0000 7f0f71600640 1 -- 172.21.15.117:0/1155500252 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0f6c09b090 con 0x7f0f741032d0 2024-03-19T03:32:40.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:40 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-19T03:32:40.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:40 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-03-19T03:32:40.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:40 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-03-19T03:32:40.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:40 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-19T03:32:40.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:40 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [1, 9]}]': finished 2024-03-19T03:32:40.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:40 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]': finished 2024-03-19T03:32:40.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:40 smithi178 ceph-mon[67578]: osdmap e92: 12 total, 12 up, 12 in 2024-03-19T03:32:40.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:40 smithi178 ceph-mon[67578]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-19T03:32:40.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:40 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2746312766' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:32:41.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.011+0000 7f0f7a200640 1 -- 172.21.15.117:0/1155500252 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0f741a2410 con 0x7f0f741032d0 2024-03-19T03:32:41.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.012+0000 7f0f71600640 1 -- 172.21.15.117:0/1155500252 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f0f6c09f130 con 0x7f0f741032d0 2024-03-19T03:32:41.013 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 32 2024-03-19T03:32:41.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.014+0000 7f0f7a200640 1 -- 172.21.15.117:0/1155500252 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0f5c0a2800 msgr2=0x7f0f5c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:41.015 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.014+0000 7f0f7a200640 1 --2- 172.21.15.117:0/1155500252 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0f5c0a2800 0x7f0f5c0a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0f6802c180 tx=0x7f0f6802cf90 comp rx=0 tx=0).stop 2024-03-19T03:32:41.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.014+0000 7f0f7a200640 1 -- 172.21.15.117:0/1155500252 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f741032d0 msgr2=0x7f0f741a1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:41.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.014+0000 7f0f7a200640 1 --2- 172.21.15.117:0/1155500252 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f741032d0 0x7f0f741a1210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c009fd0 tx=0x7f0f6c00b260 comp rx=0 tx=0).stop 2024-03-19T03:32:41.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.014+0000 7f0f7a200640 1 -- 172.21.15.117:0/1155500252 shutdown_connections 2024-03-19T03:32:41.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.014+0000 7f0f7a200640 1 --2- 172.21.15.117:0/1155500252 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0f5c0a2800 0x7f0f5c0a4cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:41.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.014+0000 7f0f7a200640 1 --2- 172.21.15.117:0/1155500252 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f741032d0 0x7f0f741a1210 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:41.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.014+0000 7f0f7a200640 1 --2- 172.21.15.117:0/1155500252 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f74102910 0x7f0f7419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:41.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.014+0000 7f0f7a200640 1 --2- 172.21.15.117:0/1155500252 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f74101720 0x7f0f74199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:41.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.014+0000 7f0f7a200640 1 -- 172.21.15.117:0/1155500252 >> 172.21.15.117:0/1155500252 conn(0x7f0f740fcc90 msgr2=0x7f0f74073a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:41.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.014+0000 7f0f7a200640 1 -- 172.21.15.117:0/1155500252 shutdown_connections 2024-03-19T03:32:41.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.015+0000 7f0f7a200640 1 -- 172.21.15.117:0/1155500252 wait complete. 2024-03-19T03:32:41.040 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T03:32:41.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.238+0000 7f1351600640 1 -- 172.21.15.117:0/936445335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f134c0761e0 msgr2=0x7f134c076600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:41.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.238+0000 7f1351600640 1 --2- 172.21.15.117:0/936445335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f134c0761e0 0x7f134c076600 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f133c009900 tx=0x7f133c02bc30 comp rx=0 tx=0).stop 2024-03-19T03:32:41.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.239+0000 7f1351600640 1 -- 172.21.15.117:0/936445335 shutdown_connections 2024-03-19T03:32:41.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.239+0000 7f1351600640 1 --2- 172.21.15.117:0/936445335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f134c077dc0 0x7f134c078660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:41.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.239+0000 7f1351600640 1 --2- 172.21.15.117:0/936445335 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f134c077420 0x7f134c077880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:41.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.239+0000 7f1351600640 1 --2- 172.21.15.117:0/936445335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f134c0761e0 0x7f134c076600 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:41.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.239+0000 7f1351600640 1 -- 172.21.15.117:0/936445335 >> 172.21.15.117:0/936445335 conn(0x7f134c0fefd0 msgr2=0x7f134c101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:41.240 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.239+0000 7f1351600640 1 -- 172.21.15.117:0/936445335 shutdown_connections 2024-03-19T03:32:41.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.239+0000 7f1351600640 1 -- 172.21.15.117:0/936445335 wait complete. 2024-03-19T03:32:41.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.240+0000 7f1351600640 1 Processor -- start 2024-03-19T03:32:41.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.240+0000 7f1351600640 1 -- start start 2024-03-19T03:32:41.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.241+0000 7f1351600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f134c0761e0 0x7f134c19e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:41.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.241+0000 7f1351600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f134c077420 0x7f134c19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:41.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.241+0000 7f1351600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f134c077dc0 0x7f134c1a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:41.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.241+0000 7f1351600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f134c113af0 con 0x7f134c0761e0 2024-03-19T03:32:41.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.241+0000 7f1351600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f134c113970 con 0x7f134c077dc0 2024-03-19T03:32:41.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.241+0000 7f1351600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f134c113c70 con 0x7f134c077420 2024-03-19T03:32:41.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.241+0000 7f134aa00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f134c0761e0 0x7f134c19e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:41.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.241+0000 7f134b400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f134c077dc0 0x7f134c1a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:41.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.241+0000 7f134b400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f134c077dc0 0x7f134c1a5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:34386/0 (socket says 172.21.15.117:34386) 2024-03-19T03:32:41.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.241+0000 7f134b400640 1 -- 172.21.15.117:0/1201517501 learned_addr learned my addr 172.21.15.117:0/1201517501 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:41.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.241+0000 7f134a000640 1 --2- 172.21.15.117:0/1201517501 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f134c077420 0x7f134c19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:41.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.241+0000 7f134b400640 1 -- 172.21.15.117:0/1201517501 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f134c077420 msgr2=0x7f134c19e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:41.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.241+0000 7f134b400640 1 --2- 172.21.15.117:0/1201517501 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f134c077420 0x7f134c19e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:41.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.241+0000 7f134b400640 1 -- 172.21.15.117:0/1201517501 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f134c0761e0 msgr2=0x7f134c19e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:41.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.241+0000 7f134b400640 1 --2- 172.21.15.117:0/1201517501 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f134c0761e0 0x7f134c19e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:41.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.241+0000 7f134b400640 1 -- 172.21.15.117:0/1201517501 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f134c1a5d70 con 0x7f134c077dc0 2024-03-19T03:32:41.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.241+0000 7f134a000640 1 --2- 172.21.15.117:0/1201517501 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f134c077420 0x7f134c19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:32:41.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.242+0000 7f134b400640 1 --2- 172.21.15.117:0/1201517501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f134c077dc0 0x7f134c1a5700 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f134400eb10 tx=0x7f134400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:41.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.242+0000 7f1343400640 1 -- 172.21.15.117:0/1201517501 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1344004b20 con 0x7f134c077dc0 2024-03-19T03:32:41.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.242+0000 7f1343400640 1 -- 172.21.15.117:0/1201517501 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1344004cc0 con 0x7f134c077dc0 2024-03-19T03:32:41.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.242+0000 7f1351600640 1 -- 172.21.15.117:0/1201517501 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f134c1a6060 con 0x7f134c077dc0 2024-03-19T03:32:41.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.242+0000 7f1343400640 1 -- 172.21.15.117:0/1201517501 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1344010760 con 0x7f134c077dc0 2024-03-19T03:32:41.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.242+0000 7f1351600640 1 -- 172.21.15.117:0/1201517501 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f134c1a65b0 con 0x7f134c077dc0 2024-03-19T03:32:41.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.243+0000 7f1351600640 1 -- 172.21.15.117:0/1201517501 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f134c1105c0 con 0x7f134c077dc0 2024-03-19T03:32:41.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.245+0000 7f1343400640 1 -- 172.21.15.117:0/1201517501 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f1344005020 con 0x7f134c077dc0 2024-03-19T03:32:41.247 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.246+0000 7f1343400640 1 --2- 172.21.15.117:0/1201517501 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f13300a2880 0x7f13300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:41.247 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.247+0000 7f1343400640 1 -- 172.21.15.117:0/1201517501 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f13440cf390 con 0x7f134c077dc0 2024-03-19T03:32:41.247 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.247+0000 7f134aa00640 1 --2- 172.21.15.117:0/1201517501 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f13300a2880 0x7f13300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:41.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.247+0000 7f134aa00640 1 --2- 172.21.15.117:0/1201517501 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f13300a2880 0x7f13300a4d40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f133c02c7b0 tx=0x7f133c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:41.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.250+0000 7f1343400640 1 -- 172.21.15.117:0/1201517501 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f134409bfe0 con 0x7f134c077dc0 2024-03-19T03:32:41.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.449+0000 7f1351600640 1 -- 172.21.15.117:0/1201517501 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f134c062420 con 0x7f134c077dc0 2024-03-19T03:32:41.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.450+0000 7f1343400640 1 -- 172.21.15.117:0/1201517501 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f134409b8e0 con 0x7f134c077dc0 2024-03-19T03:32:41.451 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 64 2024-03-19T03:32:41.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.453+0000 7f1351600640 1 -- 172.21.15.117:0/1201517501 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f13300a2880 msgr2=0x7f13300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:41.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.453+0000 7f1351600640 1 --2- 172.21.15.117:0/1201517501 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f13300a2880 0x7f13300a4d40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f133c02c7b0 tx=0x7f133c005fd0 comp rx=0 tx=0).stop 2024-03-19T03:32:41.454 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.453+0000 7f1351600640 1 -- 172.21.15.117:0/1201517501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f134c077dc0 msgr2=0x7f134c1a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:41.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.453+0000 7f1351600640 1 --2- 172.21.15.117:0/1201517501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f134c077dc0 0x7f134c1a5700 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f134400eb10 tx=0x7f134400c2d0 comp rx=0 tx=0).stop 2024-03-19T03:32:41.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.453+0000 7f1351600640 1 -- 172.21.15.117:0/1201517501 shutdown_connections 2024-03-19T03:32:41.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.453+0000 7f1351600640 1 --2- 172.21.15.117:0/1201517501 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f13300a2880 0x7f13300a4d40 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:41.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.453+0000 7f1351600640 1 --2- 172.21.15.117:0/1201517501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f134c077dc0 0x7f134c1a5700 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:41.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.453+0000 7f1351600640 1 --2- 172.21.15.117:0/1201517501 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f134c077420 0x7f134c19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:41.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.453+0000 7f1351600640 1 --2- 172.21.15.117:0/1201517501 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f134c0761e0 0x7f134c19e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:41.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.453+0000 7f1351600640 1 -- 172.21.15.117:0/1201517501 >> 172.21.15.117:0/1201517501 conn(0x7f134c0fefd0 msgr2=0x7f134c100750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:41.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.453+0000 7f1351600640 1 -- 172.21.15.117:0/1201517501 shutdown_connections 2024-03-19T03:32:41.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.453+0000 7f1351600640 1 -- 172.21.15.117:0/1201517501 wait complete. 2024-03-19T03:32:41.473 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T03:32:41.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.685+0000 7f91a5a00640 1 -- 172.21.15.117:0/4098828706 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91a0104ca0 msgr2=0x7f91a0107090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:41.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.685+0000 7f91a5a00640 1 --2- 172.21.15.117:0/4098828706 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91a0104ca0 0x7f91a0107090 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f918c009900 tx=0x7f918c02bc30 comp rx=0 tx=0).stop 2024-03-19T03:32:41.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.685+0000 7f91a5a00640 1 -- 172.21.15.117:0/4098828706 shutdown_connections 2024-03-19T03:32:41.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.685+0000 7f91a5a00640 1 --2- 172.21.15.117:0/4098828706 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91a01075d0 0x7f91a01099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:41.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.685+0000 7f91a5a00640 1 --2- 172.21.15.117:0/4098828706 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91a0104ca0 0x7f91a0107090 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:41.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.685+0000 7f91a5a00640 1 --2- 172.21.15.117:0/4098828706 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f91a0102370 0x7f91a0104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:41.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.685+0000 7f91a5a00640 1 -- 172.21.15.117:0/4098828706 >> 172.21.15.117:0/4098828706 conn(0x7f91a00fbcc0 msgr2=0x7f91a00fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:41.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.685+0000 7f91a5a00640 1 -- 172.21.15.117:0/4098828706 shutdown_connections 2024-03-19T03:32:41.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.685+0000 7f91a5a00640 1 -- 172.21.15.117:0/4098828706 wait complete. 2024-03-19T03:32:41.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.687+0000 7f91a5a00640 1 Processor -- start 2024-03-19T03:32:41.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.687+0000 7f91a5a00640 1 -- start start 2024-03-19T03:32:41.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.687+0000 7f91a5a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91a0102370 0x7f91a0197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:41.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.687+0000 7f91a5a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f91a0104ca0 0x7f91a01980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:41.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.687+0000 7f91a5a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91a01075d0 0x7f91a019f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:41.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.687+0000 7f91a5a00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f91a010d4e0 con 0x7f91a0102370 2024-03-19T03:32:41.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.687+0000 7f91a5a00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f91a010d360 con 0x7f91a01075d0 2024-03-19T03:32:41.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.688+0000 7f91a5a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f91a010d660 con 0x7f91a0104ca0 2024-03-19T03:32:41.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.688+0000 7f919ea00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91a0102370 0x7f91a0197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:41.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.688+0000 7f919ea00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91a0102370 0x7f91a0197bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:37426/0 (socket says 172.21.15.117:37426) 2024-03-19T03:32:41.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.688+0000 7f919ea00640 1 -- 172.21.15.117:0/4240289242 learned_addr learned my addr 172.21.15.117:0/4240289242 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:41.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.688+0000 7f919f400640 1 --2- 172.21.15.117:0/4240289242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91a01075d0 0x7f91a019f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:41.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.688+0000 7f919e000640 1 --2- 172.21.15.117:0/4240289242 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f91a0104ca0 0x7f91a01980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:41.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.688+0000 7f919ea00640 1 -- 172.21.15.117:0/4240289242 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f91a0104ca0 msgr2=0x7f91a01980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:41.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.688+0000 7f919ea00640 1 --2- 172.21.15.117:0/4240289242 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f91a0104ca0 0x7f91a01980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:41.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.688+0000 7f919ea00640 1 -- 172.21.15.117:0/4240289242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91a01075d0 msgr2=0x7f91a019f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:41.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.688+0000 7f919ea00640 1 --2- 172.21.15.117:0/4240289242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91a01075d0 0x7f91a019f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:41.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.688+0000 7f919ea00640 1 -- 172.21.15.117:0/4240289242 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a019f7d0 con 0x7f91a0102370 2024-03-19T03:32:41.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.688+0000 7f919f400640 1 --2- 172.21.15.117:0/4240289242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91a01075d0 0x7f91a019f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:32:41.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.689+0000 7f919ea00640 1 --2- 172.21.15.117:0/4240289242 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91a0102370 0x7f91a0197bb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f919000d950 tx=0x7f919000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:41.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.689+0000 7f9197400640 1 -- 172.21.15.117:0/4240289242 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9190017070 con 0x7f91a0102370 2024-03-19T03:32:41.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.689+0000 7f91a5a00640 1 -- 172.21.15.117:0/4240289242 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91a019fac0 con 0x7f91a0102370 2024-03-19T03:32:41.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.689+0000 7f91a5a00640 1 -- 172.21.15.117:0/4240289242 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91a01a0030 con 0x7f91a0102370 2024-03-19T03:32:41.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.689+0000 7f9197400640 1 -- 172.21.15.117:0/4240289242 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9190009c30 con 0x7f91a0102370 2024-03-19T03:32:41.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.689+0000 7f9197400640 1 -- 172.21.15.117:0/4240289242 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9190020060 con 0x7f91a0102370 2024-03-19T03:32:41.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.691+0000 7f91a5a00640 1 -- 172.21.15.117:0/4240289242 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91800052c0 con 0x7f91a0102370 2024-03-19T03:32:41.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.692+0000 7f9197400640 1 -- 172.21.15.117:0/4240289242 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f91900109d0 con 0x7f91a0102370 2024-03-19T03:32:41.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.694+0000 7f9197400640 1 --2- 172.21.15.117:0/4240289242 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f91840a2800 0x7f91840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:41.695 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.695+0000 7f9197400640 1 -- 172.21.15.117:0/4240289242 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f91900ce5f0 con 0x7f91a0102370 2024-03-19T03:32:41.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.695+0000 7f919e000640 1 --2- 172.21.15.117:0/4240289242 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f91840a2800 0x7f91840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:41.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.695+0000 7f919e000640 1 --2- 172.21.15.117:0/4240289242 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f91840a2800 0x7f91840a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f918c02c170 tx=0x7f918c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:41.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.699+0000 7f9197400640 1 -- 172.21.15.117:0/4240289242 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9190009de0 con 0x7f91a0102370 2024-03-19T03:32:41.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:41 smithi119 ceph-mon[66861]: pgmap v383: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-19T03:32:41.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:41 smithi119 ceph-mon[66861]: osdmap e93: 12 total, 12 up, 12 in 2024-03-19T03:32:41.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:41 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3603520736' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:32:41.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:41 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1155500252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:32:41.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:41 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1201517501' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:32:41.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:41 smithi117 ceph-mon[31838]: pgmap v383: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-19T03:32:41.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:41 smithi117 ceph-mon[31838]: osdmap e93: 12 total, 12 up, 12 in 2024-03-19T03:32:41.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:41 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3603520736' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:32:41.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:41 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1155500252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:32:41.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:41 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1201517501' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:32:41.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:41 smithi178 ceph-mon[67578]: pgmap v383: 129 pgs: 129 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-19T03:32:41.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:41 smithi178 ceph-mon[67578]: osdmap e93: 12 total, 12 up, 12 in 2024-03-19T03:32:41.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:41 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3603520736' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:32:41.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:41 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1155500252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:32:41.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:41 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1201517501' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:32:41.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.995+0000 7f91a5a00640 1 -- 172.21.15.117:0/4240289242 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9180005dc0 con 0x7f91a0102370 2024-03-19T03:32:41.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.996+0000 7f9197400640 1 -- 172.21.15.117:0/4240289242 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f919009ac50 con 0x7f91a0102370 2024-03-19T03:32:41.996 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 32 2024-03-19T03:32:42.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.999+0000 7f91a5a00640 1 -- 172.21.15.117:0/4240289242 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f91840a2800 msgr2=0x7f91840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:42.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.999+0000 7f91a5a00640 1 --2- 172.21.15.117:0/4240289242 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f91840a2800 0x7f91840a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f918c02c170 tx=0x7f918c02df90 comp rx=0 tx=0).stop 2024-03-19T03:32:42.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.999+0000 7f91a5a00640 1 -- 172.21.15.117:0/4240289242 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91a0102370 msgr2=0x7f91a0197bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:42.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.999+0000 7f91a5a00640 1 --2- 172.21.15.117:0/4240289242 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91a0102370 0x7f91a0197bb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f919000d950 tx=0x7f919000de10 comp rx=0 tx=0).stop 2024-03-19T03:32:42.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.999+0000 7f91a5a00640 1 -- 172.21.15.117:0/4240289242 shutdown_connections 2024-03-19T03:32:42.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.999+0000 7f91a5a00640 1 --2- 172.21.15.117:0/4240289242 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f91840a2800 0x7f91840a4cc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:42.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.999+0000 7f91a5a00640 1 --2- 172.21.15.117:0/4240289242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91a01075d0 0x7f91a019f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:42.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.999+0000 7f91a5a00640 1 --2- 172.21.15.117:0/4240289242 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f91a0104ca0 0x7f91a01980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:42.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.999+0000 7f91a5a00640 1 --2- 172.21.15.117:0/4240289242 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91a0102370 0x7f91a0197bb0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:42.000 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.999+0000 7f91a5a00640 1 -- 172.21.15.117:0/4240289242 >> 172.21.15.117:0/4240289242 conn(0x7f91a00fbcc0 msgr2=0x7f91a0102df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:42.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.999+0000 7f91a5a00640 1 -- 172.21.15.117:0/4240289242 shutdown_connections 2024-03-19T03:32:42.001 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:41.999+0000 7f91a5a00640 1 -- 172.21.15.117:0/4240289242 wait complete. 2024-03-19T03:32:42.027 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:32:42.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.224+0000 7f0977e00640 1 -- 172.21.15.117:0/4215371675 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0978104a20 msgr2=0x7f0978106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:42.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.224+0000 7f0977e00640 1 --2- 172.21.15.117:0/4215371675 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0978104a20 0x7f0978106e10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0960009900 tx=0x7f096002bc30 comp rx=0 tx=0).stop 2024-03-19T03:32:42.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.224+0000 7f0977e00640 1 -- 172.21.15.117:0/4215371675 shutdown_connections 2024-03-19T03:32:42.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.224+0000 7f0977e00640 1 --2- 172.21.15.117:0/4215371675 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0978107350 0x7f09781097a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:42.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.224+0000 7f0977e00640 1 --2- 172.21.15.117:0/4215371675 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0978104a20 0x7f0978106e10 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:42.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.224+0000 7f0977e00640 1 --2- 172.21.15.117:0/4215371675 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09781020f0 0x7f09781044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:42.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.224+0000 7f0977e00640 1 -- 172.21.15.117:0/4215371675 >> 172.21.15.117:0/4215371675 conn(0x7f09780fbb00 msgr2=0x7f09780fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:42.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.224+0000 7f0977e00640 1 -- 172.21.15.117:0/4215371675 shutdown_connections 2024-03-19T03:32:42.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.224+0000 7f0977e00640 1 -- 172.21.15.117:0/4215371675 wait complete. 2024-03-19T03:32:42.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.225+0000 7f0977e00640 1 Processor -- start 2024-03-19T03:32:42.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.226+0000 7f0977e00640 1 -- start start 2024-03-19T03:32:42.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.226+0000 7f0977e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09781020f0 0x7f0978101a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:42.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.226+0000 7f0977e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0978104a20 0x7f09781000a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:42.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.226+0000 7f0977e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0978107350 0x7f09781005e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:42.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.226+0000 7f0977e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f097810d350 con 0x7f09781020f0 2024-03-19T03:32:42.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.226+0000 7f0977e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f097810d1d0 con 0x7f0978107350 2024-03-19T03:32:42.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.226+0000 7f0977e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f097810d4d0 con 0x7f0978104a20 2024-03-19T03:32:42.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.226+0000 7f0976a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09781020f0 0x7f0978101a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:42.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.226+0000 7f0977400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0978107350 0x7f09781005e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:42.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.226+0000 7f0976a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09781020f0 0x7f0978101a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:37442/0 (socket says 172.21.15.117:37442) 2024-03-19T03:32:42.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.226+0000 7f0976a00640 1 -- 172.21.15.117:0/2944102668 learned_addr learned my addr 172.21.15.117:0/2944102668 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:42.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.226+0000 7f096fe00640 1 --2- 172.21.15.117:0/2944102668 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0978104a20 0x7f09781000a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:42.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.227+0000 7f0977400640 1 -- 172.21.15.117:0/2944102668 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0978104a20 msgr2=0x7f09781000a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:42.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.227+0000 7f0977400640 1 --2- 172.21.15.117:0/2944102668 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0978104a20 0x7f09781000a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:42.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.227+0000 7f0977400640 1 -- 172.21.15.117:0/2944102668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09781020f0 msgr2=0x7f0978101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:42.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.227+0000 7f0977400640 1 --2- 172.21.15.117:0/2944102668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09781020f0 0x7f0978101a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:42.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.227+0000 7f0977400640 1 -- 172.21.15.117:0/2944102668 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0978100e70 con 0x7f0978107350 2024-03-19T03:32:42.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.227+0000 7f0977400640 1 --2- 172.21.15.117:0/2944102668 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0978107350 0x7f09781005e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0970009fd0 tx=0x7f097000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:42.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.227+0000 7f096f400640 1 -- 172.21.15.117:0/2944102668 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097000ce20 con 0x7f0978107350 2024-03-19T03:32:42.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.227+0000 7f096f400640 1 -- 172.21.15.117:0/2944102668 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0970005020 con 0x7f0978107350 2024-03-19T03:32:42.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.227+0000 7f0977e00640 1 -- 172.21.15.117:0/2944102668 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09781a1ac0 con 0x7f0978107350 2024-03-19T03:32:42.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.227+0000 7f096f400640 1 -- 172.21.15.117:0/2944102668 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0970010720 con 0x7f0978107350 2024-03-19T03:32:42.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.227+0000 7f0977e00640 1 -- 172.21.15.117:0/2944102668 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09781a1fe0 con 0x7f0978107350 2024-03-19T03:32:42.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.229+0000 7f0977e00640 1 -- 172.21.15.117:0/2944102668 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0978068530 con 0x7f0978107350 2024-03-19T03:32:42.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.230+0000 7f096f400640 1 -- 172.21.15.117:0/2944102668 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f0970005540 con 0x7f0978107350 2024-03-19T03:32:42.232 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.231+0000 7f096f400640 1 --2- 172.21.15.117:0/2944102668 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f095c0a2880 0x7f095c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:42.232 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.232+0000 7f096f400640 1 -- 172.21.15.117:0/2944102668 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f09700cf4e0 con 0x7f0978107350 2024-03-19T03:32:42.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.232+0000 7f0976a00640 1 --2- 172.21.15.117:0/2944102668 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f095c0a2880 0x7f095c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:42.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.233+0000 7f0976a00640 1 --2- 172.21.15.117:0/2944102668 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f095c0a2880 0x7f095c0a4d40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0968009cd0 tx=0x7f0968005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:42.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.235+0000 7f096f400640 1 -- 172.21.15.117:0/2944102668 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f097009c190 con 0x7f0978107350 2024-03-19T03:32:42.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.488+0000 7f0977e00640 1 -- 172.21.15.117:0/2944102668 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0978062420 con 0x7f0978107350 2024-03-19T03:32:42.490 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.489+0000 7f096f400640 1 -- 172.21.15.117:0/2944102668 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11953 (secure 0 0 0) 0x7f097009ba90 con 0x7f0978107350 2024-03-19T03:32:42.490 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:32:42.491 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/54638259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":54638259},{"type":"v1","addr":"172.21.15.117:6835","nonce":54638259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3608185039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3608185039},{"type":"v1","addr":"172.21.15.117:6837","nonce":3608185039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T03:30:35.081740+0000","modified":"2024-03-19T03:31:09.995133+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24599,"mds_1":24581,"mds_2":24569,"mds_3":24545,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":20,"addr":"172.21.15.119:6833/1291833214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1291833214},{"type":"v1","addr":"172.21.15.119:6833","nonce":1291833214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6835/1579860804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1579860804},{"type":"v1","addr":"172.21.15.178:6835","nonce":1579860804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6835/1686938941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1686938941},{"type":"v1","addr":"172.21.15.119:6835","nonce":1686938941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6837/1770913538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":1770913538},{"type":"v1","addr":"172.21.15.178:6837","nonce":1770913538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6839/2655593383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2655593383},{"type":"v1","addr":"172.21.15.178:6839","nonce":2655593383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.119:6837/2054992738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2054992738},{"type":"v1","addr":"172.21.15.119:6837","nonce":2054992738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"g","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.117:6839/684335078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":684335078},{"type":"v1","addr":"172.21.15.117:6839","nonce":684335078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.117:6841/1329933892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6840","nonce":1329933892},{"type":"v1","addr":"172.21.15.117:6841","nonce":1329933892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/2383354833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2383354833},{"type":"v1","addr":"172.21.15.119:6839","nonce":2383354833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24581,24599,24569,24545,24467]},"id":1}]} 2024-03-19T03:32:42.491 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 28 2024-03-19T03:32:42.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.492+0000 7f0977e00640 1 -- 172.21.15.117:0/2944102668 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f095c0a2880 msgr2=0x7f095c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:42.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.492+0000 7f0977e00640 1 --2- 172.21.15.117:0/2944102668 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f095c0a2880 0x7f095c0a4d40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0968009cd0 tx=0x7f0968005f90 comp rx=0 tx=0).stop 2024-03-19T03:32:42.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.492+0000 7f0977e00640 1 -- 172.21.15.117:0/2944102668 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0978107350 msgr2=0x7f09781005e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:42.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.492+0000 7f0977e00640 1 --2- 172.21.15.117:0/2944102668 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0978107350 0x7f09781005e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0970009fd0 tx=0x7f097000ef10 comp rx=0 tx=0).stop 2024-03-19T03:32:42.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.492+0000 7f0977e00640 1 -- 172.21.15.117:0/2944102668 shutdown_connections 2024-03-19T03:32:42.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.492+0000 7f0977e00640 1 --2- 172.21.15.117:0/2944102668 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f095c0a2880 0x7f095c0a4d40 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:42.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.492+0000 7f0977e00640 1 --2- 172.21.15.117:0/2944102668 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0978107350 0x7f09781005e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:42.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.492+0000 7f0977e00640 1 --2- 172.21.15.117:0/2944102668 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0978104a20 0x7f09781000a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:42.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.492+0000 7f0977e00640 1 --2- 172.21.15.117:0/2944102668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09781020f0 0x7f0978101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:42.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.492+0000 7f0977e00640 1 -- 172.21.15.117:0/2944102668 >> 172.21.15.117:0/2944102668 conn(0x7f09780fbb00 msgr2=0x7f0978102a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:42.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.492+0000 7f0977e00640 1 -- 172.21.15.117:0/2944102668 shutdown_connections 2024-03-19T03:32:42.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.492+0000 7f0977e00640 1 -- 172.21.15.117:0/2944102668 wait complete. 2024-03-19T03:32:42.511 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:32:42.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.703+0000 7f16af800640 1 -- 172.21.15.117:0/3123883335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f16a8100a90 msgr2=0x7f16a8100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:42.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.703+0000 7f16af800640 1 --2- 172.21.15.117:0/3123883335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f16a8100a90 0x7f16a8100ef0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1694009900 tx=0x7f169402bc30 comp rx=0 tx=0).stop 2024-03-19T03:32:42.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.704+0000 7f16af800640 1 -- 172.21.15.117:0/3123883335 shutdown_connections 2024-03-19T03:32:42.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.704+0000 7f16af800640 1 --2- 172.21.15.117:0/3123883335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16a8101430 0x7f16a81018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:42.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.704+0000 7f16af800640 1 --2- 172.21.15.117:0/3123883335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f16a8100a90 0x7f16a8100ef0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:42.705 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.704+0000 7f16af800640 1 --2- 172.21.15.117:0/3123883335 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f16a8100130 0x7f16a8100550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:42.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.704+0000 7f16af800640 1 -- 172.21.15.117:0/3123883335 >> 172.21.15.117:0/3123883335 conn(0x7f16a80fbcd0 msgr2=0x7f16a80fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:42.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.704+0000 7f16af800640 1 -- 172.21.15.117:0/3123883335 shutdown_connections 2024-03-19T03:32:42.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.704+0000 7f16af800640 1 -- 172.21.15.117:0/3123883335 wait complete. 2024-03-19T03:32:42.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.705+0000 7f16af800640 1 Processor -- start 2024-03-19T03:32:42.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.705+0000 7f16af800640 1 -- start start 2024-03-19T03:32:42.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.706+0000 7f16af800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f16a8100130 0x7f16a8199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:42.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.706+0000 7f16af800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f16a8100a90 0x7f16a819a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:42.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.706+0000 7f16af800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16a8101430 0x7f16a81a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:42.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.706+0000 7f16af800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f16a810f710 con 0x7f16a8101430 2024-03-19T03:32:42.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.706+0000 7f16af800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f16a810f590 con 0x7f16a8100a90 2024-03-19T03:32:42.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.706+0000 7f16af800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f16a810f890 con 0x7f16a8100130 2024-03-19T03:32:42.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.706+0000 7f16a7e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f16a8100a90 0x7f16a819a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:42.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.706+0000 7f16ace00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f16a8100130 0x7f16a8199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:42.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.706+0000 7f16a7e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f16a8100a90 0x7f16a819a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:34438/0 (socket says 172.21.15.117:34438) 2024-03-19T03:32:42.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.706+0000 7f16a7e00640 1 -- 172.21.15.117:0/3860160577 learned_addr learned my addr 172.21.15.117:0/3860160577 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:42.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.706+0000 7f16ad800640 1 --2- 172.21.15.117:0/3860160577 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16a8101430 0x7f16a81a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:42.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.706+0000 7f16a7e00640 1 -- 172.21.15.117:0/3860160577 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f16a8100130 msgr2=0x7f16a8199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:42.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.706+0000 7f16a7e00640 1 --2- 172.21.15.117:0/3860160577 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f16a8100130 0x7f16a8199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:42.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.706+0000 7f16a7e00640 1 -- 172.21.15.117:0/3860160577 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16a8101430 msgr2=0x7f16a81a1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:42.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.706+0000 7f16a7e00640 1 --2- 172.21.15.117:0/3860160577 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16a8101430 0x7f16a81a1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:42.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.706+0000 7f16a7e00640 1 -- 172.21.15.117:0/3860160577 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a81a1970 con 0x7f16a8100a90 2024-03-19T03:32:42.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.706+0000 7f16ace00640 1 --2- 172.21.15.117:0/3860160577 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f16a8100130 0x7f16a8199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:32:42.709 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.707+0000 7f16a7e00640 1 --2- 172.21.15.117:0/3860160577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f16a8100a90 0x7f16a819a290 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1694000c00 tx=0x7f169402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:42.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.707+0000 7f16a5600640 1 -- 172.21.15.117:0/3860160577 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1694043020 con 0x7f16a8100a90 2024-03-19T03:32:42.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.707+0000 7f16a5600640 1 -- 172.21.15.117:0/3860160577 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16940054e0 con 0x7f16a8100a90 2024-03-19T03:32:42.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.707+0000 7f16af800640 1 -- 172.21.15.117:0/3860160577 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16a81a1c00 con 0x7f16a8100a90 2024-03-19T03:32:42.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.707+0000 7f16a5600640 1 -- 172.21.15.117:0/3860160577 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1694003c40 con 0x7f16a8100a90 2024-03-19T03:32:42.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.707+0000 7f16af800640 1 -- 172.21.15.117:0/3860160577 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16a81a21a0 con 0x7f16a8100a90 2024-03-19T03:32:42.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.708+0000 7f16af800640 1 -- 172.21.15.117:0/3860160577 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16a810b010 con 0x7f16a8100a90 2024-03-19T03:32:42.711 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.710+0000 7f16a5600640 1 -- 172.21.15.117:0/3860160577 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f1694046020 con 0x7f16a8100a90 2024-03-19T03:32:42.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.712+0000 7f16a5600640 1 --2- 172.21.15.117:0/3860160577 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f16900a2880 0x7f16900a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:42.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.712+0000 7f16a5600640 1 -- 172.21.15.117:0/3860160577 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f16940f0ea0 con 0x7f16a8100a90 2024-03-19T03:32:42.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.712+0000 7f16ace00640 1 --2- 172.21.15.117:0/3860160577 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f16900a2880 0x7f16900a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:42.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.713+0000 7f16ace00640 1 --2- 172.21.15.117:0/3860160577 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f16900a2880 0x7f16900a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f169c009fc0 tx=0x7f169c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:42.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.716+0000 7f16a5600640 1 -- 172.21.15.117:0/3860160577 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f169404d020 con 0x7f16a8100a90 2024-03-19T03:32:42.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:42 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/4240289242' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:32:42.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:42 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/4240289242' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:32:42.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:42 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/4240289242' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:32:42.921 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.920+0000 7f16af800640 1 -- 172.21.15.117:0/3860160577 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f16a8062420 con 0x7f16a8100a90 2024-03-19T03:32:42.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.922+0000 7f16a5600640 1 -- 172.21.15.117:0/3860160577 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24989 (secure 0 0 0) 0x7f16940bdaf0 con 0x7f16a8100a90 2024-03-19T03:32:42.923 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:32:42.923 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":93,"fsid":"a3a82456-e59f-11ee-95c9-87774f69a715","created":"2024-03-19T03:21:30.427136+0000","modified":"2024-03-19T03:32:40.453890+0000","last_up_change":"2024-03-19T03:29:32.967505+0000","last_in_change":"2024-03-19T03:29:12.848740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:25:18.824270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T03:30:33.137170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T03:30:34.598838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T03:30:38.230023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b32eb0ee-9d4b-4748-87ec-eda02c907415","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6803","nonce":3014934900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6805","nonce":3014934900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6809","nonce":3014934900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6807","nonce":3014934900}]},"public_addr":"172.21.15.117:6803/3014934900","cluster_addr":"172.21.15.117:6805/3014934900","heartbeat_back_addr":"172.21.15.117:6809/3014934900","heartbeat_front_addr":"172.21.15.117:6807/3014934900","state":["exists","up"]},{"osd":1,"uuid":"e07585be-c016-4466-b07c-abbc22a58de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6801","nonce":2425927341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6803","nonce":2425927341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6807","nonce":2425927341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6805","nonce":2425927341}]},"public_addr":"172.21.15.119:6801/2425927341","cluster_addr":"172.21.15.119:6803/2425927341","heartbeat_back_addr":"172.21.15.119:6807/2425927341","heartbeat_front_addr":"172.21.15.119:6805/2425927341","state":["exists","up"]},{"osd":2,"uuid":"ede9edda-3b9f-41ca-ad04-5dcb738ab757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6801","nonce":3733812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6803","nonce":3733812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6807","nonce":3733812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6805","nonce":3733812989}]},"public_addr":"172.21.15.178:6801/3733812989","cluster_addr":"172.21.15.178:6803/3733812989","heartbeat_back_addr":"172.21.15.178:6807/3733812989","heartbeat_front_addr":"172.21.15.178:6805/3733812989","state":["exists","up"]},{"osd":3,"uuid":"f81c41e1-3a69-40db-a90f-58151286b081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6811","nonce":2178310135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6813","nonce":2178310135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6817","nonce":2178310135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6815","nonce":2178310135}]},"public_addr":"172.21.15.117:6811/2178310135","cluster_addr":"172.21.15.117:6813/2178310135","heartbeat_back_addr":"172.21.15.117:6817/2178310135","heartbeat_front_addr":"172.21.15.117:6815/2178310135","state":["exists","up"]},{"osd":4,"uuid":"caf7ce86-caf0-4eaf-bbab-bd2d93e63c33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6809","nonce":3455398086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6811","nonce":3455398086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6815","nonce":3455398086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6813","nonce":3455398086}]},"public_addr":"172.21.15.119:6809/3455398086","cluster_addr":"172.21.15.119:6811/3455398086","heartbeat_back_addr":"172.21.15.119:6815/3455398086","heartbeat_front_addr":"172.21.15.119:6813/3455398086","state":["exists","up"]},{"osd":5,"uuid":"20ba0b8a-2e0f-4ea7-b435-430e44766200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6809","nonce":1259631912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6811","nonce":1259631912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6815","nonce":1259631912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6813","nonce":1259631912}]},"public_addr":"172.21.15.178:6809/1259631912","cluster_addr":"172.21.15.178:6811/1259631912","heartbeat_back_addr":"172.21.15.178:6815/1259631912","heartbeat_front_addr":"172.21.15.178:6813/1259631912","state":["exists","up"]},{"osd":6,"uuid":"75c2ddd7-f837-42e0-ac47-8cfbcf97a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6819","nonce":864146757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6821","nonce":864146757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6825","nonce":864146757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6823","nonce":864146757}]},"public_addr":"172.21.15.117:6819/864146757","cluster_addr":"172.21.15.117:6821/864146757","heartbeat_back_addr":"172.21.15.117:6825/864146757","heartbeat_front_addr":"172.21.15.117:6823/864146757","state":["exists","up"]},{"osd":7,"uuid":"929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6817","nonce":1273365545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6819","nonce":1273365545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6823","nonce":1273365545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6821","nonce":1273365545}]},"public_addr":"172.21.15.119:6817/1273365545","cluster_addr":"172.21.15.119:6819/1273365545","heartbeat_back_addr":"172.21.15.119:6823/1273365545","heartbeat_front_addr":"172.21.15.119:6821/1273365545","state":["exists","up"]},{"osd":8,"uuid":"fd371461-a538-4781-92e1-5eead4ca3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6817","nonce":1059587501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6819","nonce":1059587501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6823","nonce":1059587501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6821","nonce":1059587501}]},"public_addr":"172.21.15.178:6817/1059587501","cluster_addr":"172.21.15.178:6819/1059587501","heartbeat_back_addr":"172.21.15.178:6823/1059587501","heartbeat_front_addr":"172.21.15.178:6821/1059587501","state":["exists","up"]},{"osd":9,"uuid":"0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6827","nonce":3653875600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6829","nonce":3653875600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6833","nonce":3653875600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6831","nonce":3653875600}]},"public_addr":"172.21.15.117:6827/3653875600","cluster_addr":"172.21.15.117:6829/3653875600","heartbeat_back_addr":"172.21.15.117:6833/3653875600","heartbeat_front_addr":"172.21.15.117:6831/3653875600","state":["exists","up"]},{"osd":10,"uuid":"de2049da-d19e-4386-afbf-fd6b5fddbcba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6825","nonce":1050976194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6827","nonce":1050976194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6831","nonce":1050976194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6829","nonce":1050976194}]},"public_addr":"172.21.15.119:6825/1050976194","cluster_addr":"172.21.15.119:6827/1050976194","heartbeat_back_addr":"172.21.15.119:6831/1050976194","heartbeat_front_addr":"172.21.15.119:6829/1050976194","state":["exists","up"]},{"osd":11,"uuid":"3cccc0f9-d273-4b6b-9b8e-c926371472dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6825","nonce":2915080368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6827","nonce":2915080368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6831","nonce":2915080368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6829","nonce":2915080368}]},"public_addr":"172.21.15.178:6825/2915080368","cluster_addr":"172.21.15.178:6827/2915080368","heartbeat_back_addr":"172.21.15.178:6831/2915080368","heartbeat_front_addr":"172.21.15.178:6829/2915080368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:27.876274+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:53.094706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:16.598813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:48.160419+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:14.048925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:38.814304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:12.542328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:38.942747+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:04.488963+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:38.196053+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:05.254728+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:30.874915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7},{"from":5,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.117:0/185684447":"2024-03-20T03:22:38.134578+0000","172.21.15.117:0/361367101":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/2788067603":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3506200675":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6800/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3283471541":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3750920169":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/2780813298":"2024-03-20T03:22:03.905461+0000","172.21.15.117:6800/2788067603":"2024-03-20T03:22:38.134578+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:32:42.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.924+0000 7f16af800640 1 -- 172.21.15.117:0/3860160577 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f16900a2880 msgr2=0x7f16900a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:42.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.924+0000 7f16af800640 1 --2- 172.21.15.117:0/3860160577 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f16900a2880 0x7f16900a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f169c009fc0 tx=0x7f169c005f90 comp rx=0 tx=0).stop 2024-03-19T03:32:42.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.924+0000 7f16af800640 1 -- 172.21.15.117:0/3860160577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f16a8100a90 msgr2=0x7f16a819a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:42.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.924+0000 7f16af800640 1 --2- 172.21.15.117:0/3860160577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f16a8100a90 0x7f16a819a290 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1694000c00 tx=0x7f169402c650 comp rx=0 tx=0).stop 2024-03-19T03:32:42.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.925+0000 7f16af800640 1 -- 172.21.15.117:0/3860160577 shutdown_connections 2024-03-19T03:32:42.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.925+0000 7f16af800640 1 --2- 172.21.15.117:0/3860160577 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16a8101430 0x7f16a81a1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:42.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.925+0000 7f16af800640 1 --2- 172.21.15.117:0/3860160577 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f16900a2880 0x7f16900a4d40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:42.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.925+0000 7f16af800640 1 --2- 172.21.15.117:0/3860160577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f16a8100a90 0x7f16a819a290 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:42.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.925+0000 7f16af800640 1 --2- 172.21.15.117:0/3860160577 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f16a8100130 0x7f16a8199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:42.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.925+0000 7f16af800640 1 -- 172.21.15.117:0/3860160577 >> 172.21.15.117:0/3860160577 conn(0x7f16a80fbcd0 msgr2=0x7f16a80fd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:42.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.925+0000 7f16af800640 1 -- 172.21.15.117:0/3860160577 shutdown_connections 2024-03-19T03:32:42.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:42.925+0000 7f16af800640 1 -- 172.21.15.117:0/3860160577 wait complete. 2024-03-19T03:32:42.944 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T03:25:18.824270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T03:30:33.137170+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T03:30:34.598838+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T03:30:38.230023+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T03:32:42.945 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T03:32:43.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.188+0000 7fb374c00640 1 -- 172.21.15.117:0/1167283236 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb370102180 msgr2=0x7fb370104570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:43.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.188+0000 7fb374c00640 1 --2- 172.21.15.117:0/1167283236 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb370102180 0x7fb370104570 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb360009900 tx=0x7fb36002bc30 comp rx=0 tx=0).stop 2024-03-19T03:32:43.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.188+0000 7fb374c00640 1 -- 172.21.15.117:0/1167283236 shutdown_connections 2024-03-19T03:32:43.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.188+0000 7fb374c00640 1 --2- 172.21.15.117:0/1167283236 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb3701073e0 0x7fb370109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:43.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.188+0000 7fb374c00640 1 --2- 172.21.15.117:0/1167283236 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb370104ab0 0x7fb370106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:43.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.188+0000 7fb374c00640 1 --2- 172.21.15.117:0/1167283236 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb370102180 0x7fb370104570 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:43.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.188+0000 7fb374c00640 1 -- 172.21.15.117:0/1167283236 >> 172.21.15.117:0/1167283236 conn(0x7fb3700fbb90 msgr2=0x7fb3700fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:43.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.188+0000 7fb374c00640 1 -- 172.21.15.117:0/1167283236 shutdown_connections 2024-03-19T03:32:43.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.188+0000 7fb374c00640 1 -- 172.21.15.117:0/1167283236 wait complete. 2024-03-19T03:32:43.192 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.189+0000 7fb374c00640 1 Processor -- start 2024-03-19T03:32:43.192 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.189+0000 7fb374c00640 1 -- start start 2024-03-19T03:32:43.192 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.189+0000 7fb374c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb370102180 0x7fb370197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:43.192 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.189+0000 7fb374c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb370104ab0 0x7fb370197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:43.192 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.189+0000 7fb374c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3701073e0 0x7fb37019efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:43.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.189+0000 7fb374c00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb37010d380 con 0x7fb3701073e0 2024-03-19T03:32:43.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.189+0000 7fb374c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fb37010d200 con 0x7fb370102180 2024-03-19T03:32:43.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.189+0000 7fb374c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fb37010d500 con 0x7fb370104ab0 2024-03-19T03:32:43.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.190+0000 7fb36f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb370102180 0x7fb370197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:43.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.190+0000 7fb36f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb370102180 0x7fb370197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:34450/0 (socket says 172.21.15.117:34450) 2024-03-19T03:32:43.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.190+0000 7fb36f400640 1 -- 172.21.15.117:0/2880642631 learned_addr learned my addr 172.21.15.117:0/2880642631 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:43.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.190+0000 7fb36ea00640 1 --2- 172.21.15.117:0/2880642631 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb370104ab0 0x7fb370197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:43.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.190+0000 7fb36fe00640 1 --2- 172.21.15.117:0/2880642631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3701073e0 0x7fb37019efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:43.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.190+0000 7fb36f400640 1 -- 172.21.15.117:0/2880642631 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb370104ab0 msgr2=0x7fb370197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:43.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.190+0000 7fb36f400640 1 --2- 172.21.15.117:0/2880642631 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb370104ab0 0x7fb370197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:43.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.190+0000 7fb36f400640 1 -- 172.21.15.117:0/2880642631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3701073e0 msgr2=0x7fb37019efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:43.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.190+0000 7fb36f400640 1 --2- 172.21.15.117:0/2880642631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3701073e0 0x7fb37019efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:43.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.190+0000 7fb36f400640 1 -- 172.21.15.117:0/2880642631 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb37019f640 con 0x7fb370102180 2024-03-19T03:32:43.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.190+0000 7fb36ea00640 1 --2- 172.21.15.117:0/2880642631 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb370104ab0 0x7fb370197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:32:43.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.190+0000 7fb36fe00640 1 --2- 172.21.15.117:0/2880642631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3701073e0 0x7fb37019efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T03:32:43.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.190+0000 7fb36f400640 1 --2- 172.21.15.117:0/2880642631 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb370102180 0x7fb370197a20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb360009230 tx=0x7fb360005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:43.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.191+0000 7fb367e00640 1 -- 172.21.15.117:0/2880642631 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb360041070 con 0x7fb370102180 2024-03-19T03:32:43.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.191+0000 7fb367e00640 1 -- 172.21.15.117:0/2880642631 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3600044d0 con 0x7fb370102180 2024-03-19T03:32:43.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.191+0000 7fb367e00640 1 -- 172.21.15.117:0/2880642631 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb360034440 con 0x7fb370102180 2024-03-19T03:32:43.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.191+0000 7fb374c00640 1 -- 172.21.15.117:0/2880642631 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb37019f8d0 con 0x7fb370102180 2024-03-19T03:32:43.194 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.191+0000 7fb374c00640 1 -- 172.21.15.117:0/2880642631 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb37019fdc0 con 0x7fb370102180 2024-03-19T03:32:43.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.192+0000 7fb374c00640 1 -- 172.21.15.117:0/2880642631 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3700fd680 con 0x7fb370102180 2024-03-19T03:32:43.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.194+0000 7fb367e00640 1 -- 172.21.15.117:0/2880642631 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fb360044070 con 0x7fb370102180 2024-03-19T03:32:43.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.195+0000 7fb367e00640 1 --2- 172.21.15.117:0/2880642631 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fb3540a2800 0x7fb3540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:43.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.195+0000 7fb367e00640 1 -- 172.21.15.117:0/2880642631 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fb3600f0bf0 con 0x7fb370102180 2024-03-19T03:32:43.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.195+0000 7fb36ea00640 1 --2- 172.21.15.117:0/2880642631 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fb3540a2800 0x7fb3540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:43.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.196+0000 7fb36ea00640 1 --2- 172.21.15.117:0/2880642631 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fb3540a2800 0x7fb3540a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb370198f30 tx=0x7fb35c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:43.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.203+0000 7fb367e00640 1 -- 172.21.15.117:0/2880642631 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb3600bd840 con 0x7fb370102180 2024-03-19T03:32:43.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.402+0000 7fb374c00640 1 -- 172.21.15.117:0/2880642631 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb37002d410 con 0x7fb370102180 2024-03-19T03:32:43.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.402+0000 7fb367e00640 1 -- 172.21.15.117:0/2880642631 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fb3600bd140 con 0x7fb370102180 2024-03-19T03:32:43.403 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 1 2024-03-19T03:32:43.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.405+0000 7fb374c00640 1 -- 172.21.15.117:0/2880642631 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fb3540a2800 msgr2=0x7fb3540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:43.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.405+0000 7fb374c00640 1 --2- 172.21.15.117:0/2880642631 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fb3540a2800 0x7fb3540a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb370198f30 tx=0x7fb35c005f90 comp rx=0 tx=0).stop 2024-03-19T03:32:43.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.405+0000 7fb374c00640 1 -- 172.21.15.117:0/2880642631 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb370102180 msgr2=0x7fb370197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:43.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.405+0000 7fb374c00640 1 --2- 172.21.15.117:0/2880642631 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb370102180 0x7fb370197a20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb360009230 tx=0x7fb360005d50 comp rx=0 tx=0).stop 2024-03-19T03:32:43.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.405+0000 7fb374c00640 1 -- 172.21.15.117:0/2880642631 shutdown_connections 2024-03-19T03:32:43.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.405+0000 7fb374c00640 1 --2- 172.21.15.117:0/2880642631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3701073e0 0x7fb37019efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:43.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.405+0000 7fb374c00640 1 --2- 172.21.15.117:0/2880642631 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb370104ab0 0x7fb370197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:43.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.405+0000 7fb374c00640 1 --2- 172.21.15.117:0/2880642631 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fb3540a2800 0x7fb3540a4cc0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:43.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.405+0000 7fb374c00640 1 --2- 172.21.15.117:0/2880642631 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb370102180 0x7fb370197a20 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:43.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.405+0000 7fb374c00640 1 -- 172.21.15.117:0/2880642631 >> 172.21.15.117:0/2880642631 conn(0x7fb3700fbb90 msgr2=0x7fb370102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:43.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.405+0000 7fb374c00640 1 -- 172.21.15.117:0/2880642631 shutdown_connections 2024-03-19T03:32:43.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.405+0000 7fb374c00640 1 -- 172.21.15.117:0/2880642631 wait complete. 2024-03-19T03:32:43.424 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T03:32:43.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.636+0000 7feebe600640 1 -- 172.21.15.117:0/1121356745 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feeb8101720 msgr2=0x7feeb8101b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:43.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.636+0000 7feebe600640 1 --2- 172.21.15.117:0/1121356745 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feeb8101720 0x7feeb8101b40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7feeac009900 tx=0x7feeac02bc30 comp rx=0 tx=0).stop 2024-03-19T03:32:43.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.636+0000 7feebe600640 1 -- 172.21.15.117:0/1121356745 shutdown_connections 2024-03-19T03:32:43.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.636+0000 7feebe600640 1 --2- 172.21.15.117:0/1121356745 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feeb81032d0 0x7feeb8109b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:43.637 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.636+0000 7feebe600640 1 --2- 172.21.15.117:0/1121356745 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feeb8102910 0x7feeb8102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:43.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.636+0000 7feebe600640 1 --2- 172.21.15.117:0/1121356745 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feeb8101720 0x7feeb8101b40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:43.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.636+0000 7feebe600640 1 -- 172.21.15.117:0/1121356745 >> 172.21.15.117:0/1121356745 conn(0x7feeb80fcc90 msgr2=0x7feeb80ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:43.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.636+0000 7feebe600640 1 -- 172.21.15.117:0/1121356745 shutdown_connections 2024-03-19T03:32:43.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.636+0000 7feebe600640 1 -- 172.21.15.117:0/1121356745 wait complete. 2024-03-19T03:32:43.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.638+0000 7feebe600640 1 Processor -- start 2024-03-19T03:32:43.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.638+0000 7feebe600640 1 -- start start 2024-03-19T03:32:43.640 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.638+0000 7feebe600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feeb8101720 0x7feeb80755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:43.640 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.638+0000 7feebe600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feeb8102910 0x7feeb8073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:43.640 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.638+0000 7feebe600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feeb81032d0 0x7feeb8074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:43.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.638+0000 7feebe600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7feeb810d370 con 0x7feeb8101720 2024-03-19T03:32:43.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.638+0000 7feebe600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7feeb810d1f0 con 0x7feeb8102910 2024-03-19T03:32:43.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.638+0000 7feebe600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7feeb810d4f0 con 0x7feeb81032d0 2024-03-19T03:32:43.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.639+0000 7feeb7e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feeb8102910 0x7feeb8073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:43.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.639+0000 7feebd200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feeb8101720 0x7feeb80755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:43.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.639+0000 7feeb7e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feeb8102910 0x7feeb8073c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:34470/0 (socket says 172.21.15.117:34470) 2024-03-19T03:32:43.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.639+0000 7feeb7e00640 1 -- 172.21.15.117:0/2340946319 learned_addr learned my addr 172.21.15.117:0/2340946319 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:43.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.639+0000 7feebdc00640 1 --2- 172.21.15.117:0/2340946319 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feeb81032d0 0x7feeb8074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:43.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.639+0000 7feeb7e00640 1 -- 172.21.15.117:0/2340946319 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feeb81032d0 msgr2=0x7feeb8074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:43.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.639+0000 7feeb7e00640 1 --2- 172.21.15.117:0/2340946319 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feeb81032d0 0x7feeb8074150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:43.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.639+0000 7feeb7e00640 1 -- 172.21.15.117:0/2340946319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feeb8101720 msgr2=0x7feeb80755d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:43.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.639+0000 7feeb7e00640 1 --2- 172.21.15.117:0/2340946319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feeb8101720 0x7feeb80755d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:43.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.639+0000 7feeb7e00640 1 -- 172.21.15.117:0/2340946319 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeb80749e0 con 0x7feeb8102910 2024-03-19T03:32:43.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.639+0000 7feebdc00640 1 --2- 172.21.15.117:0/2340946319 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feeb81032d0 0x7feeb8074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:32:43.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.639+0000 7feeb7e00640 1 --2- 172.21.15.117:0/2340946319 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feeb8102910 0x7feeb8073c10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7feea400d950 tx=0x7feea400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:43.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.640+0000 7feeb5600640 1 -- 172.21.15.117:0/2340946319 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea4017030 con 0x7feeb8102910 2024-03-19T03:32:43.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.640+0000 7feeb5600640 1 -- 172.21.15.117:0/2340946319 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feea4004020 con 0x7feeb8102910 2024-03-19T03:32:43.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.640+0000 7feebe600640 1 -- 172.21.15.117:0/2340946319 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeb81a1b90 con 0x7feeb8102910 2024-03-19T03:32:43.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.640+0000 7feeb5600640 1 -- 172.21.15.117:0/2340946319 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea40047a0 con 0x7feeb8102910 2024-03-19T03:32:43.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.640+0000 7feebe600640 1 -- 172.21.15.117:0/2340946319 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feeb81a2130 con 0x7feeb8102910 2024-03-19T03:32:43.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.641+0000 7feebe600640 1 -- 172.21.15.117:0/2340946319 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feeb8108ac0 con 0x7feeb8102910 2024-03-19T03:32:43.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.644+0000 7feeb5600640 1 -- 172.21.15.117:0/2340946319 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7feea40041d0 con 0x7feeb8102910 2024-03-19T03:32:43.647 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.646+0000 7feeb5600640 1 --2- 172.21.15.117:0/2340946319 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7feea00a2880 0x7feea00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:43.647 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.646+0000 7feeb5600640 1 -- 172.21.15.117:0/2340946319 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7feea40ce880 con 0x7feeb8102910 2024-03-19T03:32:43.647 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.646+0000 7feebd200640 1 --2- 172.21.15.117:0/2340946319 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7feea00a2880 0x7feea00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:43.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.647+0000 7feebd200640 1 --2- 172.21.15.117:0/2340946319 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7feea00a2880 0x7feea00a4d40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7feeac000c00 tx=0x7feeac005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:43.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.654+0000 7feeb5600640 1 -- 172.21.15.117:0/2340946319 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7feea409b530 con 0x7feeb8102910 2024-03-19T03:32:43.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:43 smithi119 ceph-mon[66861]: pgmap v385: 129 pgs: 1 activating, 128 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 767 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-19T03:32:43.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:43 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2944102668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:32:43.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:43 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3860160577' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:32:43.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:43 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2880642631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:32:43.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:43 smithi117 ceph-mon[31838]: pgmap v385: 129 pgs: 1 activating, 128 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 767 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-19T03:32:43.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:43 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2944102668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:32:43.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:43 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3860160577' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:32:43.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:43 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2880642631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:32:43.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:43 smithi178 ceph-mon[67578]: pgmap v385: 129 pgs: 1 activating, 128 active+clean; 647 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 767 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-19T03:32:43.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:43 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2944102668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:32:43.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:43 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3860160577' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:32:43.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:43 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2880642631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:32:43.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.890+0000 7feebe600640 1 -- 172.21.15.117:0/2340946319 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7feeb8062420 con 0x7feeb8102910 2024-03-19T03:32:43.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.891+0000 7feeb5600640 1 -- 172.21.15.117:0/2340946319 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7feea409ae30 con 0x7feeb8102910 2024-03-19T03:32:43.892 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 32 2024-03-19T03:32:43.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.893+0000 7feebe600640 1 -- 172.21.15.117:0/2340946319 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7feea00a2880 msgr2=0x7feea00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:43.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.893+0000 7feebe600640 1 --2- 172.21.15.117:0/2340946319 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7feea00a2880 0x7feea00a4d40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7feeac000c00 tx=0x7feeac005fd0 comp rx=0 tx=0).stop 2024-03-19T03:32:43.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.893+0000 7feebe600640 1 -- 172.21.15.117:0/2340946319 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feeb8102910 msgr2=0x7feeb8073c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:43.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.893+0000 7feebe600640 1 --2- 172.21.15.117:0/2340946319 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feeb8102910 0x7feeb8073c10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7feea400d950 tx=0x7feea400de10 comp rx=0 tx=0).stop 2024-03-19T03:32:43.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.893+0000 7feebe600640 1 -- 172.21.15.117:0/2340946319 shutdown_connections 2024-03-19T03:32:43.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.893+0000 7feebe600640 1 --2- 172.21.15.117:0/2340946319 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feeb81032d0 0x7feeb8074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:43.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.893+0000 7feebe600640 1 --2- 172.21.15.117:0/2340946319 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7feea00a2880 0x7feea00a4d40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:43.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.893+0000 7feebe600640 1 --2- 172.21.15.117:0/2340946319 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feeb8102910 0x7feeb8073c10 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:43.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.893+0000 7feebe600640 1 --2- 172.21.15.117:0/2340946319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feeb8101720 0x7feeb80755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:43.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.894+0000 7feebe600640 1 -- 172.21.15.117:0/2340946319 >> 172.21.15.117:0/2340946319 conn(0x7feeb80fcc90 msgr2=0x7feeb8105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:43.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.894+0000 7feebe600640 1 -- 172.21.15.117:0/2340946319 shutdown_connections 2024-03-19T03:32:43.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:43.894+0000 7feebe600640 1 -- 172.21.15.117:0/2340946319 wait complete. 2024-03-19T03:32:43.914 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T03:32:44.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.101+0000 7fe64b800640 1 -- 172.21.15.117:0/1395392627 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6441033e0 msgr2=0x7fe644109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:44.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.101+0000 7fe64b800640 1 --2- 172.21.15.117:0/1395392627 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6441033e0 0x7fe644109c70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe63c009ca0 tx=0x7fe63c02c100 comp rx=0 tx=0).stop 2024-03-19T03:32:44.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.101+0000 7fe64b800640 1 -- 172.21.15.117:0/1395392627 shutdown_connections 2024-03-19T03:32:44.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.101+0000 7fe64b800640 1 --2- 172.21.15.117:0/1395392627 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6441033e0 0x7fe644109c70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:44.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.101+0000 7fe64b800640 1 --2- 172.21.15.117:0/1395392627 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe644102a20 0x7fe644102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:44.102 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.101+0000 7fe64b800640 1 --2- 172.21.15.117:0/1395392627 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe644101830 0x7fe644101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:44.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.101+0000 7fe64b800640 1 -- 172.21.15.117:0/1395392627 >> 172.21.15.117:0/1395392627 conn(0x7fe6440fcdc0 msgr2=0x7fe6440ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:44.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.101+0000 7fe64b800640 1 -- 172.21.15.117:0/1395392627 shutdown_connections 2024-03-19T03:32:44.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.101+0000 7fe64b800640 1 -- 172.21.15.117:0/1395392627 wait complete. 2024-03-19T03:32:44.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.102+0000 7fe64b800640 1 Processor -- start 2024-03-19T03:32:44.103 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.103+0000 7fe64b800640 1 -- start start 2024-03-19T03:32:44.105 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.103+0000 7fe64b800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe644101830 0x7fe6440755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:44.105 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.103+0000 7fe64b800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe644102a20 0x7fe644073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:44.105 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.103+0000 7fe64b800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6441033e0 0x7fe644074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:44.105 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.103+0000 7fe64b800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe64410d4b0 con 0x7fe644102a20 2024-03-19T03:32:44.105 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.103+0000 7fe64b800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe64410d330 con 0x7fe6441033e0 2024-03-19T03:32:44.105 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.103+0000 7fe64b800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fe64410d630 con 0x7fe644101830 2024-03-19T03:32:44.105 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.103+0000 7fe643e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe644102a20 0x7fe644073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:44.105 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.103+0000 7fe649800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6441033e0 0x7fe644074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:44.105 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.103+0000 7fe643e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe644102a20 0x7fe644073c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:37512/0 (socket says 172.21.15.117:37512) 2024-03-19T03:32:44.105 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.103+0000 7fe643e00640 1 -- 172.21.15.117:0/1578147543 learned_addr learned my addr 172.21.15.117:0/1578147543 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:44.105 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.103+0000 7fe648e00640 1 --2- 172.21.15.117:0/1578147543 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe644101830 0x7fe6440755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:44.105 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.104+0000 7fe649800640 1 -- 172.21.15.117:0/1578147543 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe644101830 msgr2=0x7fe6440755d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:44.105 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.104+0000 7fe649800640 1 --2- 172.21.15.117:0/1578147543 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe644101830 0x7fe6440755d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:44.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.104+0000 7fe649800640 1 -- 172.21.15.117:0/1578147543 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe644102a20 msgr2=0x7fe644073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:44.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.104+0000 7fe649800640 1 --2- 172.21.15.117:0/1578147543 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe644102a20 0x7fe644073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:44.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.104+0000 7fe649800640 1 -- 172.21.15.117:0/1578147543 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6440749e0 con 0x7fe6441033e0 2024-03-19T03:32:44.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.104+0000 7fe649800640 1 --2- 172.21.15.117:0/1578147543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6441033e0 0x7fe644074150 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe63c02c640 tx=0x7fe63c035d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:44.106 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.104+0000 7fe641600640 1 -- 172.21.15.117:0/1578147543 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63c039040 con 0x7fe6441033e0 2024-03-19T03:32:44.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.104+0000 7fe641600640 1 -- 172.21.15.117:0/1578147543 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe63c044070 con 0x7fe6441033e0 2024-03-19T03:32:44.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.104+0000 7fe64b800640 1 -- 172.21.15.117:0/1578147543 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6441a1bb0 con 0x7fe6441033e0 2024-03-19T03:32:44.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.104+0000 7fe641600640 1 -- 172.21.15.117:0/1578147543 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63c0043e0 con 0x7fe6441033e0 2024-03-19T03:32:44.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.105+0000 7fe64b800640 1 -- 172.21.15.117:0/1578147543 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6441a20d0 con 0x7fe6441033e0 2024-03-19T03:32:44.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.106+0000 7fe64b800640 1 -- 172.21.15.117:0/1578147543 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe644108b50 con 0x7fe6441033e0 2024-03-19T03:32:44.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.109+0000 7fe641600640 1 -- 172.21.15.117:0/1578147543 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fe63c004580 con 0x7fe6441033e0 2024-03-19T03:32:44.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.110+0000 7fe641600640 1 --2- 172.21.15.117:0/1578147543 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe62c0a2800 0x7fe62c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:44.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.110+0000 7fe641600640 1 -- 172.21.15.117:0/1578147543 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe63c0f1500 con 0x7fe6441033e0 2024-03-19T03:32:44.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.111+0000 7fe648e00640 1 --2- 172.21.15.117:0/1578147543 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe62c0a2800 0x7fe62c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:44.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.111+0000 7fe648e00640 1 --2- 172.21.15.117:0/1578147543 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe62c0a2800 0x7fe62c0a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe644102880 tx=0x7fe638005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:44.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.114+0000 7fe641600640 1 -- 172.21.15.117:0/1578147543 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe63c0be150 con 0x7fe6441033e0 2024-03-19T03:32:44.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.320+0000 7fe64b800640 1 -- 172.21.15.117:0/1578147543 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe644062420 con 0x7fe6441033e0 2024-03-19T03:32:44.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.321+0000 7fe641600640 1 -- 172.21.15.117:0/1578147543 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fe63c0bda50 con 0x7fe6441033e0 2024-03-19T03:32:44.322 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 64 2024-03-19T03:32:44.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.323+0000 7fe64b800640 1 -- 172.21.15.117:0/1578147543 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe62c0a2800 msgr2=0x7fe62c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:44.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.323+0000 7fe64b800640 1 --2- 172.21.15.117:0/1578147543 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe62c0a2800 0x7fe62c0a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe644102880 tx=0x7fe638005f90 comp rx=0 tx=0).stop 2024-03-19T03:32:44.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.323+0000 7fe64b800640 1 -- 172.21.15.117:0/1578147543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6441033e0 msgr2=0x7fe644074150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:44.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.323+0000 7fe64b800640 1 --2- 172.21.15.117:0/1578147543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6441033e0 0x7fe644074150 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe63c02c640 tx=0x7fe63c035d90 comp rx=0 tx=0).stop 2024-03-19T03:32:44.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.324+0000 7fe64b800640 1 -- 172.21.15.117:0/1578147543 shutdown_connections 2024-03-19T03:32:44.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.324+0000 7fe64b800640 1 --2- 172.21.15.117:0/1578147543 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe62c0a2800 0x7fe62c0a4cc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:44.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.324+0000 7fe64b800640 1 --2- 172.21.15.117:0/1578147543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6441033e0 0x7fe644074150 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:44.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.324+0000 7fe64b800640 1 --2- 172.21.15.117:0/1578147543 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe644102a20 0x7fe644073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:44.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.324+0000 7fe64b800640 1 --2- 172.21.15.117:0/1578147543 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe644101830 0x7fe6440755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:44.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.324+0000 7fe64b800640 1 -- 172.21.15.117:0/1578147543 >> 172.21.15.117:0/1578147543 conn(0x7fe6440fcdc0 msgr2=0x7fe644105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:44.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.324+0000 7fe64b800640 1 -- 172.21.15.117:0/1578147543 shutdown_connections 2024-03-19T03:32:44.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.324+0000 7fe64b800640 1 -- 172.21.15.117:0/1578147543 wait complete. 2024-03-19T03:32:44.347 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T03:32:44.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.569+0000 7fe9f7800640 1 -- 172.21.15.117:0/3877673502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9f0107e40 msgr2=0x7fe9f0108260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:44.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.569+0000 7fe9f7800640 1 --2- 172.21.15.117:0/3877673502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9f0107e40 0x7fe9f0108260 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4009900 tx=0x7fe9e402bc30 comp rx=0 tx=0).stop 2024-03-19T03:32:44.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.569+0000 7fe9f7800640 1 -- 172.21.15.117:0/3877673502 shutdown_connections 2024-03-19T03:32:44.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.569+0000 7fe9f7800640 1 --2- 172.21.15.117:0/3877673502 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9f0109950 0x7fe9f0110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:44.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.569+0000 7fe9f7800640 1 --2- 172.21.15.117:0/3877673502 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9f0108f90 0x7fe9f0109410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:44.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.569+0000 7fe9f7800640 1 --2- 172.21.15.117:0/3877673502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9f0107e40 0x7fe9f0108260 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:44.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.569+0000 7fe9f7800640 1 -- 172.21.15.117:0/3877673502 >> 172.21.15.117:0/3877673502 conn(0x7fe9f0076de0 msgr2=0x7fe9f0079220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:44.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.569+0000 7fe9f7800640 1 -- 172.21.15.117:0/3877673502 shutdown_connections 2024-03-19T03:32:44.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.569+0000 7fe9f7800640 1 -- 172.21.15.117:0/3877673502 wait complete. 2024-03-19T03:32:44.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.570+0000 7fe9f7800640 1 Processor -- start 2024-03-19T03:32:44.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.570+0000 7fe9f7800640 1 -- start start 2024-03-19T03:32:44.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.571+0000 7fe9f7800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9f0107e40 0x7fe9f019e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:44.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.571+0000 7fe9f7800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9f0108f90 0x7fe9f019e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:44.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.571+0000 7fe9f7800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9f0109950 0x7fe9f01a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:44.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.571+0000 7fe9f7800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f0113b30 con 0x7fe9f0108f90 2024-03-19T03:32:44.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.571+0000 7fe9f7800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f01139b0 con 0x7fe9f0107e40 2024-03-19T03:32:44.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.571+0000 7fe9f7800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f0113cb0 con 0x7fe9f0109950 2024-03-19T03:32:44.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.571+0000 7fe9f4e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9f0107e40 0x7fe9f019e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:44.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.571+0000 7fe9efe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9f0108f90 0x7fe9f019e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:44.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.571+0000 7fe9f5800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9f0109950 0x7fe9f01a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:44.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.571+0000 7fe9f4e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9f0107e40 0x7fe9f019e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:34502/0 (socket says 172.21.15.117:34502) 2024-03-19T03:32:44.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.571+0000 7fe9f4e00640 1 -- 172.21.15.117:0/3246507774 learned_addr learned my addr 172.21.15.117:0/3246507774 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:44.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.572+0000 7fe9f4e00640 1 -- 172.21.15.117:0/3246507774 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9f0109950 msgr2=0x7fe9f01a5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:44.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.572+0000 7fe9f4e00640 1 --2- 172.21.15.117:0/3246507774 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9f0109950 0x7fe9f01a5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:44.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.572+0000 7fe9f4e00640 1 -- 172.21.15.117:0/3246507774 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9f0108f90 msgr2=0x7fe9f019e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:44.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.572+0000 7fe9f4e00640 1 --2- 172.21.15.117:0/3246507774 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9f0108f90 0x7fe9f019e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:44.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.572+0000 7fe9f4e00640 1 -- 172.21.15.117:0/3246507774 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9f01a5d70 con 0x7fe9f0107e40 2024-03-19T03:32:44.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.572+0000 7fe9f4e00640 1 --2- 172.21.15.117:0/3246507774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9f0107e40 0x7fe9f019e150 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4000c00 tx=0x7fe9e402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:44.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.572+0000 7fe9ed600640 1 -- 172.21.15.117:0/3246507774 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4043020 con 0x7fe9f0107e40 2024-03-19T03:32:44.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.572+0000 7fe9ed600640 1 -- 172.21.15.117:0/3246507774 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9e40054e0 con 0x7fe9f0107e40 2024-03-19T03:32:44.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.572+0000 7fe9ed600640 1 -- 172.21.15.117:0/3246507774 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4003bd0 con 0x7fe9f0107e40 2024-03-19T03:32:44.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.572+0000 7fe9f7800640 1 -- 172.21.15.117:0/3246507774 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9f01a6000 con 0x7fe9f0107e40 2024-03-19T03:32:44.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.573+0000 7fe9f7800640 1 -- 172.21.15.117:0/3246507774 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9f01a6500 con 0x7fe9f0107e40 2024-03-19T03:32:44.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.574+0000 7fe9f7800640 1 -- 172.21.15.117:0/3246507774 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9f010f150 con 0x7fe9f0107e40 2024-03-19T03:32:44.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.575+0000 7fe9ed600640 1 -- 172.21.15.117:0/3246507774 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fe9e4046020 con 0x7fe9f0107e40 2024-03-19T03:32:44.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.576+0000 7fe9ed600640 1 --2- 172.21.15.117:0/3246507774 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe9d80a2880 0x7fe9d80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:44.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.577+0000 7fe9ed600640 1 -- 172.21.15.117:0/3246507774 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe9e40f0e50 con 0x7fe9f0107e40 2024-03-19T03:32:44.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.577+0000 7fe9efe00640 1 --2- 172.21.15.117:0/3246507774 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe9d80a2880 0x7fe9d80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:44.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.577+0000 7fe9efe00640 1 --2- 172.21.15.117:0/3246507774 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe9d80a2880 0x7fe9d80a4d40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe9f019f660 tx=0x7fe9dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:44.581 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.580+0000 7fe9ed600640 1 -- 172.21.15.117:0/3246507774 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe9e404d020 con 0x7fe9f0107e40 2024-03-19T03:32:44.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:44 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2340946319' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:32:44.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:44 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1578147543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:32:44.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:44 smithi117 ceph-mon[31838]: pgmap v386: 129 pgs: 1 activating, 128 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 895 B/s wr, 10 op/s; 7.9 KiB/s, 1 keys/s, 2 objects/s recovering 2024-03-19T03:32:44.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.819+0000 7fe9f7800640 1 -- 172.21.15.117:0/3246507774 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe9f0062420 con 0x7fe9f0107e40 2024-03-19T03:32:44.820 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.819+0000 7fe9ed600640 1 -- 172.21.15.117:0/3246507774 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fe9e40bdb00 con 0x7fe9f0107e40 2024-03-19T03:32:44.820 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 32 2024-03-19T03:32:44.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.822+0000 7fe9f7800640 1 -- 172.21.15.117:0/3246507774 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe9d80a2880 msgr2=0x7fe9d80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:44.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.822+0000 7fe9f7800640 1 --2- 172.21.15.117:0/3246507774 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe9d80a2880 0x7fe9d80a4d40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe9f019f660 tx=0x7fe9dc005f90 comp rx=0 tx=0).stop 2024-03-19T03:32:44.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.822+0000 7fe9f7800640 1 -- 172.21.15.117:0/3246507774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9f0107e40 msgr2=0x7fe9f019e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:44.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.822+0000 7fe9f7800640 1 --2- 172.21.15.117:0/3246507774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9f0107e40 0x7fe9f019e150 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4000c00 tx=0x7fe9e402c650 comp rx=0 tx=0).stop 2024-03-19T03:32:44.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.823+0000 7fe9f7800640 1 -- 172.21.15.117:0/3246507774 shutdown_connections 2024-03-19T03:32:44.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.823+0000 7fe9f7800640 1 --2- 172.21.15.117:0/3246507774 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9f0109950 0x7fe9f01a5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:44.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.823+0000 7fe9f7800640 1 --2- 172.21.15.117:0/3246507774 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9f0108f90 0x7fe9f019e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:44.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.823+0000 7fe9f7800640 1 --2- 172.21.15.117:0/3246507774 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe9d80a2880 0x7fe9d80a4d40 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:44.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.823+0000 7fe9f7800640 1 --2- 172.21.15.117:0/3246507774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9f0107e40 0x7fe9f019e150 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:44.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.823+0000 7fe9f7800640 1 -- 172.21.15.117:0/3246507774 >> 172.21.15.117:0/3246507774 conn(0x7fe9f0076de0 msgr2=0x7fe9f010c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:44.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.823+0000 7fe9f7800640 1 -- 172.21.15.117:0/3246507774 shutdown_connections 2024-03-19T03:32:44.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:44.823+0000 7fe9f7800640 1 -- 172.21.15.117:0/3246507774 wait complete. 2024-03-19T03:32:44.843 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:32:44.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:44 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2340946319' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:32:44.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:44 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1578147543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:32:44.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:44 smithi178 ceph-mon[67578]: pgmap v386: 129 pgs: 1 activating, 128 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 895 B/s wr, 10 op/s; 7.9 KiB/s, 1 keys/s, 2 objects/s recovering 2024-03-19T03:32:44.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:44 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2340946319' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:32:44.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:44 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1578147543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:32:44.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:44 smithi119 ceph-mon[66861]: pgmap v386: 129 pgs: 1 activating, 128 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 895 B/s wr, 10 op/s; 7.9 KiB/s, 1 keys/s, 2 objects/s recovering 2024-03-19T03:32:45.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.072+0000 7f11b9000640 1 -- 172.21.15.117:0/2426580081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f11b4107350 msgr2=0x7f11b41097a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:45.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.072+0000 7f11b9000640 1 --2- 172.21.15.117:0/2426580081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f11b4107350 0x7f11b41097a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f11ac009ca0 tx=0x7f11ac02c100 comp rx=0 tx=0).stop 2024-03-19T03:32:45.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.072+0000 7f11b9000640 1 -- 172.21.15.117:0/2426580081 shutdown_connections 2024-03-19T03:32:45.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.072+0000 7f11b9000640 1 --2- 172.21.15.117:0/2426580081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f11b4107350 0x7f11b41097a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:45.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.072+0000 7f11b9000640 1 --2- 172.21.15.117:0/2426580081 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11b4104a20 0x7f11b4106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:45.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.072+0000 7f11b9000640 1 --2- 172.21.15.117:0/2426580081 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f11b41020f0 0x7f11b41044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:45.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.072+0000 7f11b9000640 1 -- 172.21.15.117:0/2426580081 >> 172.21.15.117:0/2426580081 conn(0x7f11b40fbb20 msgr2=0x7f11b40fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:45.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.072+0000 7f11b9000640 1 -- 172.21.15.117:0/2426580081 shutdown_connections 2024-03-19T03:32:45.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.072+0000 7f11b9000640 1 -- 172.21.15.117:0/2426580081 wait complete. 2024-03-19T03:32:45.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.073+0000 7f11b9000640 1 Processor -- start 2024-03-19T03:32:45.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.073+0000 7f11b9000640 1 -- start start 2024-03-19T03:32:45.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.074+0000 7f11b9000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f11b41020f0 0x7f11b4199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:45.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.074+0000 7f11b9000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11b4104a20 0x7f11b419a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:45.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.074+0000 7f11b9000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f11b4107350 0x7f11b41a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:45.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.074+0000 7f11b9000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f11b410d2f0 con 0x7f11b4104a20 2024-03-19T03:32:45.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.074+0000 7f11b9000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f11b410d170 con 0x7f11b4107350 2024-03-19T03:32:45.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.074+0000 7f11b9000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f11b410d470 con 0x7f11b41020f0 2024-03-19T03:32:45.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.074+0000 7f11abe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11b4104a20 0x7f11b419a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:45.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.074+0000 7f11b3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f11b4107350 0x7f11b41a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:45.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.074+0000 7f11abe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11b4104a20 0x7f11b419a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:37544/0 (socket says 172.21.15.117:37544) 2024-03-19T03:32:45.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.074+0000 7f11abe00640 1 -- 172.21.15.117:0/1688127116 learned_addr learned my addr 172.21.15.117:0/1688127116 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:45.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.074+0000 7f11b3400640 1 --2- 172.21.15.117:0/1688127116 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f11b41020f0 0x7f11b4199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:45.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.075+0000 7f11b3e00640 1 -- 172.21.15.117:0/1688127116 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f11b41020f0 msgr2=0x7f11b4199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:45.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.075+0000 7f11b3e00640 1 --2- 172.21.15.117:0/1688127116 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f11b41020f0 0x7f11b4199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:45.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.075+0000 7f11b3e00640 1 -- 172.21.15.117:0/1688127116 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11b4104a20 msgr2=0x7f11b419a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:45.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.075+0000 7f11b3e00640 1 --2- 172.21.15.117:0/1688127116 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11b4104a20 0x7f11b419a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:45.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.075+0000 7f11b3e00640 1 -- 172.21.15.117:0/1688127116 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11b41a1760 con 0x7f11b4107350 2024-03-19T03:32:45.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.075+0000 7f11abe00640 1 --2- 172.21.15.117:0/1688127116 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11b4104a20 0x7f11b419a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:32:45.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.075+0000 7f11b3e00640 1 --2- 172.21.15.117:0/1688127116 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f11b4107350 0x7f11b41a10f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f11ac02c640 tx=0x7f11ac035d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:45.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.075+0000 7f11b0c00640 1 -- 172.21.15.117:0/1688127116 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11ac039040 con 0x7f11b4107350 2024-03-19T03:32:45.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.075+0000 7f11b0c00640 1 -- 172.21.15.117:0/1688127116 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11ac044070 con 0x7f11b4107350 2024-03-19T03:32:45.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.075+0000 7f11b9000640 1 -- 172.21.15.117:0/1688127116 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11b41a19f0 con 0x7f11b4107350 2024-03-19T03:32:45.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.075+0000 7f11b0c00640 1 -- 172.21.15.117:0/1688127116 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11ac0044a0 con 0x7f11b4107350 2024-03-19T03:32:45.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.076+0000 7f11b9000640 1 -- 172.21.15.117:0/1688127116 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11b41a1eb0 con 0x7f11b4107350 2024-03-19T03:32:45.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.077+0000 7f11b9000640 1 -- 172.21.15.117:0/1688127116 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11b4108e40 con 0x7f11b4107350 2024-03-19T03:32:45.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.079+0000 7f11b0c00640 1 -- 172.21.15.117:0/1688127116 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f11ac004640 con 0x7f11b4107350 2024-03-19T03:32:45.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.080+0000 7f11b0c00640 1 --2- 172.21.15.117:0/1688127116 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f11980a2800 0x7f11980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:45.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.080+0000 7f11b0c00640 1 -- 172.21.15.117:0/1688127116 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f11ac0f14f0 con 0x7f11b4107350 2024-03-19T03:32:45.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.080+0000 7f11b3400640 1 --2- 172.21.15.117:0/1688127116 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f11980a2800 0x7f11980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:45.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.081+0000 7f11b3400640 1 --2- 172.21.15.117:0/1688127116 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f11980a2800 0x7f11980a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f11a4005d00 tx=0x7f11a400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:45.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.088+0000 7f11b0c00640 1 -- 172.21.15.117:0/1688127116 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f11ac0be1a0 con 0x7f11b4107350 2024-03-19T03:32:45.431 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.430+0000 7f11b9000640 1 -- 172.21.15.117:0/1688127116 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11b4062420 con 0x7f11b4107350 2024-03-19T03:32:45.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.431+0000 7f11b0c00640 1 -- 172.21.15.117:0/1688127116 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11953 (secure 0 0 0) 0x7f11ac0098b0 con 0x7f11b4107350 2024-03-19T03:32:45.432 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:32:45.432 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/54638259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":54638259},{"type":"v1","addr":"172.21.15.117:6835","nonce":54638259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3608185039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3608185039},{"type":"v1","addr":"172.21.15.117:6837","nonce":3608185039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T03:30:35.081740+0000","modified":"2024-03-19T03:31:09.995133+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24599,"mds_1":24581,"mds_2":24569,"mds_3":24545,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":20,"addr":"172.21.15.119:6833/1291833214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1291833214},{"type":"v1","addr":"172.21.15.119:6833","nonce":1291833214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6835/1579860804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1579860804},{"type":"v1","addr":"172.21.15.178:6835","nonce":1579860804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6835/1686938941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1686938941},{"type":"v1","addr":"172.21.15.119:6835","nonce":1686938941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6837/1770913538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":1770913538},{"type":"v1","addr":"172.21.15.178:6837","nonce":1770913538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6839/2655593383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2655593383},{"type":"v1","addr":"172.21.15.178:6839","nonce":2655593383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.119:6837/2054992738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2054992738},{"type":"v1","addr":"172.21.15.119:6837","nonce":2054992738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"g","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.117:6839/684335078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":684335078},{"type":"v1","addr":"172.21.15.117:6839","nonce":684335078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.117:6841/1329933892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6840","nonce":1329933892},{"type":"v1","addr":"172.21.15.117:6841","nonce":1329933892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/2383354833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2383354833},{"type":"v1","addr":"172.21.15.119:6839","nonce":2383354833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24581,24599,24569,24545,24467]},"id":1}]} 2024-03-19T03:32:45.433 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 28 2024-03-19T03:32:45.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.434+0000 7f11b9000640 1 -- 172.21.15.117:0/1688127116 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f11980a2800 msgr2=0x7f11980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:45.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.434+0000 7f11b9000640 1 --2- 172.21.15.117:0/1688127116 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f11980a2800 0x7f11980a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f11a4005d00 tx=0x7f11a400b410 comp rx=0 tx=0).stop 2024-03-19T03:32:45.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.434+0000 7f11b9000640 1 -- 172.21.15.117:0/1688127116 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f11b4107350 msgr2=0x7f11b41a10f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:45.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.434+0000 7f11b9000640 1 --2- 172.21.15.117:0/1688127116 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f11b4107350 0x7f11b41a10f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f11ac02c640 tx=0x7f11ac035d90 comp rx=0 tx=0).stop 2024-03-19T03:32:45.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.434+0000 7f11b9000640 1 -- 172.21.15.117:0/1688127116 shutdown_connections 2024-03-19T03:32:45.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.434+0000 7f11b9000640 1 --2- 172.21.15.117:0/1688127116 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f11980a2800 0x7f11980a4cc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:45.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.434+0000 7f11b9000640 1 --2- 172.21.15.117:0/1688127116 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f11b4107350 0x7f11b41a10f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:45.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.434+0000 7f11b9000640 1 --2- 172.21.15.117:0/1688127116 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11b4104a20 0x7f11b419a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:45.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.434+0000 7f11b9000640 1 --2- 172.21.15.117:0/1688127116 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f11b41020f0 0x7f11b4199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:45.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.434+0000 7f11b9000640 1 -- 172.21.15.117:0/1688127116 >> 172.21.15.117:0/1688127116 conn(0x7f11b40fbb20 msgr2=0x7f11b41017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:45.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.434+0000 7f11b9000640 1 -- 172.21.15.117:0/1688127116 shutdown_connections 2024-03-19T03:32:45.436 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.434+0000 7f11b9000640 1 -- 172.21.15.117:0/1688127116 wait complete. 2024-03-19T03:32:45.458 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:32:45.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.659+0000 7f0a00200640 1 -- 172.21.15.117:0/984204909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09f81022c0 msgr2=0x7f09f81046b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:45.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.659+0000 7f0a00200640 1 --2- 172.21.15.117:0/984204909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09f81022c0 0x7f09f81046b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f09ec009ca0 tx=0x7f09ec02c100 comp rx=0 tx=0).stop 2024-03-19T03:32:45.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.659+0000 7f0a00200640 1 -- 172.21.15.117:0/984204909 shutdown_connections 2024-03-19T03:32:45.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.659+0000 7f0a00200640 1 --2- 172.21.15.117:0/984204909 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09f8107520 0x7f09f8109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:45.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.659+0000 7f0a00200640 1 --2- 172.21.15.117:0/984204909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09f8104bf0 0x7f09f8106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:45.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.659+0000 7f0a00200640 1 --2- 172.21.15.117:0/984204909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09f81022c0 0x7f09f81046b0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:45.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.659+0000 7f0a00200640 1 -- 172.21.15.117:0/984204909 >> 172.21.15.117:0/984204909 conn(0x7f09f80fbcd0 msgr2=0x7f09f80fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:45.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.659+0000 7f0a00200640 1 -- 172.21.15.117:0/984204909 shutdown_connections 2024-03-19T03:32:45.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.659+0000 7f0a00200640 1 -- 172.21.15.117:0/984204909 wait complete. 2024-03-19T03:32:45.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.660+0000 7f0a00200640 1 Processor -- start 2024-03-19T03:32:45.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.660+0000 7f0a00200640 1 -- start start 2024-03-19T03:32:45.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.660+0000 7f0a00200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09f81022c0 0x7f09f8199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:45.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.660+0000 7f0a00200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09f8104bf0 0x7f09f819a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:45.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.661+0000 7f0a00200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09f8107520 0x7f09f81a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:45.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.661+0000 7f0a00200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f09f810d4c0 con 0x7f09f81022c0 2024-03-19T03:32:45.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.661+0000 7f0a00200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f09f810d340 con 0x7f09f8107520 2024-03-19T03:32:45.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.661+0000 7f0a00200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f09f810d640 con 0x7f09f8104bf0 2024-03-19T03:32:45.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.661+0000 7f09fd800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09f81022c0 0x7f09f8199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:45.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.661+0000 7f09fe200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09f8107520 0x7f09f81a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:45.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.661+0000 7f09fd800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09f81022c0 0x7f09f8199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:37566/0 (socket says 172.21.15.117:37566) 2024-03-19T03:32:45.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.661+0000 7f09fe200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09f8107520 0x7f09f81a1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:34546/0 (socket says 172.21.15.117:34546) 2024-03-19T03:32:45.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.661+0000 7f09fd800640 1 -- 172.21.15.117:0/3993606018 learned_addr learned my addr 172.21.15.117:0/3993606018 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:45.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.661+0000 7f09fce00640 1 --2- 172.21.15.117:0/3993606018 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09f8104bf0 0x7f09f819a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:45.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.661+0000 7f09fe200640 1 -- 172.21.15.117:0/3993606018 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09f8104bf0 msgr2=0x7f09f819a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:45.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.661+0000 7f09fe200640 1 --2- 172.21.15.117:0/3993606018 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09f8104bf0 0x7f09f819a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:45.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.661+0000 7f09fe200640 1 -- 172.21.15.117:0/3993606018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09f81022c0 msgr2=0x7f09f8199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:45.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.661+0000 7f09fe200640 1 --2- 172.21.15.117:0/3993606018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09f81022c0 0x7f09f8199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:45.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.661+0000 7f09fe200640 1 -- 172.21.15.117:0/3993606018 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f81a19c0 con 0x7f09f8107520 2024-03-19T03:32:45.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.662+0000 7f09fe200640 1 --2- 172.21.15.117:0/3993606018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09f8107520 0x7f09f81a1320 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f09f000d6b0 tx=0x7f09f000db70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:45.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.662+0000 7f09f6000640 1 -- 172.21.15.117:0/3993606018 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f0017070 con 0x7f09f8107520 2024-03-19T03:32:45.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.662+0000 7f09f6000640 1 -- 172.21.15.117:0/3993606018 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09f0009d00 con 0x7f09f8107520 2024-03-19T03:32:45.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.662+0000 7f0a00200640 1 -- 172.21.15.117:0/3993606018 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09f81a1cb0 con 0x7f09f8107520 2024-03-19T03:32:45.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.662+0000 7f09f6000640 1 -- 172.21.15.117:0/3993606018 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f0003d60 con 0x7f09f8107520 2024-03-19T03:32:45.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.662+0000 7f0a00200640 1 -- 172.21.15.117:0/3993606018 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09f81a21b0 con 0x7f09f8107520 2024-03-19T03:32:45.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.663+0000 7f0a00200640 1 -- 172.21.15.117:0/3993606018 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09f8109070 con 0x7f09f8107520 2024-03-19T03:32:45.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.665+0000 7f09f6000640 1 -- 172.21.15.117:0/3993606018 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f09f0009820 con 0x7f09f8107520 2024-03-19T03:32:45.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.667+0000 7f09f6000640 1 --2- 172.21.15.117:0/3993606018 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f09e00a2880 0x7f09e00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:45.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.667+0000 7f09f6000640 1 -- 172.21.15.117:0/3993606018 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f09f00ceae0 con 0x7f09f8107520 2024-03-19T03:32:45.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.668+0000 7f09fd800640 1 --2- 172.21.15.117:0/3993606018 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f09e00a2880 0x7f09e00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:45.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.668+0000 7f09f6000640 1 -- 172.21.15.117:0/3993606018 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f09f009b160 con 0x7f09f8107520 2024-03-19T03:32:45.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.668+0000 7f09fd800640 1 --2- 172.21.15.117:0/3993606018 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f09e00a2880 0x7f09e00a4d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f09f819b280 tx=0x7f09ec004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:45.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:45 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3246507774' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:32:45.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:45 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1688127116' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:32:45.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:45 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3246507774' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:32:45.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:45 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1688127116' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:32:45.881 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.880+0000 7f0a00200640 1 -- 172.21.15.117:0/3993606018 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f09f802d3f0 con 0x7f09f8107520 2024-03-19T03:32:45.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.883+0000 7f09f6000640 1 -- 172.21.15.117:0/3993606018 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24989 (secure 0 0 0) 0x7f09f009f200 con 0x7f09f8107520 2024-03-19T03:32:45.884 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:32:45.884 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":93,"fsid":"a3a82456-e59f-11ee-95c9-87774f69a715","created":"2024-03-19T03:21:30.427136+0000","modified":"2024-03-19T03:32:40.453890+0000","last_up_change":"2024-03-19T03:29:32.967505+0000","last_in_change":"2024-03-19T03:29:12.848740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:25:18.824270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T03:30:33.137170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T03:30:34.598838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T03:30:38.230023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b32eb0ee-9d4b-4748-87ec-eda02c907415","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6803","nonce":3014934900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6805","nonce":3014934900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6809","nonce":3014934900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6807","nonce":3014934900}]},"public_addr":"172.21.15.117:6803/3014934900","cluster_addr":"172.21.15.117:6805/3014934900","heartbeat_back_addr":"172.21.15.117:6809/3014934900","heartbeat_front_addr":"172.21.15.117:6807/3014934900","state":["exists","up"]},{"osd":1,"uuid":"e07585be-c016-4466-b07c-abbc22a58de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6801","nonce":2425927341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6803","nonce":2425927341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6807","nonce":2425927341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6805","nonce":2425927341}]},"public_addr":"172.21.15.119:6801/2425927341","cluster_addr":"172.21.15.119:6803/2425927341","heartbeat_back_addr":"172.21.15.119:6807/2425927341","heartbeat_front_addr":"172.21.15.119:6805/2425927341","state":["exists","up"]},{"osd":2,"uuid":"ede9edda-3b9f-41ca-ad04-5dcb738ab757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6801","nonce":3733812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6803","nonce":3733812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6807","nonce":3733812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6805","nonce":3733812989}]},"public_addr":"172.21.15.178:6801/3733812989","cluster_addr":"172.21.15.178:6803/3733812989","heartbeat_back_addr":"172.21.15.178:6807/3733812989","heartbeat_front_addr":"172.21.15.178:6805/3733812989","state":["exists","up"]},{"osd":3,"uuid":"f81c41e1-3a69-40db-a90f-58151286b081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6811","nonce":2178310135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6813","nonce":2178310135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6817","nonce":2178310135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6815","nonce":2178310135}]},"public_addr":"172.21.15.117:6811/2178310135","cluster_addr":"172.21.15.117:6813/2178310135","heartbeat_back_addr":"172.21.15.117:6817/2178310135","heartbeat_front_addr":"172.21.15.117:6815/2178310135","state":["exists","up"]},{"osd":4,"uuid":"caf7ce86-caf0-4eaf-bbab-bd2d93e63c33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6809","nonce":3455398086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6811","nonce":3455398086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6815","nonce":3455398086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6813","nonce":3455398086}]},"public_addr":"172.21.15.119:6809/3455398086","cluster_addr":"172.21.15.119:6811/3455398086","heartbeat_back_addr":"172.21.15.119:6815/3455398086","heartbeat_front_addr":"172.21.15.119:6813/3455398086","state":["exists","up"]},{"osd":5,"uuid":"20ba0b8a-2e0f-4ea7-b435-430e44766200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6809","nonce":1259631912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6811","nonce":1259631912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6815","nonce":1259631912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6813","nonce":1259631912}]},"public_addr":"172.21.15.178:6809/1259631912","cluster_addr":"172.21.15.178:6811/1259631912","heartbeat_back_addr":"172.21.15.178:6815/1259631912","heartbeat_front_addr":"172.21.15.178:6813/1259631912","state":["exists","up"]},{"osd":6,"uuid":"75c2ddd7-f837-42e0-ac47-8cfbcf97a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6819","nonce":864146757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6821","nonce":864146757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6825","nonce":864146757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6823","nonce":864146757}]},"public_addr":"172.21.15.117:6819/864146757","cluster_addr":"172.21.15.117:6821/864146757","heartbeat_back_addr":"172.21.15.117:6825/864146757","heartbeat_front_addr":"172.21.15.117:6823/864146757","state":["exists","up"]},{"osd":7,"uuid":"929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6817","nonce":1273365545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6819","nonce":1273365545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6823","nonce":1273365545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6821","nonce":1273365545}]},"public_addr":"172.21.15.119:6817/1273365545","cluster_addr":"172.21.15.119:6819/1273365545","heartbeat_back_addr":"172.21.15.119:6823/1273365545","heartbeat_front_addr":"172.21.15.119:6821/1273365545","state":["exists","up"]},{"osd":8,"uuid":"fd371461-a538-4781-92e1-5eead4ca3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6817","nonce":1059587501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6819","nonce":1059587501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6823","nonce":1059587501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6821","nonce":1059587501}]},"public_addr":"172.21.15.178:6817/1059587501","cluster_addr":"172.21.15.178:6819/1059587501","heartbeat_back_addr":"172.21.15.178:6823/1059587501","heartbeat_front_addr":"172.21.15.178:6821/1059587501","state":["exists","up"]},{"osd":9,"uuid":"0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6827","nonce":3653875600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6829","nonce":3653875600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6833","nonce":3653875600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6831","nonce":3653875600}]},"public_addr":"172.21.15.117:6827/3653875600","cluster_addr":"172.21.15.117:6829/3653875600","heartbeat_back_addr":"172.21.15.117:6833/3653875600","heartbeat_front_addr":"172.21.15.117:6831/3653875600","state":["exists","up"]},{"osd":10,"uuid":"de2049da-d19e-4386-afbf-fd6b5fddbcba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6825","nonce":1050976194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6827","nonce":1050976194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6831","nonce":1050976194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6829","nonce":1050976194}]},"public_addr":"172.21.15.119:6825/1050976194","cluster_addr":"172.21.15.119:6827/1050976194","heartbeat_back_addr":"172.21.15.119:6831/1050976194","heartbeat_front_addr":"172.21.15.119:6829/1050976194","state":["exists","up"]},{"osd":11,"uuid":"3cccc0f9-d273-4b6b-9b8e-c926371472dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6825","nonce":2915080368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6827","nonce":2915080368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6831","nonce":2915080368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6829","nonce":2915080368}]},"public_addr":"172.21.15.178:6825/2915080368","cluster_addr":"172.21.15.178:6827/2915080368","heartbeat_back_addr":"172.21.15.178:6831/2915080368","heartbeat_front_addr":"172.21.15.178:6829/2915080368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:27.876274+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:53.094706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:16.598813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:48.160419+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:14.048925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:38.814304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:12.542328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:38.942747+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:04.488963+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:38.196053+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:05.254728+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:30.874915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7},{"from":5,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.117:0/185684447":"2024-03-20T03:22:38.134578+0000","172.21.15.117:0/361367101":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/2788067603":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3506200675":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6800/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3283471541":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3750920169":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/2780813298":"2024-03-20T03:22:03.905461+0000","172.21.15.117:6800/2788067603":"2024-03-20T03:22:38.134578+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:32:45.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.885+0000 7f0a00200640 1 -- 172.21.15.117:0/3993606018 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f09e00a2880 msgr2=0x7f09e00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:45.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.885+0000 7f0a00200640 1 --2- 172.21.15.117:0/3993606018 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f09e00a2880 0x7f09e00a4d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f09f819b280 tx=0x7f09ec004a40 comp rx=0 tx=0).stop 2024-03-19T03:32:45.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.885+0000 7f0a00200640 1 -- 172.21.15.117:0/3993606018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09f8107520 msgr2=0x7f09f81a1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:45.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.885+0000 7f0a00200640 1 --2- 172.21.15.117:0/3993606018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09f8107520 0x7f09f81a1320 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f09f000d6b0 tx=0x7f09f000db70 comp rx=0 tx=0).stop 2024-03-19T03:32:45.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.886+0000 7f0a00200640 1 -- 172.21.15.117:0/3993606018 shutdown_connections 2024-03-19T03:32:45.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.886+0000 7f0a00200640 1 --2- 172.21.15.117:0/3993606018 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f09e00a2880 0x7f09e00a4d40 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:45.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.886+0000 7f0a00200640 1 --2- 172.21.15.117:0/3993606018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09f8107520 0x7f09f81a1320 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:45.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.886+0000 7f0a00200640 1 --2- 172.21.15.117:0/3993606018 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09f8104bf0 0x7f09f819a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:45.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.886+0000 7f0a00200640 1 --2- 172.21.15.117:0/3993606018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09f81022c0 0x7f09f8199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:45.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.886+0000 7f0a00200640 1 -- 172.21.15.117:0/3993606018 >> 172.21.15.117:0/3993606018 conn(0x7f09f80fbcd0 msgr2=0x7f09f81019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:45.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.886+0000 7f0a00200640 1 -- 172.21.15.117:0/3993606018 shutdown_connections 2024-03-19T03:32:45.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:45.886+0000 7f0a00200640 1 -- 172.21.15.117:0/3993606018 wait complete. 2024-03-19T03:32:45.908 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:32:45.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:45 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3246507774' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:32:45.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:45 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1688127116' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:32:46.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.128+0000 7fd54ca00640 1 -- 172.21.15.117:0/2646139825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd548102a30 msgr2=0x7fd548102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:46.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.128+0000 7fd54ca00640 1 --2- 172.21.15.117:0/2646139825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd548102a30 0x7fd548102eb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd534009900 tx=0x7fd53402bc30 comp rx=0 tx=0).stop 2024-03-19T03:32:46.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.128+0000 7fd54ca00640 1 -- 172.21.15.117:0/2646139825 shutdown_connections 2024-03-19T03:32:46.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.128+0000 7fd54ca00640 1 --2- 172.21.15.117:0/2646139825 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5481033f0 0x7fd548109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.128+0000 7fd54ca00640 1 --2- 172.21.15.117:0/2646139825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd548102a30 0x7fd548102eb0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.128+0000 7fd54ca00640 1 --2- 172.21.15.117:0/2646139825 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd548101840 0x7fd548101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.128+0000 7fd54ca00640 1 -- 172.21.15.117:0/2646139825 >> 172.21.15.117:0/2646139825 conn(0x7fd5480fcdd0 msgr2=0x7fd5480ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:46.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.128+0000 7fd54ca00640 1 -- 172.21.15.117:0/2646139825 shutdown_connections 2024-03-19T03:32:46.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.128+0000 7fd54ca00640 1 -- 172.21.15.117:0/2646139825 wait complete. 2024-03-19T03:32:46.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.129+0000 7fd54ca00640 1 Processor -- start 2024-03-19T03:32:46.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.130+0000 7fd54ca00640 1 -- start start 2024-03-19T03:32:46.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.130+0000 7fd54ca00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd548101840 0x7fd548197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.130+0000 7fd54ca00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd548102a30 0x7fd5481980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.130+0000 7fd54ca00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5481033f0 0x7fd54819f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.130+0000 7fd54ca00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd54810d4f0 con 0x7fd5481033f0 2024-03-19T03:32:46.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.130+0000 7fd54ca00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fd54810d370 con 0x7fd548102a30 2024-03-19T03:32:46.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.130+0000 7fd54ca00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fd54810d670 con 0x7fd548101840 2024-03-19T03:32:46.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.130+0000 7fd545400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd548102a30 0x7fd5481980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.130+0000 7fd545400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd548102a30 0x7fd5481980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57824/0 (socket says 172.21.15.117:57824) 2024-03-19T03:32:46.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.130+0000 7fd545400640 1 -- 172.21.15.117:0/1379114545 learned_addr learned my addr 172.21.15.117:0/1379114545 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:46.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.130+0000 7fd545e00640 1 --2- 172.21.15.117:0/1379114545 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd548101840 0x7fd548197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.132 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.130+0000 7fd546800640 1 --2- 172.21.15.117:0/1379114545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5481033f0 0x7fd54819f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.131+0000 7fd545400640 1 -- 172.21.15.117:0/1379114545 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd548101840 msgr2=0x7fd548197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:46.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.131+0000 7fd545400640 1 --2- 172.21.15.117:0/1379114545 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd548101840 0x7fd548197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.131+0000 7fd545400640 1 -- 172.21.15.117:0/1379114545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5481033f0 msgr2=0x7fd54819f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:46.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.131+0000 7fd545400640 1 --2- 172.21.15.117:0/1379114545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5481033f0 0x7fd54819f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.131+0000 7fd545400640 1 -- 172.21.15.117:0/1379114545 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd54819f780 con 0x7fd548102a30 2024-03-19T03:32:46.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.131+0000 7fd545e00640 1 --2- 172.21.15.117:0/1379114545 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd548101840 0x7fd548197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:32:46.133 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.131+0000 7fd545400640 1 --2- 172.21.15.117:0/1379114545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd548102a30 0x7fd5481980a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd534005ab0 tx=0x7fd53402c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:46.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.131+0000 7fd53ea00640 1 -- 172.21.15.117:0/1379114545 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd534043020 con 0x7fd548102a30 2024-03-19T03:32:46.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.132+0000 7fd53ea00640 1 -- 172.21.15.117:0/1379114545 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5340054e0 con 0x7fd548102a30 2024-03-19T03:32:46.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.132+0000 7fd54ca00640 1 -- 172.21.15.117:0/1379114545 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd54819fa10 con 0x7fd548102a30 2024-03-19T03:32:46.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.132+0000 7fd53ea00640 1 -- 172.21.15.117:0/1379114545 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd534003ce0 con 0x7fd548102a30 2024-03-19T03:32:46.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.132+0000 7fd54ca00640 1 -- 172.21.15.117:0/1379114545 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd54819ffb0 con 0x7fd548102a30 2024-03-19T03:32:46.134 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.133+0000 7fd54ca00640 1 -- 172.21.15.117:0/1379114545 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd548108b60 con 0x7fd548102a30 2024-03-19T03:32:46.136 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.135+0000 7fd53ea00640 1 -- 172.21.15.117:0/1379114545 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fd534046020 con 0x7fd548102a30 2024-03-19T03:32:46.137 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.137+0000 7fd53ea00640 1 --2- 172.21.15.117:0/1379114545 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd52c0a2880 0x7fd52c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.138 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.137+0000 7fd53ea00640 1 -- 172.21.15.117:0/1379114545 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fd5340f1450 con 0x7fd548102a30 2024-03-19T03:32:46.138 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.137+0000 7fd545e00640 1 --2- 172.21.15.117:0/1379114545 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd52c0a2880 0x7fd52c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.138 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.138+0000 7fd545e00640 1 --2- 172.21.15.117:0/1379114545 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd52c0a2880 0x7fd52c0a4d40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd548102890 tx=0x7fd538005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:46.139 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.139+0000 7fd53ea00640 1 -- 172.21.15.117:0/1379114545 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd53404d020 con 0x7fd548102a30 2024-03-19T03:32:46.381 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.380+0000 7fd54ca00640 1 -- 172.21.15.117:0/1379114545 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd54802d440 con 0x7fd548102a30 2024-03-19T03:32:46.382 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.382+0000 7fd53ea00640 1 -- 172.21.15.117:0/1379114545 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11953 (secure 0 0 0) 0x7fd5340be0a0 con 0x7fd548102a30 2024-03-19T03:32:46.383 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:32:46.383 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/54638259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":54638259},{"type":"v1","addr":"172.21.15.117:6835","nonce":54638259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3608185039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3608185039},{"type":"v1","addr":"172.21.15.117:6837","nonce":3608185039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T03:30:35.081740+0000","modified":"2024-03-19T03:31:09.995133+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24599,"mds_1":24581,"mds_2":24569,"mds_3":24545,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":20,"addr":"172.21.15.119:6833/1291833214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1291833214},{"type":"v1","addr":"172.21.15.119:6833","nonce":1291833214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6835/1579860804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1579860804},{"type":"v1","addr":"172.21.15.178:6835","nonce":1579860804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6835/1686938941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1686938941},{"type":"v1","addr":"172.21.15.119:6835","nonce":1686938941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6837/1770913538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":1770913538},{"type":"v1","addr":"172.21.15.178:6837","nonce":1770913538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6839/2655593383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2655593383},{"type":"v1","addr":"172.21.15.178:6839","nonce":2655593383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.119:6837/2054992738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2054992738},{"type":"v1","addr":"172.21.15.119:6837","nonce":2054992738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"g","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.117:6839/684335078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":684335078},{"type":"v1","addr":"172.21.15.117:6839","nonce":684335078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.117:6841/1329933892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6840","nonce":1329933892},{"type":"v1","addr":"172.21.15.117:6841","nonce":1329933892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/2383354833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2383354833},{"type":"v1","addr":"172.21.15.119:6839","nonce":2383354833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24581,24599,24569,24545,24467]},"id":1}]} 2024-03-19T03:32:46.383 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 28 2024-03-19T03:32:46.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.384+0000 7fd54ca00640 1 -- 172.21.15.117:0/1379114545 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd52c0a2880 msgr2=0x7fd52c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:46.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.384+0000 7fd54ca00640 1 --2- 172.21.15.117:0/1379114545 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd52c0a2880 0x7fd52c0a4d40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd548102890 tx=0x7fd538005f90 comp rx=0 tx=0).stop 2024-03-19T03:32:46.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.384+0000 7fd54ca00640 1 -- 172.21.15.117:0/1379114545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd548102a30 msgr2=0x7fd5481980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:46.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.384+0000 7fd54ca00640 1 --2- 172.21.15.117:0/1379114545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd548102a30 0x7fd5481980a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd534005ab0 tx=0x7fd53402c6c0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.385+0000 7fd54ca00640 1 -- 172.21.15.117:0/1379114545 shutdown_connections 2024-03-19T03:32:46.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.385+0000 7fd54ca00640 1 --2- 172.21.15.117:0/1379114545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5481033f0 0x7fd54819f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.385+0000 7fd54ca00640 1 --2- 172.21.15.117:0/1379114545 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fd52c0a2880 0x7fd52c0a4d40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.385+0000 7fd54ca00640 1 --2- 172.21.15.117:0/1379114545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd548102a30 0x7fd5481980a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.385+0000 7fd54ca00640 1 --2- 172.21.15.117:0/1379114545 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd548101840 0x7fd548197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.386 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.385+0000 7fd54ca00640 1 -- 172.21.15.117:0/1379114545 >> 172.21.15.117:0/1379114545 conn(0x7fd5480fcdd0 msgr2=0x7fd548105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:46.387 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.385+0000 7fd54ca00640 1 -- 172.21.15.117:0/1379114545 shutdown_connections 2024-03-19T03:32:46.387 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.385+0000 7fd54ca00640 1 -- 172.21.15.117:0/1379114545 wait complete. 2024-03-19T03:32:46.410 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-19T03:30:35.081740+0000', 'modified': '2024-03-19T03:31:09.995133+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24599, 'mds_1': 24581, 'mds_2': 24569, 'mds_3': 24545, 'mds_4': 24467}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24442': {'gid': 24442, 'name': 'l', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.178:6833/3751720259', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6832', 'nonce': 3751720259}, {'type': 'v1', 'addr': '172.21.15.178:6833', 'nonce': 3751720259}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24467': {'gid': 24467, 'name': 'k', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.119:6833/1291833214', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6832', 'nonce': 1291833214}, {'type': 'v1', 'addr': '172.21.15.119:6833', 'nonce': 1291833214}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24508': {'gid': 24508, 'name': 'f', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.178:6835/1579860804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6834', 'nonce': 1579860804}, {'type': 'v1', 'addr': '172.21.15.178:6835', 'nonce': 1579860804}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24515': {'gid': 24515, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.119:6835/1686938941', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 1686938941}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 1686938941}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24520': {'gid': 24520, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.178:6837/1770913538', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6836', 'nonce': 1770913538}, {'type': 'v1', 'addr': '172.21.15.178:6837', 'nonce': 1770913538}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24541': {'gid': 24541, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.178:6839/2655593383', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6838', 'nonce': 2655593383}, {'type': 'v1', 'addr': '172.21.15.178:6839', 'nonce': 2655593383}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24545': {'gid': 24545, 'name': 'h', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.119:6837/2054992738', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6836', 'nonce': 2054992738}, {'type': 'v1', 'addr': '172.21.15.119:6837', 'nonce': 2054992738}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24569': {'gid': 24569, 'name': 'g', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.117:6839/684335078', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6838', 'nonce': 684335078}, {'type': 'v1', 'addr': '172.21.15.117:6839', 'nonce': 684335078}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24581': {'gid': 24581, 'name': 'a', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.117:6841/1329933892', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6840', 'nonce': 1329933892}, {'type': 'v1', 'addr': '172.21.15.117:6841', 'nonce': 1329933892}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24599': {'gid': 24599, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.119:6839/2383354833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6838', 'nonce': 2383354833}, {'type': 'v1', 'addr': '172.21.15.119:6839', 'nonce': 2383354833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24599, 'qdb_cluster': [24581, 24599, 24569, 24545, 24467]} 2024-03-19T03:32:46.410 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-19T03:32:46.411 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.k status 2024-03-19T03:32:46.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.602+0000 7f4962800640 1 -- 172.21.15.117:0/1948067306 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f495c109970 msgr2=0x7f495c110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:46.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.602+0000 7f4962800640 1 --2- 172.21.15.117:0/1948067306 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f495c109970 0x7f495c110290 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4954009ca0 tx=0x7f495402c100 comp rx=0 tx=0).stop 2024-03-19T03:32:46.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.602+0000 7f4962800640 1 -- 172.21.15.117:0/1948067306 shutdown_connections 2024-03-19T03:32:46.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.602+0000 7f4962800640 1 --2- 172.21.15.117:0/1948067306 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f495c109970 0x7f495c110290 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.602+0000 7f4962800640 1 --2- 172.21.15.117:0/1948067306 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f495c108fb0 0x7f495c109430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.602+0000 7f4962800640 1 --2- 172.21.15.117:0/1948067306 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f495c107e60 0x7f495c108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.602+0000 7f4962800640 1 -- 172.21.15.117:0/1948067306 >> 172.21.15.117:0/1948067306 conn(0x7f495c076df0 msgr2=0x7f495c079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:46.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.602+0000 7f4962800640 1 -- 172.21.15.117:0/1948067306 shutdown_connections 2024-03-19T03:32:46.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.602+0000 7f4962800640 1 -- 172.21.15.117:0/1948067306 wait complete. 2024-03-19T03:32:46.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.603+0000 7f4962800640 1 Processor -- start 2024-03-19T03:32:46.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.603+0000 7f4962800640 1 -- start start 2024-03-19T03:32:46.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.604+0000 7f4962800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f495c107e60 0x7f495c19e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.604+0000 7f4962800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f495c108fb0 0x7f495c19e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.604+0000 7f4962800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f495c109970 0x7f495c1a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.604+0000 7f4962800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f495c113b00 con 0x7f495c108fb0 2024-03-19T03:32:46.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.604+0000 7f4962800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f495c113980 con 0x7f495c109970 2024-03-19T03:32:46.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.604+0000 7f4962800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f495c113c80 con 0x7f495c107e60 2024-03-19T03:32:46.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.604+0000 7f495aa00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f495c108fb0 0x7f495c19e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.604+0000 7f495be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f495c109970 0x7f495c1a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.604+0000 7f495be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f495c109970 0x7f495c1a5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57838/0 (socket says 172.21.15.117:57838) 2024-03-19T03:32:46.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.604+0000 7f495be00640 1 -- 172.21.15.117:0/2353817832 learned_addr learned my addr 172.21.15.117:0/2353817832 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:46.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.604+0000 7f495b400640 1 --2- 172.21.15.117:0/2353817832 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f495c107e60 0x7f495c19e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.604+0000 7f495be00640 1 -- 172.21.15.117:0/2353817832 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f495c107e60 msgr2=0x7f495c19e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:46.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.604+0000 7f495be00640 1 --2- 172.21.15.117:0/2353817832 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f495c107e60 0x7f495c19e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.604+0000 7f495be00640 1 -- 172.21.15.117:0/2353817832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f495c108fb0 msgr2=0x7f495c19e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:46.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.604+0000 7f495be00640 1 --2- 172.21.15.117:0/2353817832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f495c108fb0 0x7f495c19e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.604+0000 7f495be00640 1 -- 172.21.15.117:0/2353817832 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f495c1a5cd0 con 0x7f495c109970 2024-03-19T03:32:46.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.605+0000 7f495be00640 1 --2- 172.21.15.117:0/2353817832 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f495c109970 0x7f495c1a5660 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f495402c640 tx=0x7f4954035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:46.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.605+0000 7f4953e00640 1 -- 172.21.15.117:0/2353817832 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4954039040 con 0x7f495c109970 2024-03-19T03:32:46.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.605+0000 7f4953e00640 1 -- 172.21.15.117:0/2353817832 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4954044070 con 0x7f495c109970 2024-03-19T03:32:46.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.605+0000 7f4962800640 1 -- 172.21.15.117:0/2353817832 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f495c1a5f60 con 0x7f495c109970 2024-03-19T03:32:46.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.605+0000 7f4953e00640 1 -- 172.21.15.117:0/2353817832 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49540044a0 con 0x7f495c109970 2024-03-19T03:32:46.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.605+0000 7f4962800640 1 -- 172.21.15.117:0/2353817832 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f495c1a63c0 con 0x7f495c109970 2024-03-19T03:32:46.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.609+0000 7f49655b3740 1 Processor -- start 2024-03-19T03:32:46.612 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.609+0000 7f4953e00640 1 -- 172.21.15.117:0/2353817832 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f4954004640 con 0x7f495c109970 2024-03-19T03:32:46.612 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.610+0000 7f49655b3740 1 -- start start 2024-03-19T03:32:46.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.610+0000 7f49655b3740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fcb9d233f0 0x55fcb9d23810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.610+0000 7f49655b3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcb9d23de0 0x55fcb9d288e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.610+0000 7f49655b3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d28f50 0x55fcb9d2b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.610+0000 7f49655b3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fcb9c87d00 con 0x55fcb9d23de0 2024-03-19T03:32:46.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.610+0000 7f49655b3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55fcb9c30b00 con 0x55fcb9d28f50 2024-03-19T03:32:46.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.610+0000 7f49655b3740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x55fcb9c83da0 con 0x55fcb9d233f0 2024-03-19T03:32:46.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.610+0000 7f495aa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d28f50 0x55fcb9d2b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.610+0000 7f495aa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d28f50 0x55fcb9d2b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57848/0 (socket says 172.21.15.117:57848) 2024-03-19T03:32:46.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.610+0000 7f495aa00640 1 -- 172.21.15.117:0/2395797325 learned_addr learned my addr 172.21.15.117:0/2395797325 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:46.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.610+0000 7f495aa00640 1 --2- 172.21.15.117:0/2395797325 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fcb9d233f0 0x55fcb9d23810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.610+0000 7f495b400640 1 --2- 172.21.15.117:0/2395797325 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcb9d23de0 0x55fcb9d288e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.611+0000 7f495aa00640 1 -- 172.21.15.117:0/2395797325 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fcb9d233f0 msgr2=0x55fcb9d23810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:46.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.611+0000 7f495aa00640 1 --2- 172.21.15.117:0/2395797325 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fcb9d233f0 0x55fcb9d23810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.611+0000 7f495aa00640 1 -- 172.21.15.117:0/2395797325 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcb9d23de0 msgr2=0x55fcb9d288e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:46.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.611+0000 7f495aa00640 1 --2- 172.21.15.117:0/2395797325 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcb9d23de0 0x55fcb9d288e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.611+0000 7f495aa00640 1 -- 172.21.15.117:0/2395797325 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcb999c2d0 con 0x55fcb9d28f50 2024-03-19T03:32:46.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.611+0000 7f495aa00640 1 --2- 172.21.15.117:0/2395797325 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d28f50 0x55fcb9d2b3e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x55fcb9d24640 tx=0x7f49480046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7655b4cf104df276 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:46.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.611+0000 7f4950c00640 1 -- 172.21.15.117:0/2395797325 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4948012cc0 con 0x55fcb9d28f50 2024-03-19T03:32:46.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.611+0000 7f4950c00640 1 -- 172.21.15.117:0/2395797325 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4948010770 con 0x55fcb9d28f50 2024-03-19T03:32:46.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.611+0000 7f4950c00640 1 -- 172.21.15.117:0/2395797325 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4948012e60 con 0x55fcb9d28f50 2024-03-19T03:32:46.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.611+0000 7f4953e00640 1 --2- 172.21.15.117:0/2353817832 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f49400a2800 0x7f49400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.612+0000 7f4953e00640 1 -- 172.21.15.117:0/2353817832 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f49540f1580 con 0x7f495c109970 2024-03-19T03:32:46.615 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.612+0000 7f495b400640 1 --2- 172.21.15.117:0/2353817832 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f49400a2800 0x7f49400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.615 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.612+0000 7f49655b3740 1 -- 172.21.15.117:0/2395797325 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d28f50 msgr2=0x55fcb9d2b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:46.615 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.612+0000 7f49655b3740 1 --2- 172.21.15.117:0/2395797325 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d28f50 0x55fcb9d2b3e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x55fcb9d24640 tx=0x7f49480046c0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.612+0000 7f49655b3740 1 -- 172.21.15.117:0/2395797325 shutdown_connections 2024-03-19T03:32:46.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.612+0000 7f49655b3740 1 --2- 172.21.15.117:0/2395797325 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d28f50 0x55fcb9d2b3e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.612+0000 7f49655b3740 1 --2- 172.21.15.117:0/2395797325 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcb9d23de0 0x55fcb9d288e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.612+0000 7f49655b3740 1 --2- 172.21.15.117:0/2395797325 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fcb9d233f0 0x55fcb9d23810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.612+0000 7f49655b3740 1 -- 172.21.15.117:0/2395797325 >> 172.21.15.117:0/2395797325 conn(0x55fcb9d1e960 msgr2=0x55fcb9d1ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:46.620 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.612+0000 7f49655b3740 1 -- 172.21.15.117:0/2395797325 shutdown_connections 2024-03-19T03:32:46.621 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.612+0000 7f49655b3740 1 -- 172.21.15.117:0/2395797325 wait complete. 2024-03-19T03:32:46.621 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.612+0000 7f495b400640 1 --2- 172.21.15.117:0/2353817832 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f49400a2800 0x7f49400a4cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x55fcb9d1f8d0 tx=0x7f494c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:46.621 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.613+0000 7f49655b3740 1 -- start start 2024-03-19T03:32:46.621 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.613+0000 7f49655b3740 1 Processor -- start 2024-03-19T03:32:46.621 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.614+0000 7f49655b3740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fcb9d233f0 0x55fcb9d2d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.621 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.614+0000 7f49655b3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d23de0 0x55fcb9d2de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.621 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.614+0000 7f49655b3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcb9d28f50 0x55fcb9d34ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.621 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.614+0000 7f49655b3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fcb9d2bf00 con 0x55fcb9d28f50 2024-03-19T03:32:46.621 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.614+0000 7f49655b3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55fcb9c43ed0 con 0x55fcb9d23de0 2024-03-19T03:32:46.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.614+0000 7f49655b3740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x55fcb9d2c080 con 0x55fcb9d233f0 2024-03-19T03:32:46.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.614+0000 7f49655b3740 10 client.0 _ll_register_callbacks cb 0x55fcb9c41c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:32:46.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.614+0000 7f495b400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fcb9d233f0 0x55fcb9d2d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.614+0000 7f495aa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d23de0 0x55fcb9d2de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.614+0000 7f495b400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fcb9d233f0 0x55fcb9d2d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:38894/0 (socket says 172.21.15.117:38894) 2024-03-19T03:32:46.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.614+0000 7f495b400640 1 -- 172.21.15.117:0/2918355525 learned_addr learned my addr 172.21.15.117:0/2918355525 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:46.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.614+0000 7f495be00640 1 --2- 172.21.15.117:0/2918355525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcb9d28f50 0x55fcb9d34ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.614+0000 7f495aa00640 1 -- 172.21.15.117:0/2918355525 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fcb9d233f0 msgr2=0x55fcb9d2d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:46.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.614+0000 7f495aa00640 1 --2- 172.21.15.117:0/2918355525 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fcb9d233f0 0x55fcb9d2d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.614+0000 7f495aa00640 1 -- 172.21.15.117:0/2918355525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcb9d28f50 msgr2=0x55fcb9d34ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:46.622 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.614+0000 7f495aa00640 1 --2- 172.21.15.117:0/2918355525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcb9d28f50 0x55fcb9d34ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.614+0000 7f495aa00640 1 -- 172.21.15.117:0/2918355525 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcb9a666b0 con 0x55fcb9d23de0 2024-03-19T03:32:46.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.614+0000 7f495be00640 1 --2- 172.21.15.117:0/2918355525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcb9d28f50 0x55fcb9d34ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T03:32:46.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.615+0000 7f495aa00640 1 --2- 172.21.15.117:0/2918355525 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d23de0 0x55fcb9d2de80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4948004650 tx=0x7f4948010c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afc1c07bf383d951 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:46.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.615+0000 7f4945600640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-19T03:32:46.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.615+0000 7f4945600640 1 -- 172.21.15.117:0/2918355525 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4948005450 con 0x55fcb9d23de0 2024-03-19T03:32:46.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.615+0000 7f4945600640 1 -- 172.21.15.117:0/2918355525 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f49480055f0 con 0x55fcb9d23de0 2024-03-19T03:32:46.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.615+0000 7f49655b3740 1 -- 172.21.15.117:0/2918355525 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4948010770 con 0x55fcb9d23de0 2024-03-19T03:32:46.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.615+0000 7f4945600640 1 -- 172.21.15.117:0/2918355525 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4948004730 con 0x55fcb9d23de0 2024-03-19T03:32:46.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.615+0000 7f49655b3740 10 client.25169 fetch_fsmap learned FSMap version 28 2024-03-19T03:32:46.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.615+0000 7f49655b3740 1 -- 172.21.15.117:0/2918355525 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fcb9d38230 con 0x55fcb9d23de0 2024-03-19T03:32:46.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.616+0000 7f4945600640 1 -- 172.21.15.117:0/2918355525 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4948010770 con 0x55fcb9d23de0 2024-03-19T03:32:46.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.616+0000 7f4945600640 1 -- 172.21.15.117:0/2918355525 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f494801faa0 con 0x55fcb9d23de0 2024-03-19T03:32:46.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.616+0000 7f49655b3740 10 client.25169 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T03:32:46.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.616+0000 7f49655b3740 10 client.25169 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-19T03:32:46.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.616+0000 7f49655b3740 20 client.25169 populate_metadata read hostname 'smithi117' 2024-03-19T03:32:46.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.616+0000 7f49655b3740 1 --2- 172.21.15.117:0/2918355525 >> [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] conn(0x55fcb9d3a5c0 0x55fcb9d3ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.616+0000 7f49655b3740 4 client.25169 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:32:46.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.616+0000 7f49655b3740 1 -- 172.21.15.117:0/2918355525 --> [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcb9c95240 con 0x55fcb9d3a5c0 2024-03-19T03:32:46.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.616+0000 7f495b400640 1 --2- 172.21.15.117:0/2918355525 >> [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] conn(0x55fcb9d3a5c0 0x55fcb9d3ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.617+0000 7f495b400640 1 --2- 172.21.15.117:0/2918355525 >> [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] conn(0x55fcb9d3a5c0 0x55fcb9d3ba30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=81a512d8f25e8c9b server_cookie=c01100ca3854ede4 in_seq=0 out_seq=0 2024-03-19T03:32:46.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.617+0000 7f4945600640 10 client.25169 ms_handle_connect on v2:172.21.15.119:6832/1291833214 2024-03-19T03:32:46.624 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.618+0000 7f4945600640 1 -- 172.21.15.117:0/2918355525 <== mds.4 v2:172.21.15.119:6832/1291833214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fcb9c95240 con 0x55fcb9d3a5c0 2024-03-19T03:32:46.625 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.618+0000 7f4945600640 10 client.25169 handle_command_reply: tid=0 2024-03-19T03:32:46.625 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.618+0000 7f49655b3740 1 client.25169 shutdown 2024-03-19T03:32:46.625 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.618+0000 7f49655b3740 1 -- 172.21.15.117:0/2918355525 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d23de0 msgr2=0x55fcb9d2de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:46.625 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.618+0000 7f49655b3740 1 --2- 172.21.15.117:0/2918355525 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d23de0 0x55fcb9d2de80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4948004650 tx=0x7f4948010c90 comp rx=0 tx=0).stop 2024-03-19T03:32:46.625 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.618+0000 7f49655b3740 1 -- 172.21.15.117:0/2918355525 shutdown_connections 2024-03-19T03:32:46.625 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.618+0000 7f49655b3740 1 --2- 172.21.15.117:0/2918355525 >> [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] conn(0x55fcb9d3a5c0 0x55fcb9d3ba30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.625 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.618+0000 7f49655b3740 1 --2- 172.21.15.117:0/2918355525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcb9d28f50 0x55fcb9d34ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.625 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.618+0000 7f49655b3740 1 --2- 172.21.15.117:0/2918355525 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d23de0 0x55fcb9d2de80 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.625 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.618+0000 7f49655b3740 1 --2- 172.21.15.117:0/2918355525 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fcb9d233f0 0x55fcb9d2d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.625 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.618+0000 7f49655b3740 1 -- 172.21.15.117:0/2918355525 >> 172.21.15.117:0/2918355525 conn(0x55fcb9d1e960 msgr2=0x55fcb9d1daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:46.625 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.618+0000 7f4945600640 0 client.25169 ms_handle_reset on v2:172.21.15.119:6832/1291833214 2024-03-19T03:32:46.625 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.619+0000 7f49655b3740 1 -- 172.21.15.117:0/2918355525 shutdown_connections 2024-03-19T03:32:46.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.619+0000 7f49655b3740 1 -- 172.21.15.117:0/2918355525 wait complete. 2024-03-19T03:32:46.626 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.619+0000 7f49655b3740 20 client.25169 trim_cache size 0 max 16384 2024-03-19T03:32:46.657 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.655+0000 7f49655b3740 1 Processor -- start 2024-03-19T03:32:46.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.655+0000 7f49655b3740 1 -- start start 2024-03-19T03:32:46.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.655+0000 7f49655b3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcb9d233f0 0x55fcb9d2c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.655+0000 7f49655b3740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fcb9d23de0 0x55fcb9d34e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.655+0000 7f49655b3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d28f50 0x55fcb9d1ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.656+0000 7f49655b3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fcb9d39d70 con 0x55fcb9d233f0 2024-03-19T03:32:46.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.656+0000 7f49655b3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55fcb9d3a440 con 0x55fcb9d28f50 2024-03-19T03:32:46.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.656+0000 7f49655b3740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x55fcb9d3ab10 con 0x55fcb9d23de0 2024-03-19T03:32:46.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.656+0000 7f495b400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcb9d233f0 0x55fcb9d2c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.656+0000 7f495be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d28f50 0x55fcb9d1ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.656+0000 7f495b400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcb9d233f0 0x55fcb9d2c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:55604/0 (socket says 172.21.15.117:55604) 2024-03-19T03:32:46.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.656+0000 7f495be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d28f50 0x55fcb9d1ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57862/0 (socket says 172.21.15.117:57862) 2024-03-19T03:32:46.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.656+0000 7f495b400640 1 -- 172.21.15.117:0/1031789400 learned_addr learned my addr 172.21.15.117:0/1031789400 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:46.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.656+0000 7f495aa00640 1 --2- 172.21.15.117:0/1031789400 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fcb9d23de0 0x55fcb9d34e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.656+0000 7f495be00640 1 -- 172.21.15.117:0/1031789400 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fcb9d23de0 msgr2=0x55fcb9d34e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:46.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.656+0000 7f495be00640 1 --2- 172.21.15.117:0/1031789400 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fcb9d23de0 0x55fcb9d34e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.656+0000 7f495be00640 1 -- 172.21.15.117:0/1031789400 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcb9d233f0 msgr2=0x55fcb9d2c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:46.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.656+0000 7f495be00640 1 --2- 172.21.15.117:0/1031789400 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcb9d233f0 0x55fcb9d2c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.656+0000 7f495be00640 1 -- 172.21.15.117:0/1031789400 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcb9d241c0 con 0x55fcb9d28f50 2024-03-19T03:32:46.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.657+0000 7f495be00640 1 --2- 172.21.15.117:0/1031789400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d28f50 0x55fcb9d1ed40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f49540eec30 tx=0x7f49540b9df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a16e6689a864b306 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:46.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.657+0000 7f4947e00640 1 -- 172.21.15.117:0/1031789400 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49540be1d0 con 0x55fcb9d28f50 2024-03-19T03:32:46.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.657+0000 7f4947e00640 1 -- 172.21.15.117:0/1031789400 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49540bdad0 con 0x55fcb9d28f50 2024-03-19T03:32:46.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.657+0000 7f4947e00640 1 -- 172.21.15.117:0/1031789400 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49540ba140 con 0x55fcb9d28f50 2024-03-19T03:32:46.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.657+0000 7f49655b3740 1 -- 172.21.15.117:0/1031789400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d28f50 msgr2=0x55fcb9d1ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:46.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.657+0000 7f49655b3740 1 --2- 172.21.15.117:0/1031789400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d28f50 0x55fcb9d1ed40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f49540eec30 tx=0x7f49540b9df0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.657+0000 7f49655b3740 1 -- 172.21.15.117:0/1031789400 shutdown_connections 2024-03-19T03:32:46.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.657+0000 7f49655b3740 1 --2- 172.21.15.117:0/1031789400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d28f50 0x55fcb9d1ed40 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.657+0000 7f49655b3740 1 --2- 172.21.15.117:0/1031789400 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fcb9d23de0 0x55fcb9d34e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.657+0000 7f49655b3740 1 --2- 172.21.15.117:0/1031789400 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcb9d233f0 0x55fcb9d2c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.657+0000 7f49655b3740 1 -- 172.21.15.117:0/1031789400 >> 172.21.15.117:0/1031789400 conn(0x55fcb9d1e960 msgr2=0x55fcb9d395e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:46.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.658+0000 7f49655b3740 1 -- 172.21.15.117:0/1031789400 shutdown_connections 2024-03-19T03:32:46.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.658+0000 7f49655b3740 1 -- 172.21.15.117:0/1031789400 wait complete. 2024-03-19T03:32:46.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.658+0000 7f49655b3740 1 -- start start 2024-03-19T03:32:46.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.658+0000 7f49655b3740 1 Processor -- start 2024-03-19T03:32:46.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.658+0000 7f49655b3740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fcb9d233f0 0x55fcb9d32e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.658+0000 7f49655b3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcb9d23de0 0x55fcb9d9c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.659+0000 7f49655b3740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d28f50 0x55fcb9d9e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.659+0000 7f49655b3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fcb9d3b8b0 con 0x55fcb9d23de0 2024-03-19T03:32:46.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.659+0000 7f49655b3740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55fcb9d3b1e0 con 0x55fcb9d28f50 2024-03-19T03:32:46.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.659+0000 7f49655b3740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x55fcb9d3bf80 con 0x55fcb9d233f0 2024-03-19T03:32:46.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.659+0000 7f49655b3740 10 client.0 _ll_register_callbacks cb 0x7f494800be60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:32:46.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.659+0000 7f495be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d28f50 0x55fcb9d9e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.665 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.659+0000 7f495b400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fcb9d233f0 0x55fcb9d32e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.659+0000 7f495aa00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcb9d23de0 0x55fcb9d9c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.659+0000 7f495be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d28f50 0x55fcb9d9e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57870/0 (socket says 172.21.15.117:57870) 2024-03-19T03:32:46.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.659+0000 7f495be00640 1 -- 172.21.15.117:0/1102108748 learned_addr learned my addr 172.21.15.117:0/1102108748 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:46.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.659+0000 7f495be00640 1 -- 172.21.15.117:0/1102108748 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fcb9d233f0 msgr2=0x55fcb9d32e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:46.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.659+0000 7f495be00640 1 --2- 172.21.15.117:0/1102108748 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fcb9d233f0 0x55fcb9d32e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.659+0000 7f495be00640 1 -- 172.21.15.117:0/1102108748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcb9d23de0 msgr2=0x55fcb9d9c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:46.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.659+0000 7f495be00640 1 --2- 172.21.15.117:0/1102108748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcb9d23de0 0x55fcb9d9c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.659+0000 7f495be00640 1 -- 172.21.15.117:0/1102108748 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcb9d9eda0 con 0x55fcb9d28f50 2024-03-19T03:32:46.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.659+0000 7f495be00640 1 --2- 172.21.15.117:0/1102108748 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d28f50 0x55fcb9d9e860 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4954006010 tx=0x7f49540c1c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e63e690cfdcfc311 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:46.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.659+0000 7f4945600640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-19T03:32:46.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.660+0000 7f4945600640 1 -- 172.21.15.117:0/1102108748 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49540db060 con 0x55fcb9d28f50 2024-03-19T03:32:46.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.660+0000 7f4945600640 1 -- 172.21.15.117:0/1102108748 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f49540bdd30 con 0x55fcb9d28f50 2024-03-19T03:32:46.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.660+0000 7f49655b3740 1 -- 172.21.15.117:0/1102108748 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49540bdad0 con 0x55fcb9d28f50 2024-03-19T03:32:46.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.660+0000 7f4945600640 1 -- 172.21.15.117:0/1102108748 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f49540f1370 con 0x55fcb9d28f50 2024-03-19T03:32:46.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.660+0000 7f49655b3740 10 client.25175 fetch_fsmap learned FSMap version 28 2024-03-19T03:32:46.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.660+0000 7f49655b3740 1 -- 172.21.15.117:0/1102108748 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fcb9d9ffd0 con 0x55fcb9d28f50 2024-03-19T03:32:46.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.660+0000 7f4945600640 1 -- 172.21.15.117:0/1102108748 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49540bdad0 con 0x55fcb9d28f50 2024-03-19T03:32:46.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.660+0000 7f4945600640 1 -- 172.21.15.117:0/1102108748 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f495403c030 con 0x55fcb9d28f50 2024-03-19T03:32:46.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.660+0000 7f49655b3740 10 client.25175 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T03:32:46.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.660+0000 7f49655b3740 10 client.25175 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-19T03:32:46.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.660+0000 7f49655b3740 20 client.25175 populate_metadata read hostname 'smithi117' 2024-03-19T03:32:46.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.660+0000 7f49655b3740 1 --2- 172.21.15.117:0/1102108748 >> [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] conn(0x55fcb9da49c0 0x55fcb9da36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.661+0000 7f49655b3740 4 client.25175 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T03:32:46.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.661+0000 7f49655b3740 1 -- 172.21.15.117:0/1102108748 --> [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] -- command(tid 0: {"prefix": "status"}) -- 0x55fcb9c8f590 con 0x55fcb9da49c0 2024-03-19T03:32:46.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.661+0000 7f495b400640 1 --2- 172.21.15.117:0/1102108748 >> [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] conn(0x55fcb9da49c0 0x55fcb9da36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.661+0000 7f495b400640 1 --2- 172.21.15.117:0/1102108748 >> [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] conn(0x55fcb9da49c0 0x55fcb9da36d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=638f8b7dbbe13925 server_cookie=800e1fbabe53ea4e in_seq=0 out_seq=0 2024-03-19T03:32:46.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.661+0000 7f4945600640 10 client.25175 ms_handle_connect on v2:172.21.15.119:6832/1291833214 2024-03-19T03:32:46.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.661+0000 7f4945600640 1 -- 172.21.15.117:0/1102108748 <== mds.4 v2:172.21.15.119:6832/1291833214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55fcb9c8f590 con 0x55fcb9da49c0 2024-03-19T03:32:46.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.661+0000 7f4945600640 10 client.25175 handle_command_reply: tid=0 2024-03-19T03:32:46.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.661+0000 7f49655b3740 1 client.25175 shutdown 2024-03-19T03:32:46.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.662+0000 7f49655b3740 1 -- 172.21.15.117:0/1102108748 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d28f50 msgr2=0x55fcb9d9e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:46.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.662+0000 7f49655b3740 1 --2- 172.21.15.117:0/1102108748 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d28f50 0x55fcb9d9e860 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4954006010 tx=0x7f49540c1c20 comp rx=0 tx=0).stop 2024-03-19T03:32:46.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.662+0000 7f49655b3740 1 -- 172.21.15.117:0/1102108748 shutdown_connections 2024-03-19T03:32:46.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.662+0000 7f49655b3740 1 --2- 172.21.15.117:0/1102108748 >> [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] conn(0x55fcb9da49c0 0x55fcb9da36d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.662+0000 7f49655b3740 1 --2- 172.21.15.117:0/1102108748 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55fcb9d28f50 0x55fcb9d9e860 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.662+0000 7f49655b3740 1 --2- 172.21.15.117:0/1102108748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fcb9d23de0 0x55fcb9d9c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.662+0000 7f49655b3740 1 --2- 172.21.15.117:0/1102108748 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fcb9d233f0 0x55fcb9d32e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.662+0000 7f49655b3740 1 -- 172.21.15.117:0/1102108748 >> 172.21.15.117:0/1102108748 conn(0x55fcb9d1e960 msgr2=0x55fcb9d28530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:46.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.662+0000 7f4945600640 0 client.25175 ms_handle_reset on v2:172.21.15.119:6832/1291833214 2024-03-19T03:32:46.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.662+0000 7f49655b3740 1 -- 172.21.15.117:0/1102108748 shutdown_connections 2024-03-19T03:32:46.668 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.662+0000 7f49655b3740 1 -- 172.21.15.117:0/1102108748 wait complete. 2024-03-19T03:32:46.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.662+0000 7f49655b3740 20 client.25175 trim_cache size 0 max 16384 2024-03-19T03:32:46.669 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-19T03:32:46.669 INFO:teuthology.orchestra.run.smithi117.stdout: "cluster_fsid": "a3a82456-e59f-11ee-95c9-87774f69a715", 2024-03-19T03:32:46.669 INFO:teuthology.orchestra.run.smithi117.stdout: "whoami": 4, 2024-03-19T03:32:46.669 INFO:teuthology.orchestra.run.smithi117.stdout: "id": 24467, 2024-03-19T03:32:46.669 INFO:teuthology.orchestra.run.smithi117.stdout: "want_state": "up:active", 2024-03-19T03:32:46.669 INFO:teuthology.orchestra.run.smithi117.stdout: "state": "up:active", 2024-03-19T03:32:46.669 INFO:teuthology.orchestra.run.smithi117.stdout: "fs_name": "cephfs", 2024-03-19T03:32:46.669 INFO:teuthology.orchestra.run.smithi117.stdout: "rank_uptime": 97.671416731999997, 2024-03-19T03:32:46.669 INFO:teuthology.orchestra.run.smithi117.stdout: "mdsmap_epoch": 28, 2024-03-19T03:32:46.670 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch": 93, 2024-03-19T03:32:46.670 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch_barrier": 89, 2024-03-19T03:32:46.670 INFO:teuthology.orchestra.run.smithi117.stdout: "uptime": 171.92349917000001 2024-03-19T03:32:46.670 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-19T03:32:46.670 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.663+0000 7f4962800640 1 -- 172.21.15.117:0/2353817832 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f49400a2800 msgr2=0x7f49400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:46.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.663+0000 7f4962800640 1 --2- 172.21.15.117:0/2353817832 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f49400a2800 0x7f49400a4cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x55fcb9d1f8d0 tx=0x7f494c00b390 comp rx=0 tx=0).stop 2024-03-19T03:32:46.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.663+0000 7f4962800640 1 -- 172.21.15.117:0/2353817832 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f495c109970 msgr2=0x7f495c1a5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:46.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.663+0000 7f4962800640 1 --2- 172.21.15.117:0/2353817832 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f495c109970 0x7f495c1a5660 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f495402c640 tx=0x7f4954035d10 comp rx=0 tx=0).stop 2024-03-19T03:32:46.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.663+0000 7f4962800640 1 -- 172.21.15.117:0/2353817832 shutdown_connections 2024-03-19T03:32:46.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.663+0000 7f4962800640 1 --2- 172.21.15.117:0/2353817832 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f49400a2800 0x7f49400a4cc0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.663+0000 7f4962800640 1 --2- 172.21.15.117:0/2353817832 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f495c109970 0x7f495c1a5660 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.663+0000 7f4962800640 1 --2- 172.21.15.117:0/2353817832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f495c108fb0 0x7f495c19e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.663+0000 7f4962800640 1 --2- 172.21.15.117:0/2353817832 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f495c107e60 0x7f495c19e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.663+0000 7f4962800640 1 -- 172.21.15.117:0/2353817832 >> 172.21.15.117:0/2353817832 conn(0x7f495c076df0 msgr2=0x7f495c10c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:46.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.663+0000 7f4962800640 1 -- 172.21.15.117:0/2353817832 shutdown_connections 2024-03-19T03:32:46.671 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.663+0000 7f4962800640 1 -- 172.21.15.117:0/2353817832 wait complete. 2024-03-19T03:32:46.679 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:46 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3993606018' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:32:46.680 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:46 smithi117 ceph-mon[31838]: pgmap v387: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 895 B/s wr, 12 op/s; 7.9 KiB/s, 1 keys/s, 2 objects/s recovering 2024-03-19T03:32:46.680 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:46 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1379114545' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:32:46.682 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-19T03:32:46.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:46 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3993606018' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:32:46.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:46 smithi178 ceph-mon[67578]: pgmap v387: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 895 B/s wr, 12 op/s; 7.9 KiB/s, 1 keys/s, 2 objects/s recovering 2024-03-19T03:32:46.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:46 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1379114545' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:32:46.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.883+0000 7f7646400640 1 -- 172.21.15.117:0/3201857444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76401075b0 msgr2=0x7f76401099a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:46.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.883+0000 7f7646400640 1 --2- 172.21.15.117:0/3201857444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76401075b0 0x7f76401099a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7638009ca0 tx=0x7f763802c100 comp rx=0 tx=0).stop 2024-03-19T03:32:46.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.883+0000 7f7646400640 1 -- 172.21.15.117:0/3201857444 shutdown_connections 2024-03-19T03:32:46.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.883+0000 7f7646400640 1 --2- 172.21.15.117:0/3201857444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76401075b0 0x7f76401099a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.883+0000 7f7646400640 1 --2- 172.21.15.117:0/3201857444 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7640104c80 0x7f7640107070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.884 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.883+0000 7f7646400640 1 --2- 172.21.15.117:0/3201857444 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7640102350 0x7f7640104740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.883+0000 7f7646400640 1 -- 172.21.15.117:0/3201857444 >> 172.21.15.117:0/3201857444 conn(0x7f76400fbcc0 msgr2=0x7f76400fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:46.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.883+0000 7f7646400640 1 -- 172.21.15.117:0/3201857444 shutdown_connections 2024-03-19T03:32:46.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.883+0000 7f7646400640 1 -- 172.21.15.117:0/3201857444 wait complete. 2024-03-19T03:32:46.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.884+0000 7f7646400640 1 Processor -- start 2024-03-19T03:32:46.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.884+0000 7f7646400640 1 -- start start 2024-03-19T03:32:46.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.884+0000 7f7646400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7640102350 0x7f7640199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.884+0000 7f7646400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7640104c80 0x7f764019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.884+0000 7f7646400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76401075b0 0x7f76401a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.884+0000 7f7646400640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f764010d4b0 con 0x7f76401075b0 2024-03-19T03:32:46.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.884+0000 7f7646400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f764010d330 con 0x7f7640102350 2024-03-19T03:32:46.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.884+0000 7f7646400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f764010d630 con 0x7f7640104c80 2024-03-19T03:32:46.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.884+0000 7f763f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7640102350 0x7f7640199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.885+0000 7f763f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7640102350 0x7f7640199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57890/0 (socket says 172.21.15.117:57890) 2024-03-19T03:32:46.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.885+0000 7f763f400640 1 -- 172.21.15.117:0/2854799473 learned_addr learned my addr 172.21.15.117:0/2854799473 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:46.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.885+0000 7f763fe00640 1 --2- 172.21.15.117:0/2854799473 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76401075b0 0x7f76401a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.885+0000 7f763ea00640 1 --2- 172.21.15.117:0/2854799473 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7640104c80 0x7f764019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.885+0000 7f763f400640 1 -- 172.21.15.117:0/2854799473 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7640104c80 msgr2=0x7f764019a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:46.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.885+0000 7f763f400640 1 --2- 172.21.15.117:0/2854799473 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7640104c80 0x7f764019a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.885+0000 7f763f400640 1 -- 172.21.15.117:0/2854799473 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76401075b0 msgr2=0x7f76401a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:46.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.885+0000 7f763f400640 1 --2- 172.21.15.117:0/2854799473 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76401075b0 0x7f76401a1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.885+0000 7f763f400640 1 -- 172.21.15.117:0/2854799473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76401a1910 con 0x7f7640102350 2024-03-19T03:32:46.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.885+0000 7f763f400640 1 --2- 172.21.15.117:0/2854799473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7640102350 0x7f7640199d00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f763000e9f0 tx=0x7f763000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:46.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.885+0000 7f7637e00640 1 -- 172.21.15.117:0/2854799473 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f763000cd80 con 0x7f7640102350 2024-03-19T03:32:46.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.885+0000 7f7637e00640 1 -- 172.21.15.117:0/2854799473 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7630010910 con 0x7f7640102350 2024-03-19T03:32:46.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.885+0000 7f7646400640 1 -- 172.21.15.117:0/2854799473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76401a1c00 con 0x7f7640102350 2024-03-19T03:32:46.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.886+0000 7f7637e00640 1 -- 172.21.15.117:0/2854799473 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7630018990 con 0x7f7640102350 2024-03-19T03:32:46.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.886+0000 7f7646400640 1 -- 172.21.15.117:0/2854799473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76401a21a0 con 0x7f7640102350 2024-03-19T03:32:46.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.890+0000 7f76491f9740 1 Processor -- start 2024-03-19T03:32:46.891 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.890+0000 7f76491f9740 1 -- start start 2024-03-19T03:32:46.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.890+0000 7f7637e00640 1 -- 172.21.15.117:0/2854799473 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f7630010ac0 con 0x7f7640102350 2024-03-19T03:32:46.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.890+0000 7f76491f9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555bbe0ca3f0 0x555bbe0ca810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.890+0000 7f76491f9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bbe0cade0 0x555bbe0cf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.890+0000 7f76491f9740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555bbe0cff50 0x555bbe0d23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.890+0000 7f76491f9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555bbe02ed00 con 0x555bbe0cade0 2024-03-19T03:32:46.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.890+0000 7f76491f9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x555bbdfd7b00 con 0x555bbe0ca3f0 2024-03-19T03:32:46.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.890+0000 7f76491f9740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x555bbe02ada0 con 0x555bbe0cff50 2024-03-19T03:32:46.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.891+0000 7f763fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555bbe0ca3f0 0x555bbe0ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.892 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.891+0000 7f763fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555bbe0ca3f0 0x555bbe0ca810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57906/0 (socket says 172.21.15.117:57906) 2024-03-19T03:32:46.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.891+0000 7f763fe00640 1 -- 172.21.15.117:0/2090610382 learned_addr learned my addr 172.21.15.117:0/2090610382 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:46.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.891+0000 7f763f400640 1 --2- 172.21.15.117:0/2090610382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bbe0cade0 0x555bbe0cf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.891+0000 7f763ea00640 1 --2- 172.21.15.117:0/2090610382 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555bbe0cff50 0x555bbe0d23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.891+0000 7f763fe00640 1 -- 172.21.15.117:0/2090610382 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555bbe0cff50 msgr2=0x555bbe0d23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:46.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.891+0000 7f763fe00640 1 --2- 172.21.15.117:0/2090610382 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555bbe0cff50 0x555bbe0d23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.891+0000 7f763fe00640 1 -- 172.21.15.117:0/2090610382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bbe0cade0 msgr2=0x555bbe0cf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:46.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.891+0000 7f763fe00640 1 --2- 172.21.15.117:0/2090610382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bbe0cade0 0x555bbe0cf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.893 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.891+0000 7f763fe00640 1 -- 172.21.15.117:0/2090610382 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bbdd432d0 con 0x555bbe0ca3f0 2024-03-19T03:32:46.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.891+0000 7f763fe00640 1 --2- 172.21.15.117:0/2090610382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555bbe0ca3f0 0x555bbe0ca810 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7638005e30 tx=0x7f76380323d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bef06b7838808b61 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:46.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.892+0000 7f7634c00640 1 -- 172.21.15.117:0/2090610382 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7638043020 con 0x555bbe0ca3f0 2024-03-19T03:32:46.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.892+0000 7f7634c00640 1 -- 172.21.15.117:0/2090610382 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7638032c90 con 0x555bbe0ca3f0 2024-03-19T03:32:46.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.892+0000 7f7634c00640 1 -- 172.21.15.117:0/2090610382 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7638039040 con 0x555bbe0ca3f0 2024-03-19T03:32:46.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.892+0000 7f76491f9740 1 -- 172.21.15.117:0/2090610382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555bbe0ca3f0 msgr2=0x555bbe0ca810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:46.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.892+0000 7f76491f9740 1 --2- 172.21.15.117:0/2090610382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555bbe0ca3f0 0x555bbe0ca810 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7638005e30 tx=0x7f76380323d0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.892+0000 7f76491f9740 1 -- 172.21.15.117:0/2090610382 shutdown_connections 2024-03-19T03:32:46.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.892+0000 7f76491f9740 1 --2- 172.21.15.117:0/2090610382 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555bbe0cff50 0x555bbe0d23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.892+0000 7f76491f9740 1 --2- 172.21.15.117:0/2090610382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bbe0cade0 0x555bbe0cf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.892+0000 7f76491f9740 1 --2- 172.21.15.117:0/2090610382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555bbe0ca3f0 0x555bbe0ca810 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.892+0000 7f76491f9740 1 -- 172.21.15.117:0/2090610382 >> 172.21.15.117:0/2090610382 conn(0x555bbe0c5960 msgr2=0x555bbe0c5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:46.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.893+0000 7f76491f9740 1 -- 172.21.15.117:0/2090610382 shutdown_connections 2024-03-19T03:32:46.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.893+0000 7f76491f9740 1 -- 172.21.15.117:0/2090610382 wait complete. 2024-03-19T03:32:46.897 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.893+0000 7f7637e00640 1 --2- 172.21.15.117:0/2854799473 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f76240a2790 0x7f76240a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.893+0000 7f7637e00640 1 -- 172.21.15.117:0/2854799473 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f76300cf2f0 con 0x7f7640102350 2024-03-19T03:32:46.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.893+0000 7f763ea00640 1 --2- 172.21.15.117:0/2854799473 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f76240a2790 0x7f76240a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.893+0000 7f76491f9740 1 -- start start 2024-03-19T03:32:46.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.893+0000 7f76491f9740 1 Processor -- start 2024-03-19T03:32:46.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.894+0000 7f763ea00640 1 --2- 172.21.15.117:0/2854799473 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f76240a2790 0x7f76240a4c50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x555bbe0c68d0 tx=0x7f762c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:46.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.894+0000 7f76491f9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bbe0ca3f0 0x555bbe0d4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.894+0000 7f76491f9740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555bbe0cade0 0x555bbe0d4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.894+0000 7f76491f9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555bbe0cff50 0x555bbe0dbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.894+0000 7f76491f9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555bbe0d2ea0 con 0x555bbe0ca3f0 2024-03-19T03:32:46.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.894+0000 7f76491f9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x555bbdfeaed0 con 0x555bbe0cff50 2024-03-19T03:32:46.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.894+0000 7f76491f9740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x555bbe0d3020 con 0x555bbe0cade0 2024-03-19T03:32:46.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.894+0000 7f76491f9740 10 client.0 _ll_register_callbacks cb 0x555bbdfe8c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:32:46.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.894+0000 7f763ea00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555bbe0cade0 0x555bbe0d4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.898 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.894+0000 7f763f400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bbe0ca3f0 0x555bbe0d4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.894+0000 7f763fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555bbe0cff50 0x555bbe0dbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.894+0000 7f763ea00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555bbe0cade0 0x555bbe0d4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:38932/0 (socket says 172.21.15.117:38932) 2024-03-19T03:32:46.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.894+0000 7f763ea00640 1 -- 172.21.15.117:0/1544767190 learned_addr learned my addr 172.21.15.117:0/1544767190 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:46.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.895+0000 7f763fe00640 1 -- 172.21.15.117:0/1544767190 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555bbe0cade0 msgr2=0x555bbe0d4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:46.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.895+0000 7f763fe00640 1 --2- 172.21.15.117:0/1544767190 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555bbe0cade0 0x555bbe0d4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.895+0000 7f763fe00640 1 -- 172.21.15.117:0/1544767190 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bbe0ca3f0 msgr2=0x555bbe0d4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:46.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.895+0000 7f763fe00640 1 --2- 172.21.15.117:0/1544767190 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bbe0ca3f0 0x555bbe0d4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.895+0000 7f763fe00640 1 -- 172.21.15.117:0/1544767190 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bbde0d6b0 con 0x555bbe0cff50 2024-03-19T03:32:46.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.895+0000 7f763fe00640 1 --2- 172.21.15.117:0/1544767190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555bbe0cff50 0x555bbe0dbe10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7638009bf0 tx=0x7f7638005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=214fb746ebd4b358 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:46.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.895+0000 7f7629600640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-19T03:32:46.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.895+0000 7f7629600640 1 -- 172.21.15.117:0/1544767190 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7638043030 con 0x555bbe0cff50 2024-03-19T03:32:46.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.895+0000 7f7629600640 1 -- 172.21.15.117:0/1544767190 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7638003b70 con 0x555bbe0cff50 2024-03-19T03:32:46.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.895+0000 7f76491f9740 1 -- 172.21.15.117:0/1544767190 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7638032c90 con 0x555bbe0cff50 2024-03-19T03:32:46.899 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.896+0000 7f7629600640 1 -- 172.21.15.117:0/1544767190 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7638058050 con 0x555bbe0cff50 2024-03-19T03:32:46.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.896+0000 7f76491f9740 10 client.25187 fetch_fsmap learned FSMap version 28 2024-03-19T03:32:46.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.896+0000 7f76491f9740 1 -- 172.21.15.117:0/1544767190 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555bbe0df180 con 0x555bbe0cff50 2024-03-19T03:32:46.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.896+0000 7f7629600640 1 -- 172.21.15.117:0/1544767190 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7638032c90 con 0x555bbe0cff50 2024-03-19T03:32:46.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.896+0000 7f7629600640 1 -- 172.21.15.117:0/1544767190 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f763803f850 con 0x555bbe0cff50 2024-03-19T03:32:46.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.896+0000 7f76491f9740 10 client.25187 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T03:32:46.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.896+0000 7f76491f9740 10 client.25187 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-19T03:32:46.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.896+0000 7f76491f9740 20 client.25187 populate_metadata read hostname 'smithi117' 2024-03-19T03:32:46.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.896+0000 7f76491f9740 1 --2- 172.21.15.117:0/1544767190 >> [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] conn(0x555bbe0e13c0 0x555bbe0e2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.896+0000 7f76491f9740 4 client.25187 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:32:46.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.896+0000 7f76491f9740 1 -- 172.21.15.117:0/1544767190 --> [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555bbe03c240 con 0x555bbe0e13c0 2024-03-19T03:32:46.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.896+0000 7f763f400640 1 --2- 172.21.15.117:0/1544767190 >> [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] conn(0x555bbe0e13c0 0x555bbe0e2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.897+0000 7f763f400640 1 --2- 172.21.15.117:0/1544767190 >> [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] conn(0x555bbe0e13c0 0x555bbe0e2830 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ac9923d3c19c53de server_cookie=c04ddadf6d84b734 in_seq=0 out_seq=0 2024-03-19T03:32:46.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.897+0000 7f7629600640 10 client.25187 ms_handle_connect on v2:172.21.15.119:6836/2054992738 2024-03-19T03:32:46.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.898+0000 7f7629600640 1 -- 172.21.15.117:0/1544767190 <== mds.3 v2:172.21.15.119:6836/2054992738 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555bbe03c240 con 0x555bbe0e13c0 2024-03-19T03:32:46.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.898+0000 7f7629600640 10 client.25187 handle_command_reply: tid=0 2024-03-19T03:32:46.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.898+0000 7f76491f9740 1 client.25187 shutdown 2024-03-19T03:32:46.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.898+0000 7f76491f9740 1 -- 172.21.15.117:0/1544767190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555bbe0cff50 msgr2=0x555bbe0dbe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:46.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.898+0000 7f76491f9740 1 --2- 172.21.15.117:0/1544767190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555bbe0cff50 0x555bbe0dbe10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7638009bf0 tx=0x7f7638005bd0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.898+0000 7f76491f9740 1 -- 172.21.15.117:0/1544767190 shutdown_connections 2024-03-19T03:32:46.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.898+0000 7f76491f9740 1 --2- 172.21.15.117:0/1544767190 >> [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] conn(0x555bbe0e13c0 0x555bbe0e2830 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.898+0000 7f76491f9740 1 --2- 172.21.15.117:0/1544767190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555bbe0cff50 0x555bbe0dbe10 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.898+0000 7f76491f9740 1 --2- 172.21.15.117:0/1544767190 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555bbe0cade0 0x555bbe0d4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.898+0000 7f76491f9740 1 --2- 172.21.15.117:0/1544767190 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bbe0ca3f0 0x555bbe0d4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.898+0000 7f76491f9740 1 -- 172.21.15.117:0/1544767190 >> 172.21.15.117:0/1544767190 conn(0x555bbe0c5960 msgr2=0x555bbe0c4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:46.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.898+0000 7f7629600640 0 client.25187 ms_handle_reset on v2:172.21.15.119:6836/2054992738 2024-03-19T03:32:46.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.898+0000 7f76491f9740 1 -- 172.21.15.117:0/1544767190 shutdown_connections 2024-03-19T03:32:46.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.898+0000 7f76491f9740 1 -- 172.21.15.117:0/1544767190 wait complete. 2024-03-19T03:32:46.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.898+0000 7f76491f9740 20 client.25187 trim_cache size 0 max 16384 2024-03-19T03:32:46.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.920+0000 7f76491f9740 1 Processor -- start 2024-03-19T03:32:46.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.920+0000 7f76491f9740 1 -- start start 2024-03-19T03:32:46.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.920+0000 7f76491f9740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555bbe0ca3f0 0x555bbe0cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.920+0000 7f76491f9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555bbe0cade0 0x555bbe0dbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.920+0000 7f76491f9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bbe0cff50 0x555bbe0d3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.920+0000 7f76491f9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555bbe0e0b70 con 0x555bbe0cff50 2024-03-19T03:32:46.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.920+0000 7f76491f9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x555bbe0e1240 con 0x555bbe0cade0 2024-03-19T03:32:46.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.920+0000 7f76491f9740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x555bbe0e1910 con 0x555bbe0ca3f0 2024-03-19T03:32:46.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.921+0000 7f763ea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555bbe0cade0 0x555bbe0dbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.921+0000 7f763f400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555bbe0ca3f0 0x555bbe0cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.921+0000 7f763ea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555bbe0cade0 0x555bbe0dbdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57918/0 (socket says 172.21.15.117:57918) 2024-03-19T03:32:46.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.921+0000 7f763fe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bbe0cff50 0x555bbe0d3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.921+0000 7f763ea00640 1 -- 172.21.15.117:0/4256712624 learned_addr learned my addr 172.21.15.117:0/4256712624 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:46.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.921+0000 7f763f400640 1 -- 172.21.15.117:0/4256712624 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555bbe0cade0 msgr2=0x555bbe0dbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:46.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.921+0000 7f763f400640 1 --2- 172.21.15.117:0/4256712624 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555bbe0cade0 0x555bbe0dbdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.921+0000 7f763f400640 1 -- 172.21.15.117:0/4256712624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bbe0cff50 msgr2=0x555bbe0d3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:46.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.921+0000 7f763f400640 1 --2- 172.21.15.117:0/4256712624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bbe0cff50 0x555bbe0d3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.921+0000 7f763f400640 1 -- 172.21.15.117:0/4256712624 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bbe0cbdc0 con 0x555bbe0ca3f0 2024-03-19T03:32:46.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.921+0000 7f763f400640 1 --2- 172.21.15.117:0/4256712624 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555bbe0ca3f0 0x555bbe0cb880 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f76300cca50 tx=0x7f7630000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f014160fad2e0ad server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:46.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.921+0000 7f762be00640 1 -- 172.21.15.117:0/4256712624 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76300999e0 con 0x555bbe0ca3f0 2024-03-19T03:32:46.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.922+0000 7f762be00640 1 -- 172.21.15.117:0/4256712624 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f763009bfa0 con 0x555bbe0ca3f0 2024-03-19T03:32:46.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.922+0000 7f762be00640 1 -- 172.21.15.117:0/4256712624 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76300cf150 con 0x555bbe0ca3f0 2024-03-19T03:32:46.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.922+0000 7f76491f9740 1 -- 172.21.15.117:0/4256712624 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555bbe0ca3f0 msgr2=0x555bbe0cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:46.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.922+0000 7f76491f9740 1 --2- 172.21.15.117:0/4256712624 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555bbe0ca3f0 0x555bbe0cb880 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f76300cca50 tx=0x7f7630000bc0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.922+0000 7f76491f9740 1 -- 172.21.15.117:0/4256712624 shutdown_connections 2024-03-19T03:32:46.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.922+0000 7f76491f9740 1 --2- 172.21.15.117:0/4256712624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bbe0cff50 0x555bbe0d3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.922+0000 7f76491f9740 1 --2- 172.21.15.117:0/4256712624 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555bbe0cade0 0x555bbe0dbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.922+0000 7f76491f9740 1 --2- 172.21.15.117:0/4256712624 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555bbe0ca3f0 0x555bbe0cb880 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.922+0000 7f76491f9740 1 -- 172.21.15.117:0/4256712624 >> 172.21.15.117:0/4256712624 conn(0x555bbe0c5960 msgr2=0x555bbe0c4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:46.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.922+0000 7f76491f9740 1 -- 172.21.15.117:0/4256712624 shutdown_connections 2024-03-19T03:32:46.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.922+0000 7f76491f9740 1 -- 172.21.15.117:0/4256712624 wait complete. 2024-03-19T03:32:46.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.923+0000 7f76491f9740 1 -- start start 2024-03-19T03:32:46.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.923+0000 7f76491f9740 1 Processor -- start 2024-03-19T03:32:46.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.924+0000 7f76491f9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bbe0ca3f0 0x555bbe0d7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.924+0000 7f76491f9740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555bbe0cade0 0x555bbe0d84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.924+0000 7f76491f9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555bbe0cff50 0x555bbe0dae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.924+0000 7f76491f9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555bbe0e26b0 con 0x555bbe0ca3f0 2024-03-19T03:32:46.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.924+0000 7f76491f9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x555bbe0e1fe0 con 0x555bbe0cff50 2024-03-19T03:32:46.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.924+0000 7f76491f9740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x555bbe0e2d80 con 0x555bbe0cade0 2024-03-19T03:32:46.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.924+0000 7f76491f9740 10 client.0 _ll_register_callbacks cb 0x7f762c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:32:46.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.924+0000 7f763f400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bbe0ca3f0 0x555bbe0d7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.924+0000 7f763f400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bbe0ca3f0 0x555bbe0d7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:55656/0 (socket says 172.21.15.117:55656) 2024-03-19T03:32:46.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.924+0000 7f763f400640 1 -- 172.21.15.117:0/4045532594 learned_addr learned my addr 172.21.15.117:0/4045532594 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:46.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.924+0000 7f763fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555bbe0cff50 0x555bbe0dae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.924+0000 7f763ea00640 1 --2- 172.21.15.117:0/4045532594 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555bbe0cade0 0x555bbe0d84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.924+0000 7f763fe00640 1 -- 172.21.15.117:0/4045532594 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555bbe0cade0 msgr2=0x555bbe0d84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:46.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.924+0000 7f763fe00640 1 --2- 172.21.15.117:0/4045532594 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555bbe0cade0 0x555bbe0d84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.924+0000 7f763fe00640 1 -- 172.21.15.117:0/4045532594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bbe0ca3f0 msgr2=0x555bbe0d7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:46.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.924+0000 7f763fe00640 1 --2- 172.21.15.117:0/4045532594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bbe0ca3f0 0x555bbe0d7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.924+0000 7f763fe00640 1 -- 172.21.15.117:0/4045532594 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bbe0db420 con 0x555bbe0cff50 2024-03-19T03:32:46.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.925+0000 7f763f400640 1 --2- 172.21.15.117:0/4045532594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bbe0ca3f0 0x555bbe0d7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:32:46.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.925+0000 7f763fe00640 1 --2- 172.21.15.117:0/4045532594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555bbe0cff50 0x555bbe0dae60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7638004af0 tx=0x7f7638032ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef0270f5d6865069 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:46.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.925+0000 7f7629600640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-19T03:32:46.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.925+0000 7f7629600640 1 -- 172.21.15.117:0/4045532594 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7638043040 con 0x555bbe0cff50 2024-03-19T03:32:46.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.925+0000 7f7629600640 1 -- 172.21.15.117:0/4045532594 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7638003b70 con 0x555bbe0cff50 2024-03-19T03:32:46.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.925+0000 7f76491f9740 1 -- 172.21.15.117:0/4045532594 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f763009bfa0 con 0x555bbe0cff50 2024-03-19T03:32:46.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.925+0000 7f7629600640 1 -- 172.21.15.117:0/4045532594 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7638039050 con 0x555bbe0cff50 2024-03-19T03:32:46.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.925+0000 7f76491f9740 10 client.25193 fetch_fsmap learned FSMap version 28 2024-03-19T03:32:46.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.925+0000 7f76491f9740 1 -- 172.21.15.117:0/4045532594 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555bbe147540 con 0x555bbe0cff50 2024-03-19T03:32:46.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.925+0000 7f7629600640 1 -- 172.21.15.117:0/4045532594 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f763009bfa0 con 0x555bbe0cff50 2024-03-19T03:32:46.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.926+0000 7f7629600640 1 -- 172.21.15.117:0/4045532594 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7638004290 con 0x555bbe0cff50 2024-03-19T03:32:46.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.926+0000 7f76491f9740 10 client.25193 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T03:32:46.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.926+0000 7f76491f9740 10 client.25193 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-19T03:32:46.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.926+0000 7f76491f9740 20 client.25193 populate_metadata read hostname 'smithi117' 2024-03-19T03:32:46.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.926+0000 7f76491f9740 1 --2- 172.21.15.117:0/4045532594 >> [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] conn(0x555bbe14c0f0 0x555bbe14ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:46.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.926+0000 7f76491f9740 4 client.25193 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T03:32:46.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.926+0000 7f76491f9740 1 -- 172.21.15.117:0/4045532594 --> [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] -- command(tid 0: {"prefix": "status"}) -- 0x555bbe036590 con 0x555bbe14c0f0 2024-03-19T03:32:46.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.926+0000 7f763f400640 1 --2- 172.21.15.117:0/4045532594 >> [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] conn(0x555bbe14c0f0 0x555bbe14ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:46.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.927+0000 7f763f400640 1 --2- 172.21.15.117:0/4045532594 >> [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] conn(0x555bbe14c0f0 0x555bbe14ae00 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=faaa9dcd44159929 server_cookie=1b5a126cff777837 in_seq=0 out_seq=0 2024-03-19T03:32:46.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.927+0000 7f7629600640 10 client.25193 ms_handle_connect on v2:172.21.15.119:6836/2054992738 2024-03-19T03:32:46.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.927+0000 7f7629600640 1 -- 172.21.15.117:0/4045532594 <== mds.3 v2:172.21.15.119:6836/2054992738 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x555bbe036590 con 0x555bbe14c0f0 2024-03-19T03:32:46.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.927+0000 7f7629600640 10 client.25193 handle_command_reply: tid=0 2024-03-19T03:32:46.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.927+0000 7f76491f9740 1 client.25193 shutdown 2024-03-19T03:32:46.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.927+0000 7f76491f9740 1 -- 172.21.15.117:0/4045532594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555bbe0cff50 msgr2=0x555bbe0dae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:46.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.927+0000 7f76491f9740 1 --2- 172.21.15.117:0/4045532594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555bbe0cff50 0x555bbe0dae60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7638004af0 tx=0x7f7638032ae0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.927+0000 7f76491f9740 1 -- 172.21.15.117:0/4045532594 shutdown_connections 2024-03-19T03:32:46.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.927+0000 7f76491f9740 1 --2- 172.21.15.117:0/4045532594 >> [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] conn(0x555bbe14c0f0 0x555bbe14ae00 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.927+0000 7f76491f9740 1 --2- 172.21.15.117:0/4045532594 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555bbe0cff50 0x555bbe0dae60 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.927+0000 7f76491f9740 1 --2- 172.21.15.117:0/4045532594 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x555bbe0cade0 0x555bbe0d84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.927+0000 7f7629600640 0 client.25193 ms_handle_reset on v2:172.21.15.119:6836/2054992738 2024-03-19T03:32:46.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.927+0000 7f76491f9740 1 --2- 172.21.15.117:0/4045532594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bbe0ca3f0 0x555bbe0d7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.927+0000 7f76491f9740 1 -- 172.21.15.117:0/4045532594 >> 172.21.15.117:0/4045532594 conn(0x555bbe0c5960 msgr2=0x555bbe0c4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:46.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.927+0000 7f76491f9740 1 -- 172.21.15.117:0/4045532594 shutdown_connections 2024-03-19T03:32:46.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.927+0000 7f76491f9740 1 -- 172.21.15.117:0/4045532594 wait complete. 2024-03-19T03:32:46.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.927+0000 7f76491f9740 20 client.25193 trim_cache size 0 max 16384 2024-03-19T03:32:46.936 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-19T03:32:46.936 INFO:teuthology.orchestra.run.smithi117.stdout: "cluster_fsid": "a3a82456-e59f-11ee-95c9-87774f69a715", 2024-03-19T03:32:46.936 INFO:teuthology.orchestra.run.smithi117.stdout: "whoami": 3, 2024-03-19T03:32:46.936 INFO:teuthology.orchestra.run.smithi117.stdout: "id": 24545, 2024-03-19T03:32:46.936 INFO:teuthology.orchestra.run.smithi117.stdout: "want_state": "up:active", 2024-03-19T03:32:46.936 INFO:teuthology.orchestra.run.smithi117.stdout: "state": "up:active", 2024-03-19T03:32:46.937 INFO:teuthology.orchestra.run.smithi117.stdout: "fs_name": "cephfs", 2024-03-19T03:32:46.937 INFO:teuthology.orchestra.run.smithi117.stdout: "rank_uptime": 98.446096592999993, 2024-03-19T03:32:46.937 INFO:teuthology.orchestra.run.smithi117.stdout: "mdsmap_epoch": 28, 2024-03-19T03:32:46.937 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch": 93, 2024-03-19T03:32:46.937 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch_barrier": 89, 2024-03-19T03:32:46.937 INFO:teuthology.orchestra.run.smithi117.stdout: "uptime": 152.69350494 2024-03-19T03:32:46.937 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-19T03:32:46.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.928+0000 7f7646400640 1 -- 172.21.15.117:0/2854799473 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f76240a2790 msgr2=0x7f76240a4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:46.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.928+0000 7f7646400640 1 --2- 172.21.15.117:0/2854799473 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f76240a2790 0x7f76240a4c50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x555bbe0c68d0 tx=0x7f762c005f90 comp rx=0 tx=0).stop 2024-03-19T03:32:46.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.928+0000 7f7646400640 1 -- 172.21.15.117:0/2854799473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7640102350 msgr2=0x7f7640199d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:46.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.928+0000 7f7646400640 1 --2- 172.21.15.117:0/2854799473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7640102350 0x7f7640199d00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f763000e9f0 tx=0x7f763000eeb0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.928+0000 7f7646400640 1 -- 172.21.15.117:0/2854799473 shutdown_connections 2024-03-19T03:32:46.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.928+0000 7f7646400640 1 --2- 172.21.15.117:0/2854799473 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76401075b0 0x7f76401a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.928+0000 7f7646400640 1 --2- 172.21.15.117:0/2854799473 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7640104c80 0x7f764019a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.928+0000 7f7646400640 1 --2- 172.21.15.117:0/2854799473 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f76240a2790 0x7f76240a4c50 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.928+0000 7f7646400640 1 --2- 172.21.15.117:0/2854799473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7640102350 0x7f7640199d00 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:46.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.928+0000 7f7646400640 1 -- 172.21.15.117:0/2854799473 >> 172.21.15.117:0/2854799473 conn(0x7f76400fbcc0 msgr2=0x7f76401019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:46.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.928+0000 7f7646400640 1 -- 172.21.15.117:0/2854799473 shutdown_connections 2024-03-19T03:32:46.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:46.928+0000 7f7646400640 1 -- 172.21.15.117:0/2854799473 wait complete. 2024-03-19T03:32:46.946 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-03-19T03:32:46.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:46 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3993606018' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:32:46.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:46 smithi119 ceph-mon[66861]: pgmap v387: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 895 B/s wr, 12 op/s; 7.9 KiB/s, 1 keys/s, 2 objects/s recovering 2024-03-19T03:32:46.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:46 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1379114545' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:32:47.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.168+0000 7f0934e00640 1 -- 172.21.15.117:0/2336398753 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0930108fb0 msgr2=0x7f0930109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.168+0000 7f0934e00640 1 --2- 172.21.15.117:0/2336398753 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0930108fb0 0x7f0930109430 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f091c009900 tx=0x7f091c02bc40 comp rx=0 tx=0).stop 2024-03-19T03:32:47.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.168+0000 7f0934e00640 1 -- 172.21.15.117:0/2336398753 shutdown_connections 2024-03-19T03:32:47.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.168+0000 7f0934e00640 1 --2- 172.21.15.117:0/2336398753 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0930109970 0x7f0930110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.168+0000 7f0934e00640 1 --2- 172.21.15.117:0/2336398753 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0930108fb0 0x7f0930109430 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.168+0000 7f0934e00640 1 --2- 172.21.15.117:0/2336398753 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0930107e60 0x7f0930108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.168+0000 7f0934e00640 1 -- 172.21.15.117:0/2336398753 >> 172.21.15.117:0/2336398753 conn(0x7f0930076df0 msgr2=0x7f0930079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:47.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.168+0000 7f0934e00640 1 -- 172.21.15.117:0/2336398753 shutdown_connections 2024-03-19T03:32:47.170 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.168+0000 7f0934e00640 1 -- 172.21.15.117:0/2336398753 wait complete. 2024-03-19T03:32:47.172 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.170+0000 7f0934e00640 1 Processor -- start 2024-03-19T03:32:47.172 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.170+0000 7f0934e00640 1 -- start start 2024-03-19T03:32:47.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.170+0000 7f0934e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0930107e60 0x7f093019e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.170+0000 7f0934e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0930108fb0 0x7f093019e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.170+0000 7f0934e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0930109970 0x7f09301a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.170+0000 7f0934e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0930113b50 con 0x7f0930108fb0 2024-03-19T03:32:47.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.170+0000 7f0934e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f09301139d0 con 0x7f0930107e60 2024-03-19T03:32:47.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.170+0000 7f0934e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f0930113cd0 con 0x7f0930109970 2024-03-19T03:32:47.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.170+0000 7f092de00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0930107e60 0x7f093019e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.170+0000 7f092de00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0930107e60 0x7f093019e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57944/0 (socket says 172.21.15.117:57944) 2024-03-19T03:32:47.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.170+0000 7f092de00640 1 -- 172.21.15.117:0/2726923889 learned_addr learned my addr 172.21.15.117:0/2726923889 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:47.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.171+0000 7f092d400640 1 --2- 172.21.15.117:0/2726923889 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0930108fb0 0x7f093019e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.171+0000 7f092e800640 1 --2- 172.21.15.117:0/2726923889 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0930109970 0x7f09301a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.171+0000 7f092de00640 1 -- 172.21.15.117:0/2726923889 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0930109970 msgr2=0x7f09301a56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.173 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.171+0000 7f092de00640 1 --2- 172.21.15.117:0/2726923889 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0930109970 0x7f09301a56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.171+0000 7f092de00640 1 -- 172.21.15.117:0/2726923889 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0930108fb0 msgr2=0x7f093019e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.171+0000 7f092de00640 1 --2- 172.21.15.117:0/2726923889 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0930108fb0 0x7f093019e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.171+0000 7f092de00640 1 -- 172.21.15.117:0/2726923889 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09301a5d20 con 0x7f0930107e60 2024-03-19T03:32:47.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.171+0000 7f092d400640 1 --2- 172.21.15.117:0/2726923889 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0930108fb0 0x7f093019e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:32:47.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.171+0000 7f092de00640 1 --2- 172.21.15.117:0/2726923889 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0930107e60 0x7f093019e100 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f092000e9f0 tx=0x7f092000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:47.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.171+0000 7f0926a00640 1 -- 172.21.15.117:0/2726923889 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f092000cd80 con 0x7f0930107e60 2024-03-19T03:32:47.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.171+0000 7f0926a00640 1 -- 172.21.15.117:0/2726923889 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0920005500 con 0x7f0930107e60 2024-03-19T03:32:47.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.171+0000 7f0926a00640 1 -- 172.21.15.117:0/2726923889 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0920010690 con 0x7f0930107e60 2024-03-19T03:32:47.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.172+0000 7f0934e00640 1 -- 172.21.15.117:0/2726923889 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09301a5fb0 con 0x7f0930107e60 2024-03-19T03:32:47.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.172+0000 7f0934e00640 1 -- 172.21.15.117:0/2726923889 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09301a6470 con 0x7f0930107e60 2024-03-19T03:32:47.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.175+0000 7f0937ba9740 1 Processor -- start 2024-03-19T03:32:47.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.175+0000 7f0937ba9740 1 -- start start 2024-03-19T03:32:47.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.175+0000 7f0937ba9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5579ea5493f0 0x5579ea549810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.175+0000 7f0937ba9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea549de0 0x5579ea54e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.175+0000 7f0937ba9740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5579ea54ef50 0x5579ea5513e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.175+0000 7f0937ba9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5579ea4add00 con 0x5579ea5493f0 2024-03-19T03:32:47.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.176+0000 7f0937ba9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5579ea456b00 con 0x5579ea549de0 2024-03-19T03:32:47.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.176+0000 7f0937ba9740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x5579ea4a9da0 con 0x5579ea54ef50 2024-03-19T03:32:47.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.176+0000 7f092e800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5579ea5493f0 0x5579ea549810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.177 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.176+0000 7f092e800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5579ea5493f0 0x5579ea549810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:55682/0 (socket says 172.21.15.117:55682) 2024-03-19T03:32:47.178 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.176+0000 7f092e800640 1 -- 172.21.15.117:0/346993677 learned_addr learned my addr 172.21.15.117:0/346993677 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:47.178 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.176+0000 7f092d400640 1 --2- 172.21.15.117:0/346993677 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5579ea54ef50 0x5579ea5513e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.178 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.176+0000 7f0926a00640 1 -- 172.21.15.117:0/2726923889 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f09200046c0 con 0x7f0930107e60 2024-03-19T03:32:47.178 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.176+0000 7f092e800640 1 -- 172.21.15.117:0/346993677 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5579ea54ef50 msgr2=0x5579ea5513e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:47.178 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.176+0000 7f092e800640 1 --2- 172.21.15.117:0/346993677 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5579ea54ef50 0x5579ea5513e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.178 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.176+0000 7f092e800640 1 -- 172.21.15.117:0/346993677 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea549de0 msgr2=0x5579ea54e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:47.178 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.176+0000 7f092de00640 1 --2- 172.21.15.117:0/346993677 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea549de0 0x5579ea54e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.178 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.176+0000 7f092e800640 1 --2- 172.21.15.117:0/346993677 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea549de0 0x5579ea54e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.178 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.176+0000 7f092e800640 1 -- 172.21.15.117:0/346993677 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579ea1c22d0 con 0x5579ea5493f0 2024-03-19T03:32:47.178 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.176+0000 7f092de00640 1 --2- 172.21.15.117:0/346993677 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea549de0 0x5579ea54e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T03:32:47.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.177+0000 7f092e800640 1 --2- 172.21.15.117:0/346993677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5579ea5493f0 0x5579ea549810 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0928009ee0 tx=0x7f0928010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c361869d622b1d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:47.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.177+0000 7f091b400640 1 -- 172.21.15.117:0/346993677 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0928016070 con 0x5579ea5493f0 2024-03-19T03:32:47.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.177+0000 7f091b400640 1 -- 172.21.15.117:0/346993677 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0928004d20 con 0x5579ea5493f0 2024-03-19T03:32:47.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.177+0000 7f091b400640 1 -- 172.21.15.117:0/346993677 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09280127b0 con 0x5579ea5493f0 2024-03-19T03:32:47.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.178+0000 7f0926a00640 1 --2- 172.21.15.117:0/2726923889 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f09140a2880 0x7f09140a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.178+0000 7f0926a00640 1 -- 172.21.15.117:0/2726923889 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f09200cf330 con 0x7f0930107e60 2024-03-19T03:32:47.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.178+0000 7f0937ba9740 1 -- 172.21.15.117:0/346993677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5579ea5493f0 msgr2=0x5579ea549810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.178+0000 7f0937ba9740 1 --2- 172.21.15.117:0/346993677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5579ea5493f0 0x5579ea549810 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0928009ee0 tx=0x7f0928010740 comp rx=0 tx=0).stop 2024-03-19T03:32:47.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.178+0000 7f092de00640 1 --2- 172.21.15.117:0/2726923889 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f09140a2880 0x7f09140a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.178+0000 7f0937ba9740 1 -- 172.21.15.117:0/346993677 shutdown_connections 2024-03-19T03:32:47.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.178+0000 7f0937ba9740 1 --2- 172.21.15.117:0/346993677 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5579ea54ef50 0x5579ea5513e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.178+0000 7f0937ba9740 1 --2- 172.21.15.117:0/346993677 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea549de0 0x5579ea54e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.178+0000 7f0937ba9740 1 --2- 172.21.15.117:0/346993677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5579ea5493f0 0x5579ea549810 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.178+0000 7f0937ba9740 1 -- 172.21.15.117:0/346993677 >> 172.21.15.117:0/346993677 conn(0x5579ea544960 msgr2=0x5579ea544d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:47.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.179+0000 7f0937ba9740 1 -- 172.21.15.117:0/346993677 shutdown_connections 2024-03-19T03:32:47.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.179+0000 7f092de00640 1 --2- 172.21.15.117:0/2726923889 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f09140a2880 0x7f09140a4d40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f09200ccdb0 tx=0x7f0920004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:47.181 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.179+0000 7f0937ba9740 1 -- 172.21.15.117:0/346993677 wait complete. 2024-03-19T03:32:47.184 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.179+0000 7f0937ba9740 1 -- start start 2024-03-19T03:32:47.184 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.180+0000 7f0937ba9740 1 Processor -- start 2024-03-19T03:32:47.184 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.180+0000 7f0937ba9740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5579ea5493f0 0x5579ea553890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.184 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.180+0000 7f0937ba9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea549de0 0x5579ea553dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.184 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.180+0000 7f0937ba9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5579ea54ef50 0x5579ea55ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.180+0000 7f0937ba9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5579ea551f00 con 0x5579ea54ef50 2024-03-19T03:32:47.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.180+0000 7f0937ba9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5579ea469ed0 con 0x5579ea549de0 2024-03-19T03:32:47.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.180+0000 7f0937ba9740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x5579ea552080 con 0x5579ea5493f0 2024-03-19T03:32:47.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.180+0000 7f0937ba9740 10 client.0 _ll_register_callbacks cb 0x5579ea467c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:32:47.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.180+0000 7f092d400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea549de0 0x5579ea553dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.180+0000 7f092d400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea549de0 0x5579ea553dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57966/0 (socket says 172.21.15.117:57966) 2024-03-19T03:32:47.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.180+0000 7f092d400640 1 -- 172.21.15.117:0/3287260122 learned_addr learned my addr 172.21.15.117:0/3287260122 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:47.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.180+0000 7f092e800640 1 --2- 172.21.15.117:0/3287260122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5579ea5493f0 0x5579ea553890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.181+0000 7f092e800640 1 --2- 172.21.15.117:0/3287260122 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5579ea54ef50 0x5579ea55ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.181+0000 7f092d400640 1 -- 172.21.15.117:0/3287260122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5579ea5493f0 msgr2=0x5579ea553890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:47.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.181+0000 7f092d400640 1 --2- 172.21.15.117:0/3287260122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5579ea5493f0 0x5579ea553890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.181+0000 7f092d400640 1 -- 172.21.15.117:0/3287260122 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5579ea54ef50 msgr2=0x5579ea55ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:47.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.181+0000 7f092d400640 1 --2- 172.21.15.117:0/3287260122 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5579ea54ef50 0x5579ea55ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.185 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.181+0000 7f092d400640 1 -- 172.21.15.117:0/3287260122 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579ea28c6b0 con 0x5579ea549de0 2024-03-19T03:32:47.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.181+0000 7f092e800640 1 --2- 172.21.15.117:0/3287260122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5579ea5493f0 0x5579ea553890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:32:47.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.181+0000 7f092d400640 1 --2- 172.21.15.117:0/3287260122 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea549de0 0x5579ea553dd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f091c006010 tx=0x7f091c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3531f1976f111628 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:47.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.181+0000 7f090fe00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-19T03:32:47.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.181+0000 7f090fe00640 1 -- 172.21.15.117:0/3287260122 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091c005d50 con 0x5579ea549de0 2024-03-19T03:32:47.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.182+0000 7f0937ba9740 1 -- 172.21.15.117:0/3287260122 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0928004d20 con 0x5579ea549de0 2024-03-19T03:32:47.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.182+0000 7f090fe00640 1 -- 172.21.15.117:0/3287260122 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f091c005020 con 0x5579ea549de0 2024-03-19T03:32:47.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.182+0000 7f090fe00640 1 -- 172.21.15.117:0/3287260122 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f091c04f050 con 0x5579ea549de0 2024-03-19T03:32:47.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.182+0000 7f0937ba9740 10 client.25202 fetch_fsmap learned FSMap version 28 2024-03-19T03:32:47.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.182+0000 7f0937ba9740 1 -- 172.21.15.117:0/3287260122 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5579ea55e1b0 con 0x5579ea549de0 2024-03-19T03:32:47.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.182+0000 7f090fe00640 1 -- 172.21.15.117:0/3287260122 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0928004d20 con 0x5579ea549de0 2024-03-19T03:32:47.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.182+0000 7f090fe00640 1 -- 172.21.15.117:0/3287260122 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f091c0035f0 con 0x5579ea549de0 2024-03-19T03:32:47.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.182+0000 7f0937ba9740 10 client.25202 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T03:32:47.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.182+0000 7f0937ba9740 10 client.25202 resolve_mds: resolved name 'g' to daemon mds.g 2024-03-19T03:32:47.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.182+0000 7f0937ba9740 20 client.25202 populate_metadata read hostname 'smithi117' 2024-03-19T03:32:47.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.182+0000 7f0937ba9740 1 --2- 172.21.15.117:0/3287260122 >> [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] conn(0x5579ea560720 0x5579ea561b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.183+0000 7f0937ba9740 4 client.25202 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:32:47.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.183+0000 7f0937ba9740 1 -- 172.21.15.117:0/3287260122 --> [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579ea4bb240 con 0x5579ea560720 2024-03-19T03:32:47.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.183+0000 7f092e800640 1 --2- 172.21.15.117:0/3287260122 >> [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] conn(0x5579ea560720 0x5579ea561b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.183+0000 7f092e800640 1 --2- 172.21.15.117:0/3287260122 >> [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] conn(0x5579ea560720 0x5579ea561b90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=11c7d1dc6426d321 server_cookie=9f9883c32d859043 in_seq=0 out_seq=0 2024-03-19T03:32:47.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.183+0000 7f090fe00640 10 client.25202 ms_handle_connect on v2:172.21.15.117:6838/684335078 2024-03-19T03:32:47.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.184+0000 7f090fe00640 1 -- 172.21.15.117:0/3287260122 <== mds.2 v2:172.21.15.117:6838/684335078 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5579ea4bb240 con 0x5579ea560720 2024-03-19T03:32:47.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.184+0000 7f090fe00640 10 client.25202 handle_command_reply: tid=0 2024-03-19T03:32:47.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.184+0000 7f0937ba9740 1 client.25202 shutdown 2024-03-19T03:32:47.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.185+0000 7f0937ba9740 1 -- 172.21.15.117:0/3287260122 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea549de0 msgr2=0x5579ea553dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.185+0000 7f0937ba9740 1 --2- 172.21.15.117:0/3287260122 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea549de0 0x5579ea553dd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f091c006010 tx=0x7f091c004270 comp rx=0 tx=0).stop 2024-03-19T03:32:47.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.185+0000 7f0937ba9740 1 -- 172.21.15.117:0/3287260122 shutdown_connections 2024-03-19T03:32:47.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.185+0000 7f0937ba9740 1 --2- 172.21.15.117:0/3287260122 >> [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] conn(0x5579ea560720 0x5579ea561b90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.185+0000 7f0937ba9740 1 --2- 172.21.15.117:0/3287260122 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5579ea54ef50 0x5579ea55ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.185+0000 7f0937ba9740 1 --2- 172.21.15.117:0/3287260122 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea549de0 0x5579ea553dd0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.185+0000 7f0937ba9740 1 --2- 172.21.15.117:0/3287260122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5579ea5493f0 0x5579ea553890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.185+0000 7f0937ba9740 1 -- 172.21.15.117:0/3287260122 >> 172.21.15.117:0/3287260122 conn(0x5579ea544960 msgr2=0x5579ea543aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:47.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.185+0000 7f090fe00640 0 client.25202 ms_handle_reset on v2:172.21.15.117:6838/684335078 2024-03-19T03:32:47.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.185+0000 7f0937ba9740 1 -- 172.21.15.117:0/3287260122 shutdown_connections 2024-03-19T03:32:47.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.185+0000 7f0937ba9740 1 -- 172.21.15.117:0/3287260122 wait complete. 2024-03-19T03:32:47.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.185+0000 7f0937ba9740 20 client.25202 trim_cache size 0 max 16384 2024-03-19T03:32:47.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.204+0000 7f0937ba9740 1 Processor -- start 2024-03-19T03:32:47.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.205+0000 7f0937ba9740 1 -- start start 2024-03-19T03:32:47.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.205+0000 7f0937ba9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea5493f0 0x5579ea543210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.205+0000 7f0937ba9740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5579ea549de0 0x5579ea55add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.205+0000 7f0937ba9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5579ea54ef50 0x5579ea543750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.205+0000 7f0937ba9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5579ea55fed0 con 0x5579ea54ef50 2024-03-19T03:32:47.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.205+0000 7f0937ba9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5579ea5605a0 con 0x5579ea5493f0 2024-03-19T03:32:47.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.205+0000 7f0937ba9740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x5579ea560c70 con 0x5579ea549de0 2024-03-19T03:32:47.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.205+0000 7f092e800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea5493f0 0x5579ea543210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.205+0000 7f092e800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea5493f0 0x5579ea543210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57978/0 (socket says 172.21.15.117:57978) 2024-03-19T03:32:47.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.205+0000 7f092e800640 1 -- 172.21.15.117:0/2386892843 learned_addr learned my addr 172.21.15.117:0/2386892843 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:47.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.205+0000 7f092e800640 1 --2- 172.21.15.117:0/2386892843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5579ea54ef50 0x5579ea543750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.205+0000 7f092d400640 1 --2- 172.21.15.117:0/2386892843 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5579ea549de0 0x5579ea55add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.205+0000 7f092e800640 1 -- 172.21.15.117:0/2386892843 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5579ea549de0 msgr2=0x5579ea55add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:47.208 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.205+0000 7f092e800640 1 --2- 172.21.15.117:0/2386892843 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5579ea549de0 0x5579ea55add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.205+0000 7f092e800640 1 -- 172.21.15.117:0/2386892843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5579ea54ef50 msgr2=0x5579ea543750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:47.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.205+0000 7f092e800640 1 --2- 172.21.15.117:0/2386892843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5579ea54ef50 0x5579ea543750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.205+0000 7f092e800640 1 -- 172.21.15.117:0/2386892843 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579ea54a1c0 con 0x5579ea5493f0 2024-03-19T03:32:47.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.205+0000 7f092d400640 1 --2- 172.21.15.117:0/2386892843 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5579ea549de0 0x5579ea55add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:32:47.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.206+0000 7f092e800640 1 --2- 172.21.15.117:0/2386892843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea5493f0 0x5579ea543210 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0928009ca0 tx=0x7f0928000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27e4d0511ebe56a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:47.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.206+0000 7f091aa00640 1 -- 172.21.15.117:0/2386892843 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0928031070 con 0x5579ea5493f0 2024-03-19T03:32:47.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.206+0000 7f091aa00640 1 -- 172.21.15.117:0/2386892843 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0928005990 con 0x5579ea5493f0 2024-03-19T03:32:47.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.206+0000 7f091aa00640 1 -- 172.21.15.117:0/2386892843 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0928016040 con 0x5579ea5493f0 2024-03-19T03:32:47.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.207+0000 7f0937ba9740 1 -- 172.21.15.117:0/2386892843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea5493f0 msgr2=0x5579ea543210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.207+0000 7f0937ba9740 1 --2- 172.21.15.117:0/2386892843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea5493f0 0x5579ea543210 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0928009ca0 tx=0x7f0928000f40 comp rx=0 tx=0).stop 2024-03-19T03:32:47.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.207+0000 7f0937ba9740 1 -- 172.21.15.117:0/2386892843 shutdown_connections 2024-03-19T03:32:47.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.207+0000 7f0937ba9740 1 --2- 172.21.15.117:0/2386892843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5579ea54ef50 0x5579ea543750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.207+0000 7f0937ba9740 1 --2- 172.21.15.117:0/2386892843 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5579ea549de0 0x5579ea55add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.207+0000 7f0937ba9740 1 --2- 172.21.15.117:0/2386892843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea5493f0 0x5579ea543210 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.207+0000 7f0937ba9740 1 -- 172.21.15.117:0/2386892843 >> 172.21.15.117:0/2386892843 conn(0x5579ea544960 msgr2=0x5579ea544d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:47.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.207+0000 7f0937ba9740 1 -- 172.21.15.117:0/2386892843 shutdown_connections 2024-03-19T03:32:47.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.207+0000 7f0937ba9740 1 -- 172.21.15.117:0/2386892843 wait complete. 2024-03-19T03:32:47.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.208+0000 7f0937ba9740 1 -- start start 2024-03-19T03:32:47.213 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.208+0000 7f0937ba9740 1 Processor -- start 2024-03-19T03:32:47.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.208+0000 7f0937ba9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea5493f0 0x5579ea557000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.208+0000 7f0937ba9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5579ea549de0 0x5579ea557540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.208+0000 7f0937ba9740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5579ea54ef50 0x5579ea55a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.208+0000 7f0937ba9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5579ea561a10 con 0x5579ea549de0 2024-03-19T03:32:47.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.208+0000 7f0937ba9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5579ea561340 con 0x5579ea5493f0 2024-03-19T03:32:47.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.208+0000 7f0937ba9740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x5579ea5620e0 con 0x5579ea54ef50 2024-03-19T03:32:47.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.208+0000 7f0937ba9740 10 client.0 _ll_register_callbacks cb 0x7f091c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:32:47.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.209+0000 7f092e800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea5493f0 0x5579ea557000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.209+0000 7f092e800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea5493f0 0x5579ea557000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57980/0 (socket says 172.21.15.117:57980) 2024-03-19T03:32:47.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.209+0000 7f092e800640 1 -- 172.21.15.117:0/266246398 learned_addr learned my addr 172.21.15.117:0/266246398 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:47.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.209+0000 7f092d400640 1 --2- 172.21.15.117:0/266246398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5579ea549de0 0x5579ea557540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.214 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.209+0000 7f092e800640 1 -- 172.21.15.117:0/266246398 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5579ea54ef50 msgr2=0x5579ea55a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:47.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.209+0000 7f092e800640 1 --2- 172.21.15.117:0/266246398 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5579ea54ef50 0x5579ea55a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.209+0000 7f092e800640 1 -- 172.21.15.117:0/266246398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5579ea549de0 msgr2=0x5579ea557540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:47.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.209+0000 7f092e800640 1 --2- 172.21.15.117:0/266246398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5579ea549de0 0x5579ea557540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.209+0000 7f092e800640 1 -- 172.21.15.117:0/266246398 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579ea55aa10 con 0x5579ea5493f0 2024-03-19T03:32:47.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.209+0000 7f092e800640 1 --2- 172.21.15.117:0/266246398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea5493f0 0x5579ea557000 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0928005fd0 tx=0x7f0928004ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66d0c4888cec5aaa server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:47.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.209+0000 7f090fe00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-19T03:32:47.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.209+0000 7f090fe00640 1 -- 172.21.15.117:0/266246398 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0928031070 con 0x5579ea5493f0 2024-03-19T03:32:47.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.209+0000 7f090fe00640 1 -- 172.21.15.117:0/266246398 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f09280054f0 con 0x5579ea5493f0 2024-03-19T03:32:47.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.209+0000 7f0937ba9740 1 -- 172.21.15.117:0/266246398 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0928005990 con 0x5579ea5493f0 2024-03-19T03:32:47.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.210+0000 7f090fe00640 1 -- 172.21.15.117:0/266246398 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0928016040 con 0x5579ea5493f0 2024-03-19T03:32:47.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.210+0000 7f0937ba9740 10 client.25208 fetch_fsmap learned FSMap version 28 2024-03-19T03:32:47.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.210+0000 7f0937ba9740 1 -- 172.21.15.117:0/266246398 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5579ea5c6db0 con 0x5579ea5493f0 2024-03-19T03:32:47.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.210+0000 7f090fe00640 1 -- 172.21.15.117:0/266246398 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0928005990 con 0x5579ea5493f0 2024-03-19T03:32:47.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.210+0000 7f090fe00640 1 -- 172.21.15.117:0/266246398 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f092800f040 con 0x5579ea5493f0 2024-03-19T03:32:47.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.210+0000 7f0937ba9740 10 client.25208 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T03:32:47.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.210+0000 7f0937ba9740 10 client.25208 resolve_mds: resolved name 'g' to daemon mds.g 2024-03-19T03:32:47.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.210+0000 7f0937ba9740 20 client.25208 populate_metadata read hostname 'smithi117' 2024-03-19T03:32:47.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.210+0000 7f0937ba9740 1 --2- 172.21.15.117:0/266246398 >> [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] conn(0x5579ea5cb2c0 0x5579ea5c9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.210+0000 7f0937ba9740 4 client.25208 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T03:32:47.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.210+0000 7f0937ba9740 1 -- 172.21.15.117:0/266246398 --> [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] -- command(tid 0: {"prefix": "status"}) -- 0x5579ea4b5590 con 0x5579ea5cb2c0 2024-03-19T03:32:47.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.211+0000 7f092d400640 1 --2- 172.21.15.117:0/266246398 >> [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] conn(0x5579ea5cb2c0 0x5579ea5c9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.211+0000 7f092d400640 1 --2- 172.21.15.117:0/266246398 >> [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] conn(0x5579ea5cb2c0 0x5579ea5c9fd0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=79f9623d5f5da3a3 server_cookie=bbc42f3b2ae3e77e in_seq=0 out_seq=0 2024-03-19T03:32:47.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.211+0000 7f090fe00640 10 client.25208 ms_handle_connect on v2:172.21.15.117:6838/684335078 2024-03-19T03:32:47.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.211+0000 7f090fe00640 1 -- 172.21.15.117:0/266246398 <== mds.2 v2:172.21.15.117:6838/684335078 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5579ea4b5590 con 0x5579ea5cb2c0 2024-03-19T03:32:47.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.211+0000 7f090fe00640 10 client.25208 handle_command_reply: tid=0 2024-03-19T03:32:47.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.211+0000 7f0937ba9740 1 client.25208 shutdown 2024-03-19T03:32:47.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.211+0000 7f0937ba9740 1 -- 172.21.15.117:0/266246398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea5493f0 msgr2=0x5579ea557000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.212+0000 7f0937ba9740 1 --2- 172.21.15.117:0/266246398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea5493f0 0x5579ea557000 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0928005fd0 tx=0x7f0928004ef0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.212+0000 7f0937ba9740 1 -- 172.21.15.117:0/266246398 shutdown_connections 2024-03-19T03:32:47.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.212+0000 7f0937ba9740 1 --2- 172.21.15.117:0/266246398 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5579ea54ef50 0x5579ea55a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.212+0000 7f0937ba9740 1 --2- 172.21.15.117:0/266246398 >> [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] conn(0x5579ea5cb2c0 0x5579ea5c9fd0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.212+0000 7f0937ba9740 1 --2- 172.21.15.117:0/266246398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5579ea549de0 0x5579ea557540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.212+0000 7f0937ba9740 1 --2- 172.21.15.117:0/266246398 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5579ea5493f0 0x5579ea557000 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.212+0000 7f090fe00640 0 client.25208 ms_handle_reset on v2:172.21.15.117:6838/684335078 2024-03-19T03:32:47.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.212+0000 7f0937ba9740 1 -- 172.21.15.117:0/266246398 >> 172.21.15.117:0/266246398 conn(0x5579ea544960 msgr2=0x5579ea543210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:47.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.212+0000 7f0937ba9740 1 -- 172.21.15.117:0/266246398 shutdown_connections 2024-03-19T03:32:47.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.212+0000 7f0937ba9740 1 -- 172.21.15.117:0/266246398 wait complete. 2024-03-19T03:32:47.217 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.212+0000 7f0937ba9740 20 client.25208 trim_cache size 0 max 16384 2024-03-19T03:32:47.217 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-19T03:32:47.218 INFO:teuthology.orchestra.run.smithi117.stdout: "cluster_fsid": "a3a82456-e59f-11ee-95c9-87774f69a715", 2024-03-19T03:32:47.218 INFO:teuthology.orchestra.run.smithi117.stdout: "whoami": 2, 2024-03-19T03:32:47.218 INFO:teuthology.orchestra.run.smithi117.stdout: "id": 24569, 2024-03-19T03:32:47.218 INFO:teuthology.orchestra.run.smithi117.stdout: "want_state": "up:active", 2024-03-19T03:32:47.218 INFO:teuthology.orchestra.run.smithi117.stdout: "state": "up:active", 2024-03-19T03:32:47.218 INFO:teuthology.orchestra.run.smithi117.stdout: "fs_name": "cephfs", 2024-03-19T03:32:47.218 INFO:teuthology.orchestra.run.smithi117.stdout: "rank_uptime": 99.746839734000005, 2024-03-19T03:32:47.218 INFO:teuthology.orchestra.run.smithi117.stdout: "mdsmap_epoch": 28, 2024-03-19T03:32:47.218 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch": 93, 2024-03-19T03:32:47.218 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch_barrier": 89, 2024-03-19T03:32:47.218 INFO:teuthology.orchestra.run.smithi117.stdout: "uptime": 145.994956669 2024-03-19T03:32:47.218 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-19T03:32:47.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.213+0000 7f0934e00640 1 -- 172.21.15.117:0/2726923889 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f09140a2880 msgr2=0x7f09140a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.213+0000 7f0934e00640 1 --2- 172.21.15.117:0/2726923889 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f09140a2880 0x7f09140a4d40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f09200ccdb0 tx=0x7f0920004a40 comp rx=0 tx=0).stop 2024-03-19T03:32:47.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.213+0000 7f0934e00640 1 -- 172.21.15.117:0/2726923889 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0930107e60 msgr2=0x7f093019e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.213+0000 7f0934e00640 1 --2- 172.21.15.117:0/2726923889 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0930107e60 0x7f093019e100 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f092000e9f0 tx=0x7f092000edf0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.213+0000 7f0934e00640 1 -- 172.21.15.117:0/2726923889 shutdown_connections 2024-03-19T03:32:47.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.213+0000 7f0934e00640 1 --2- 172.21.15.117:0/2726923889 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0930109970 0x7f09301a56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.213+0000 7f0934e00640 1 --2- 172.21.15.117:0/2726923889 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0930108fb0 0x7f093019e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.213+0000 7f0934e00640 1 --2- 172.21.15.117:0/2726923889 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f09140a2880 0x7f09140a4d40 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.213+0000 7f0934e00640 1 --2- 172.21.15.117:0/2726923889 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0930107e60 0x7f093019e100 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.213+0000 7f0934e00640 1 -- 172.21.15.117:0/2726923889 >> 172.21.15.117:0/2726923889 conn(0x7f0930076df0 msgr2=0x7f093010c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:47.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.213+0000 7f0934e00640 1 -- 172.21.15.117:0/2726923889 shutdown_connections 2024-03-19T03:32:47.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.213+0000 7f0934e00640 1 -- 172.21.15.117:0/2726923889 wait complete. 2024-03-19T03:32:47.233 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-19T03:32:47.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.441+0000 7f4e33e00640 1 -- 172.21.15.117:0/246244619 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e34104a60 msgr2=0x7f4e34104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.441+0000 7f4e33e00640 1 --2- 172.21.15.117:0/246244619 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e34104a60 0x7f4e34104ee0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4e20009900 tx=0x7f4e2002bc30 comp rx=0 tx=0).stop 2024-03-19T03:32:47.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.441+0000 7f4e33e00640 1 -- 172.21.15.117:0/246244619 shutdown_connections 2024-03-19T03:32:47.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.441+0000 7f4e33e00640 1 --2- 172.21.15.117:0/246244619 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4e34105420 0x7f4e3410bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.441+0000 7f4e33e00640 1 --2- 172.21.15.117:0/246244619 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e34104a60 0x7f4e34104ee0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.441+0000 7f4e33e00640 1 --2- 172.21.15.117:0/246244619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e34103870 0x7f4e34103c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.441+0000 7f4e33e00640 1 -- 172.21.15.117:0/246244619 >> 172.21.15.117:0/246244619 conn(0x7f4e340fee00 msgr2=0x7f4e34101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:47.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.441+0000 7f4e33e00640 1 -- 172.21.15.117:0/246244619 shutdown_connections 2024-03-19T03:32:47.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.441+0000 7f4e33e00640 1 -- 172.21.15.117:0/246244619 wait complete. 2024-03-19T03:32:47.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.442+0000 7f4e33e00640 1 Processor -- start 2024-03-19T03:32:47.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.442+0000 7f4e33e00640 1 -- start start 2024-03-19T03:32:47.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.442+0000 7f4e33e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e34103870 0x7f4e34199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.442+0000 7f4e33e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e34104a60 0x7f4e3419a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.442+0000 7f4e33e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4e34105420 0x7f4e341a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.442+0000 7f4e33e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3410f540 con 0x7f4e34103870 2024-03-19T03:32:47.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.442+0000 7f4e33e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3410f3c0 con 0x7f4e34104a60 2024-03-19T03:32:47.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.442+0000 7f4e33e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3410f6c0 con 0x7f4e34105420 2024-03-19T03:32:47.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.443+0000 7f4e32a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e34103870 0x7f4e34199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.443+0000 7f4e32000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e34104a60 0x7f4e3419a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.443+0000 7f4e32a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e34103870 0x7f4e34199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:55730/0 (socket says 172.21.15.117:55730) 2024-03-19T03:32:47.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.443+0000 7f4e32a00640 1 -- 172.21.15.117:0/2409981280 learned_addr learned my addr 172.21.15.117:0/2409981280 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:47.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.443+0000 7f4e33400640 1 --2- 172.21.15.117:0/2409981280 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4e34105420 0x7f4e341a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.443+0000 7f4e32000640 1 -- 172.21.15.117:0/2409981280 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4e34105420 msgr2=0x7f4e341a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.443+0000 7f4e32000640 1 --2- 172.21.15.117:0/2409981280 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4e34105420 0x7f4e341a1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.443+0000 7f4e32000640 1 -- 172.21.15.117:0/2409981280 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e34103870 msgr2=0x7f4e34199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.443+0000 7f4e32000640 1 --2- 172.21.15.117:0/2409981280 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e34103870 0x7f4e34199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.443+0000 7f4e32000640 1 -- 172.21.15.117:0/2409981280 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e341a16f0 con 0x7f4e34104a60 2024-03-19T03:32:47.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.444+0000 7f4e32000640 1 --2- 172.21.15.117:0/2409981280 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e34104a60 0x7f4e3419a010 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4e20036040 tx=0x7f4e2002c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:47.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.444+0000 7f4e2b400640 1 -- 172.21.15.117:0/2409981280 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e20043020 con 0x7f4e34104a60 2024-03-19T03:32:47.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.444+0000 7f4e2b400640 1 -- 172.21.15.117:0/2409981280 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e200054e0 con 0x7f4e34104a60 2024-03-19T03:32:47.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.444+0000 7f4e33e00640 1 -- 172.21.15.117:0/2409981280 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e341a1980 con 0x7f4e34104a60 2024-03-19T03:32:47.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.444+0000 7f4e2b400640 1 -- 172.21.15.117:0/2409981280 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e20003ce0 con 0x7f4e34104a60 2024-03-19T03:32:47.446 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.444+0000 7f4e33e00640 1 -- 172.21.15.117:0/2409981280 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e341a1ef0 con 0x7f4e34104a60 2024-03-19T03:32:47.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.447+0000 7f4e2b400640 1 -- 172.21.15.117:0/2409981280 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f4e20046020 con 0x7f4e34104a60 2024-03-19T03:32:47.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.448+0000 7f4e3c78a740 1 Processor -- start 2024-03-19T03:32:47.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.448+0000 7f4e3c78a740 1 -- start start 2024-03-19T03:32:47.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.448+0000 7f4e3c78a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601b67165c0 0x5601b67169e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.448+0000 7f4e3c78a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b6716fb0 0x5601b671bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.448+0000 7f4e3c78a740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5601b671c120 0x5601b671e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.448+0000 7f4e3c78a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5601b667ad00 con 0x5601b67165c0 2024-03-19T03:32:47.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.448+0000 7f4e3c78a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5601b6623b00 con 0x5601b6716fb0 2024-03-19T03:32:47.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.448+0000 7f4e3c78a740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x5601b6676da0 con 0x5601b671c120 2024-03-19T03:32:47.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.448+0000 7f4e33400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601b67165c0 0x5601b67169e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.448+0000 7f4e32a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b6716fb0 0x5601b671bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.448+0000 7f4e32a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b6716fb0 0x5601b671bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:57996/0 (socket says 172.21.15.117:57996) 2024-03-19T03:32:47.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.448+0000 7f4e32a00640 1 -- 172.21.15.117:0/1927928373 learned_addr learned my addr 172.21.15.117:0/1927928373 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:47.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.449+0000 7f4e32000640 1 --2- 172.21.15.117:0/1927928373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5601b671c120 0x5601b671e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.449+0000 7f4e2b400640 1 --2- 172.21.15.117:0/2409981280 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4e180a2880 0x7f4e180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.449+0000 7f4e32a00640 1 -- 172.21.15.117:0/1927928373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5601b671c120 msgr2=0x5601b671e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:47.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.449+0000 7f4e32a00640 1 --2- 172.21.15.117:0/1927928373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5601b671c120 0x5601b671e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.449+0000 7f4e32a00640 1 -- 172.21.15.117:0/1927928373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601b67165c0 msgr2=0x5601b67169e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:47.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.449+0000 7f4e32a00640 1 --2- 172.21.15.117:0/1927928373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601b67165c0 0x5601b67169e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.449+0000 7f4e32a00640 1 -- 172.21.15.117:0/1927928373 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601b638f2d0 con 0x5601b6716fb0 2024-03-19T03:32:47.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.449+0000 7f4e32000640 1 --2- 172.21.15.117:0/1927928373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5601b671c120 0x5601b671e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T03:32:47.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.449+0000 7f4e2b400640 1 -- 172.21.15.117:0/2409981280 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4e200f0f40 con 0x7f4e34104a60 2024-03-19T03:32:47.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.449+0000 7f4e33400640 1 --2- 172.21.15.117:0/2409981280 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4e180a2880 0x7f4e180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.449+0000 7f4e32a00640 1 --2- 172.21.15.117:0/1927928373 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b6716fb0 0x5601b671bab0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4e24009d80 tx=0x7f4e24011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd19e9828bfeaa18 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:47.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.449+0000 7f4e1fe00640 1 -- 172.21.15.117:0/1927928373 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2401e070 con 0x5601b6716fb0 2024-03-19T03:32:47.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.449+0000 7f4e1fe00640 1 -- 172.21.15.117:0/1927928373 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e24019b10 con 0x5601b6716fb0 2024-03-19T03:32:47.452 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.449+0000 7f4e1fe00640 1 -- 172.21.15.117:0/1927928373 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2400eb80 con 0x5601b6716fb0 2024-03-19T03:32:47.453 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.450+0000 7f4e3c78a740 1 -- 172.21.15.117:0/1927928373 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b6716fb0 msgr2=0x5601b671bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.453 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.450+0000 7f4e3c78a740 1 --2- 172.21.15.117:0/1927928373 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b6716fb0 0x5601b671bab0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4e24009d80 tx=0x7f4e24011810 comp rx=0 tx=0).stop 2024-03-19T03:32:47.453 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.450+0000 7f4e33400640 1 --2- 172.21.15.117:0/2409981280 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4e180a2880 0x7f4e180a4d40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x5601b6717810 tx=0x7f4e2c004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:47.453 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.450+0000 7f4e3c78a740 1 -- 172.21.15.117:0/1927928373 shutdown_connections 2024-03-19T03:32:47.453 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.450+0000 7f4e3c78a740 1 --2- 172.21.15.117:0/1927928373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5601b671c120 0x5601b671e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.453 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.450+0000 7f4e3c78a740 1 --2- 172.21.15.117:0/1927928373 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b6716fb0 0x5601b671bab0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.453 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.450+0000 7f4e3c78a740 1 --2- 172.21.15.117:0/1927928373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601b67165c0 0x5601b67169e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.453 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.450+0000 7f4e3c78a740 1 -- 172.21.15.117:0/1927928373 >> 172.21.15.117:0/1927928373 conn(0x5601b6711b30 msgr2=0x5601b6711f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:47.453 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.450+0000 7f4e3c78a740 1 -- 172.21.15.117:0/1927928373 shutdown_connections 2024-03-19T03:32:47.453 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.450+0000 7f4e3c78a740 1 -- 172.21.15.117:0/1927928373 wait complete. 2024-03-19T03:32:47.453 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.451+0000 7f4e3c78a740 1 -- start start 2024-03-19T03:32:47.453 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.451+0000 7f4e3c78a740 1 Processor -- start 2024-03-19T03:32:47.456 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.451+0000 7f4e3c78a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601b67165c0 0x5601b6720ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.456 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.451+0000 7f4e3c78a740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5601b6716fb0 0x5601b6721020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.456 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.451+0000 7f4e3c78a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b671c120 0x5601b6728060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.456 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.452+0000 7f4e3c78a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5601b671f0d0 con 0x5601b67165c0 2024-03-19T03:32:47.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.452+0000 7f4e3c78a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5601b6636ed0 con 0x5601b671c120 2024-03-19T03:32:47.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.452+0000 7f4e3c78a740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x5601b671f250 con 0x5601b6716fb0 2024-03-19T03:32:47.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.452+0000 7f4e32a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601b67165c0 0x5601b6720ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.452+0000 7f4e33400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b671c120 0x5601b6728060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.452+0000 7f4e3c78a740 10 client.0 _ll_register_callbacks cb 0x5601b6634c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:32:47.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.452+0000 7f4e33400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b671c120 0x5601b6728060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:58008/0 (socket says 172.21.15.117:58008) 2024-03-19T03:32:47.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.452+0000 7f4e33400640 1 -- 172.21.15.117:0/1967050495 learned_addr learned my addr 172.21.15.117:0/1967050495 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:47.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.452+0000 7f4e32000640 1 --2- 172.21.15.117:0/1967050495 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5601b6716fb0 0x5601b6721020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.452+0000 7f4e33400640 1 -- 172.21.15.117:0/1967050495 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5601b6716fb0 msgr2=0x5601b6721020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:47.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.452+0000 7f4e33400640 1 --2- 172.21.15.117:0/1967050495 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5601b6716fb0 0x5601b6721020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.452+0000 7f4e33400640 1 -- 172.21.15.117:0/1967050495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601b67165c0 msgr2=0x5601b6720ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:47.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.452+0000 7f4e33400640 1 --2- 172.21.15.117:0/1967050495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601b67165c0 0x5601b6720ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.452+0000 7f4e33400640 1 -- 172.21.15.117:0/1967050495 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601b64596b0 con 0x5601b671c120 2024-03-19T03:32:47.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.453+0000 7f4e33400640 1 --2- 172.21.15.117:0/1967050495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b671c120 0x5601b6728060 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c013890 tx=0x7f4e2c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f861e442641dd3a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:47.457 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.453+0000 7f4e1cc00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-19T03:32:47.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.453+0000 7f4e1cc00640 1 -- 172.21.15.117:0/1967050495 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c01ed70 con 0x5601b671c120 2024-03-19T03:32:47.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.453+0000 7f4e1cc00640 1 -- 172.21.15.117:0/1967050495 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4e2c028470 con 0x5601b671c120 2024-03-19T03:32:47.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.453+0000 7f4e3c78a740 1 -- 172.21.15.117:0/1967050495 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e24019b10 con 0x5601b671c120 2024-03-19T03:32:47.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.453+0000 7f4e1cc00640 1 -- 172.21.15.117:0/1967050495 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4e2c013e10 con 0x5601b671c120 2024-03-19T03:32:47.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.453+0000 7f4e3c78a740 10 client.25220 fetch_fsmap learned FSMap version 28 2024-03-19T03:32:47.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.453+0000 7f4e3c78a740 1 -- 172.21.15.117:0/1967050495 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5601b672b400 con 0x5601b671c120 2024-03-19T03:32:47.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.454+0000 7f4e1cc00640 1 -- 172.21.15.117:0/1967050495 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e24019b10 con 0x5601b671c120 2024-03-19T03:32:47.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.454+0000 7f4e1cc00640 1 -- 172.21.15.117:0/1967050495 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4e2c00f040 con 0x5601b671c120 2024-03-19T03:32:47.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.454+0000 7f4e3c78a740 10 client.25220 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T03:32:47.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.454+0000 7f4e3c78a740 10 client.25220 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-19T03:32:47.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.454+0000 7f4e3c78a740 20 client.25220 populate_metadata read hostname 'smithi117' 2024-03-19T03:32:47.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.454+0000 7f4e3c78a740 1 --2- 172.21.15.117:0/1967050495 >> [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] conn(0x5601b672db60 0x5601b672efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.454+0000 7f4e3c78a740 4 client.25220 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:32:47.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.454+0000 7f4e3c78a740 1 -- 172.21.15.117:0/1967050495 --> [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5601b6688240 con 0x5601b672db60 2024-03-19T03:32:47.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.454+0000 7f4e32a00640 1 --2- 172.21.15.117:0/1967050495 >> [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] conn(0x5601b672db60 0x5601b672efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.455+0000 7f4e1cc00640 10 client.25220 ms_handle_connect on v2:172.21.15.117:6840/1329933892 2024-03-19T03:32:47.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.455+0000 7f4e32a00640 1 --2- 172.21.15.117:0/1967050495 >> [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] conn(0x5601b672db60 0x5601b672efd0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ef7442ef35df6dd7 server_cookie=22ed66611403e22a in_seq=0 out_seq=0 2024-03-19T03:32:47.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.457+0000 7f4e1cc00640 1 -- 172.21.15.117:0/1967050495 <== mds.1 v2:172.21.15.117:6840/1329933892 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5601b6688240 con 0x5601b672db60 2024-03-19T03:32:47.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.457+0000 7f4e1cc00640 10 client.25220 handle_command_reply: tid=0 2024-03-19T03:32:47.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.457+0000 7f4e3c78a740 1 client.25220 shutdown 2024-03-19T03:32:47.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.457+0000 7f4e3c78a740 1 -- 172.21.15.117:0/1967050495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b671c120 msgr2=0x5601b6728060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.457+0000 7f4e3c78a740 1 --2- 172.21.15.117:0/1967050495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b671c120 0x5601b6728060 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c013890 tx=0x7f4e2c004ab0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.457+0000 7f4e3c78a740 1 -- 172.21.15.117:0/1967050495 shutdown_connections 2024-03-19T03:32:47.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.457+0000 7f4e3c78a740 1 --2- 172.21.15.117:0/1967050495 >> [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] conn(0x5601b672db60 0x5601b672efd0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.460 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.457+0000 7f4e3c78a740 1 --2- 172.21.15.117:0/1967050495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b671c120 0x5601b6728060 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.457+0000 7f4e3c78a740 1 --2- 172.21.15.117:0/1967050495 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5601b6716fb0 0x5601b6721020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.457+0000 7f4e1cc00640 0 client.25220 ms_handle_reset on v2:172.21.15.117:6840/1329933892 2024-03-19T03:32:47.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.457+0000 7f4e3c78a740 1 --2- 172.21.15.117:0/1967050495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601b67165c0 0x5601b6720ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.457+0000 7f4e3c78a740 1 -- 172.21.15.117:0/1967050495 >> 172.21.15.117:0/1967050495 conn(0x5601b6711b30 msgr2=0x5601b6710d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:47.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.457+0000 7f4e3c78a740 1 -- 172.21.15.117:0/1967050495 shutdown_connections 2024-03-19T03:32:47.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.457+0000 7f4e3c78a740 1 -- 172.21.15.117:0/1967050495 wait complete. 2024-03-19T03:32:47.461 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.457+0000 7f4e3c78a740 20 client.25220 trim_cache size 0 max 16384 2024-03-19T03:32:47.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.477+0000 7f4e3c78a740 1 Processor -- start 2024-03-19T03:32:47.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.477+0000 7f4e3c78a740 1 -- start start 2024-03-19T03:32:47.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.477+0000 7f4e3c78a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b67165c0 0x5601b6711f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.477+0000 7f4e3c78a740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5601b6716fb0 0x5601b6727f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.477+0000 7f4e3c78a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601b671c120 0x5601b67206f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.477+0000 7f4e3c78a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5601b672d310 con 0x5601b671c120 2024-03-19T03:32:47.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.477+0000 7f4e3c78a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5601b672d9e0 con 0x5601b67165c0 2024-03-19T03:32:47.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.477+0000 7f4e3c78a740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x5601b672e0b0 con 0x5601b6716fb0 2024-03-19T03:32:47.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.478+0000 7f4e33400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601b671c120 0x5601b67206f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.478+0000 7f4e32a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b67165c0 0x5601b6711f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.478+0000 7f4e32a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b67165c0 0x5601b6711f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:58014/0 (socket says 172.21.15.117:58014) 2024-03-19T03:32:47.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.478+0000 7f4e32a00640 1 -- 172.21.15.117:0/1153550808 learned_addr learned my addr 172.21.15.117:0/1153550808 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:47.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.478+0000 7f4e32000640 1 --2- 172.21.15.117:0/1153550808 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5601b6716fb0 0x5601b6727f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.478+0000 7f4e32a00640 1 -- 172.21.15.117:0/1153550808 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5601b6716fb0 msgr2=0x5601b6727f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:47.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.478+0000 7f4e32a00640 1 --2- 172.21.15.117:0/1153550808 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5601b6716fb0 0x5601b6727f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.478+0000 7f4e32a00640 1 -- 172.21.15.117:0/1153550808 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601b671c120 msgr2=0x5601b67206f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:47.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.478+0000 7f4e32a00640 1 --2- 172.21.15.117:0/1153550808 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601b671c120 0x5601b67206f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.478+0000 7f4e32a00640 1 -- 172.21.15.117:0/1153550808 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601b671f410 con 0x5601b67165c0 2024-03-19T03:32:47.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.478+0000 7f4e32000640 1 --2- 172.21.15.117:0/1153550808 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5601b6716fb0 0x5601b6727f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:32:47.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.478+0000 7f4e32a00640 1 --2- 172.21.15.117:0/1153550808 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b67165c0 0x5601b6711f10 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4e24013010 tx=0x7f4e240051d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95fa1cc0d705782c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:47.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.479+0000 7f4e1f400640 1 -- 172.21.15.117:0/1153550808 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2401e040 con 0x5601b67165c0 2024-03-19T03:32:47.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.479+0000 7f4e1f400640 1 -- 172.21.15.117:0/1153550808 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e2401f040 con 0x5601b67165c0 2024-03-19T03:32:47.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.479+0000 7f4e1f400640 1 -- 172.21.15.117:0/1153550808 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2402a430 con 0x5601b67165c0 2024-03-19T03:32:47.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.479+0000 7f4e3c78a740 1 -- 172.21.15.117:0/1153550808 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b67165c0 msgr2=0x5601b6711f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.479+0000 7f4e3c78a740 1 --2- 172.21.15.117:0/1153550808 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b67165c0 0x5601b6711f10 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4e24013010 tx=0x7f4e240051d0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.479+0000 7f4e3c78a740 1 -- 172.21.15.117:0/1153550808 shutdown_connections 2024-03-19T03:32:47.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.479+0000 7f4e3c78a740 1 --2- 172.21.15.117:0/1153550808 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601b671c120 0x5601b67206f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.479+0000 7f4e3c78a740 1 --2- 172.21.15.117:0/1153550808 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5601b6716fb0 0x5601b6727f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.479+0000 7f4e3c78a740 1 --2- 172.21.15.117:0/1153550808 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b67165c0 0x5601b6711f10 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.479+0000 7f4e3c78a740 1 -- 172.21.15.117:0/1153550808 >> 172.21.15.117:0/1153550808 conn(0x5601b6711b30 msgr2=0x5601b671f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:47.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.479+0000 7f4e3c78a740 1 -- 172.21.15.117:0/1153550808 shutdown_connections 2024-03-19T03:32:47.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.479+0000 7f4e3c78a740 1 -- 172.21.15.117:0/1153550808 wait complete. 2024-03-19T03:32:47.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.480+0000 7f4e3c78a740 1 -- start start 2024-03-19T03:32:47.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.480+0000 7f4e3c78a740 1 Processor -- start 2024-03-19T03:32:47.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.481+0000 7f4e3c78a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b67165c0 0x5601b6723f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.481+0000 7f4e3c78a740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5601b6716fb0 0x5601b67244b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.481+0000 7f4e3c78a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601b671c120 0x5601b6727410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.481+0000 7f4e3c78a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5601b672ee50 con 0x5601b671c120 2024-03-19T03:32:47.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.481+0000 7f4e3c78a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x5601b672e780 con 0x5601b67165c0 2024-03-19T03:32:47.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.481+0000 7f4e3c78a740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x5601b672f520 con 0x5601b6716fb0 2024-03-19T03:32:47.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.481+0000 7f4e3c78a740 10 client.0 _ll_register_callbacks cb 0x7f4e2c009f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:32:47.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.481+0000 7f4e32a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b67165c0 0x5601b6723f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.481+0000 7f4e32a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b67165c0 0x5601b6723f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:58026/0 (socket says 172.21.15.117:58026) 2024-03-19T03:32:47.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.481+0000 7f4e32a00640 1 -- 172.21.15.117:0/888089672 learned_addr learned my addr 172.21.15.117:0/888089672 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:47.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.481+0000 7f4e33400640 1 --2- 172.21.15.117:0/888089672 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601b671c120 0x5601b6727410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.481+0000 7f4e32000640 1 --2- 172.21.15.117:0/888089672 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5601b6716fb0 0x5601b67244b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.481+0000 7f4e32a00640 1 -- 172.21.15.117:0/888089672 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5601b6716fb0 msgr2=0x5601b67244b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:47.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.481+0000 7f4e32a00640 1 --2- 172.21.15.117:0/888089672 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5601b6716fb0 0x5601b67244b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.481+0000 7f4e32a00640 1 -- 172.21.15.117:0/888089672 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601b671c120 msgr2=0x5601b6727410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:47.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.481+0000 7f4e32a00640 1 --2- 172.21.15.117:0/888089672 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601b671c120 0x5601b6727410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.485 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.481+0000 7f4e32a00640 1 -- 172.21.15.117:0/888089672 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601b6727a00 con 0x5601b67165c0 2024-03-19T03:32:47.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.482+0000 7f4e32a00640 1 --2- 172.21.15.117:0/888089672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b67165c0 0x5601b6723f70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4e240058c0 tx=0x7f4e24009d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=963e6d5460003d95 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:47.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.482+0000 7f4e1cc00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-19T03:32:47.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.482+0000 7f4e1cc00640 1 -- 172.21.15.117:0/888089672 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e24039070 con 0x5601b67165c0 2024-03-19T03:32:47.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.482+0000 7f4e3c78a740 1 -- 172.21.15.117:0/888089672 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e2401f040 con 0x5601b67165c0 2024-03-19T03:32:47.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.482+0000 7f4e1cc00640 1 -- 172.21.15.117:0/888089672 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4e24004020 con 0x5601b67165c0 2024-03-19T03:32:47.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.482+0000 7f4e1cc00640 1 -- 172.21.15.117:0/888089672 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4e24045050 con 0x5601b67165c0 2024-03-19T03:32:47.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.482+0000 7f4e3c78a740 10 client.25226 fetch_fsmap learned FSMap version 28 2024-03-19T03:32:47.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.482+0000 7f4e3c78a740 1 -- 172.21.15.117:0/888089672 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5601b67940d0 con 0x5601b67165c0 2024-03-19T03:32:47.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.483+0000 7f4e1cc00640 1 -- 172.21.15.117:0/888089672 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e2401f040 con 0x5601b67165c0 2024-03-19T03:32:47.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.483+0000 7f4e1cc00640 1 -- 172.21.15.117:0/888089672 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4e24024040 con 0x5601b67165c0 2024-03-19T03:32:47.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.483+0000 7f4e3c78a740 10 client.25226 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T03:32:47.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.483+0000 7f4e3c78a740 10 client.25226 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-19T03:32:47.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.483+0000 7f4e3c78a740 20 client.25226 populate_metadata read hostname 'smithi117' 2024-03-19T03:32:47.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.483+0000 7f4e3c78a740 1 --2- 172.21.15.117:0/888089672 >> [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] conn(0x5601b6798560 0x5601b6797270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.483+0000 7f4e3c78a740 4 client.25226 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T03:32:47.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.483+0000 7f4e3c78a740 1 -- 172.21.15.117:0/888089672 --> [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] -- command(tid 0: {"prefix": "status"}) -- 0x5601b6682590 con 0x5601b6798560 2024-03-19T03:32:47.487 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.483+0000 7f4e32000640 1 --2- 172.21.15.117:0/888089672 >> [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] conn(0x5601b6798560 0x5601b6797270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.488 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.484+0000 7f4e32000640 1 --2- 172.21.15.117:0/888089672 >> [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] conn(0x5601b6798560 0x5601b6797270 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cb85e40ed86afa97 server_cookie=a677cc68ec3d894c in_seq=0 out_seq=0 2024-03-19T03:32:47.488 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.484+0000 7f4e1cc00640 10 client.25226 ms_handle_connect on v2:172.21.15.117:6840/1329933892 2024-03-19T03:32:47.488 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.484+0000 7f4e1cc00640 1 -- 172.21.15.117:0/888089672 <== mds.1 v2:172.21.15.117:6840/1329933892 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5601b6682590 con 0x5601b6798560 2024-03-19T03:32:47.488 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.484+0000 7f4e1cc00640 10 client.25226 handle_command_reply: tid=0 2024-03-19T03:32:47.488 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.484+0000 7f4e3c78a740 1 client.25226 shutdown 2024-03-19T03:32:47.488 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.484+0000 7f4e3c78a740 1 -- 172.21.15.117:0/888089672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b67165c0 msgr2=0x5601b6723f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.488 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.484+0000 7f4e3c78a740 1 --2- 172.21.15.117:0/888089672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b67165c0 0x5601b6723f70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4e240058c0 tx=0x7f4e24009d30 comp rx=0 tx=0).stop 2024-03-19T03:32:47.488 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.485+0000 7f4e3c78a740 1 -- 172.21.15.117:0/888089672 shutdown_connections 2024-03-19T03:32:47.488 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.485+0000 7f4e3c78a740 1 --2- 172.21.15.117:0/888089672 >> [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] conn(0x5601b6798560 0x5601b6797270 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.488 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.485+0000 7f4e3c78a740 1 --2- 172.21.15.117:0/888089672 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601b671c120 0x5601b6727410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.488 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.485+0000 7f4e3c78a740 1 --2- 172.21.15.117:0/888089672 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5601b6716fb0 0x5601b67244b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.485+0000 7f4e1cc00640 0 client.25226 ms_handle_reset on v2:172.21.15.117:6840/1329933892 2024-03-19T03:32:47.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.485+0000 7f4e3c78a740 1 --2- 172.21.15.117:0/888089672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5601b67165c0 0x5601b6723f70 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.485+0000 7f4e3c78a740 1 -- 172.21.15.117:0/888089672 >> 172.21.15.117:0/888089672 conn(0x5601b6711b30 msgr2=0x5601b6711f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:47.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.485+0000 7f4e3c78a740 1 -- 172.21.15.117:0/888089672 shutdown_connections 2024-03-19T03:32:47.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.485+0000 7f4e3c78a740 1 -- 172.21.15.117:0/888089672 wait complete. 2024-03-19T03:32:47.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.485+0000 7f4e3c78a740 20 client.25226 trim_cache size 0 max 16384 2024-03-19T03:32:47.489 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-19T03:32:47.489 INFO:teuthology.orchestra.run.smithi117.stdout: "cluster_fsid": "a3a82456-e59f-11ee-95c9-87774f69a715", 2024-03-19T03:32:47.489 INFO:teuthology.orchestra.run.smithi117.stdout: "whoami": 1, 2024-03-19T03:32:47.489 INFO:teuthology.orchestra.run.smithi117.stdout: "id": 24581, 2024-03-19T03:32:47.489 INFO:teuthology.orchestra.run.smithi117.stdout: "want_state": "up:active", 2024-03-19T03:32:47.490 INFO:teuthology.orchestra.run.smithi117.stdout: "state": "up:active", 2024-03-19T03:32:47.490 INFO:teuthology.orchestra.run.smithi117.stdout: "fs_name": "cephfs", 2024-03-19T03:32:47.490 INFO:teuthology.orchestra.run.smithi117.stdout: "rank_uptime": 101.03181220899999, 2024-03-19T03:32:47.490 INFO:teuthology.orchestra.run.smithi117.stdout: "mdsmap_epoch": 28, 2024-03-19T03:32:47.490 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch": 93, 2024-03-19T03:32:47.490 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch_barrier": 89, 2024-03-19T03:32:47.490 INFO:teuthology.orchestra.run.smithi117.stdout: "uptime": 142.10186467099999 2024-03-19T03:32:47.490 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-19T03:32:47.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.486+0000 7f4e33e00640 1 -- 172.21.15.117:0/2409981280 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4e180a2880 msgr2=0x7f4e180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.486+0000 7f4e33e00640 1 --2- 172.21.15.117:0/2409981280 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4e180a2880 0x7f4e180a4d40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x5601b6717810 tx=0x7f4e2c004a40 comp rx=0 tx=0).stop 2024-03-19T03:32:47.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.486+0000 7f4e33e00640 1 -- 172.21.15.117:0/2409981280 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e34104a60 msgr2=0x7f4e3419a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.486+0000 7f4e33e00640 1 --2- 172.21.15.117:0/2409981280 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e34104a60 0x7f4e3419a010 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4e20036040 tx=0x7f4e2002c6c0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.486+0000 7f4e33e00640 1 -- 172.21.15.117:0/2409981280 shutdown_connections 2024-03-19T03:32:47.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.486+0000 7f4e33e00640 1 --2- 172.21.15.117:0/2409981280 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4e34105420 0x7f4e341a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.486+0000 7f4e33e00640 1 --2- 172.21.15.117:0/2409981280 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4e180a2880 0x7f4e180a4d40 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.486+0000 7f4e33e00640 1 --2- 172.21.15.117:0/2409981280 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e34104a60 0x7f4e3419a010 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.486+0000 7f4e33e00640 1 --2- 172.21.15.117:0/2409981280 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e34103870 0x7f4e34199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.486+0000 7f4e33e00640 1 -- 172.21.15.117:0/2409981280 >> 172.21.15.117:0/2409981280 conn(0x7f4e340fee00 msgr2=0x7f4e34107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:47.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.486+0000 7f4e33e00640 1 -- 172.21.15.117:0/2409981280 shutdown_connections 2024-03-19T03:32:47.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.486+0000 7f4e33e00640 1 -- 172.21.15.117:0/2409981280 wait complete. 2024-03-19T03:32:47.505 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-19T03:32:47.696 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.695+0000 7f7b9e600640 1 -- 172.21.15.117:0/2406470513 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b981000c0 msgr2=0x7f7b981004e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.695+0000 7f7b9e600640 1 --2- 172.21.15.117:0/2406470513 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b981000c0 0x7f7b981004e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7b88009900 tx=0x7f7b8802bc30 comp rx=0 tx=0).stop 2024-03-19T03:32:47.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.696+0000 7f7b9e600640 1 -- 172.21.15.117:0/2406470513 shutdown_connections 2024-03-19T03:32:47.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.696+0000 7f7b9e600640 1 --2- 172.21.15.117:0/2406470513 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b981013c0 0x7f7b98101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.696+0000 7f7b9e600640 1 --2- 172.21.15.117:0/2406470513 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b98100a20 0x7f7b98100e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.696+0000 7f7b9e600640 1 --2- 172.21.15.117:0/2406470513 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b981000c0 0x7f7b981004e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.696+0000 7f7b9e600640 1 -- 172.21.15.117:0/2406470513 >> 172.21.15.117:0/2406470513 conn(0x7f7b980fbca0 msgr2=0x7f7b980fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:47.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.696+0000 7f7b9e600640 1 -- 172.21.15.117:0/2406470513 shutdown_connections 2024-03-19T03:32:47.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.696+0000 7f7b9e600640 1 -- 172.21.15.117:0/2406470513 wait complete. 2024-03-19T03:32:47.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.697+0000 7f7b9e600640 1 Processor -- start 2024-03-19T03:32:47.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.697+0000 7f7b9e600640 1 -- start start 2024-03-19T03:32:47.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.697+0000 7f7b9e600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b981000c0 0x7f7b98199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.697+0000 7f7b9e600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b98100a20 0x7f7b9819a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.697+0000 7f7b9e600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b981013c0 0x7f7b981a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.697+0000 7f7b9e600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9810f6f0 con 0x7f7b98100a20 2024-03-19T03:32:47.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.697+0000 7f7b9e600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9810f570 con 0x7f7b981013c0 2024-03-19T03:32:47.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.697+0000 7f7b9e600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9810f870 con 0x7f7b981000c0 2024-03-19T03:32:47.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.697+0000 7f7b97e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b981013c0 0x7f7b981a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.697+0000 7f7b97400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b981000c0 0x7f7b98199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.697+0000 7f7b8fe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b98100a20 0x7f7b9819a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.697+0000 7f7b97400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b981000c0 0x7f7b98199d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:39034/0 (socket says 172.21.15.117:39034) 2024-03-19T03:32:47.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.697+0000 7f7b97400640 1 -- 172.21.15.117:0/4282366007 learned_addr learned my addr 172.21.15.117:0/4282366007 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:47.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.698+0000 7f7b97e00640 1 -- 172.21.15.117:0/4282366007 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b981000c0 msgr2=0x7f7b98199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.698+0000 7f7b97e00640 1 --2- 172.21.15.117:0/4282366007 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b981000c0 0x7f7b98199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.698+0000 7f7b97e00640 1 -- 172.21.15.117:0/4282366007 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b98100a20 msgr2=0x7f7b9819a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.698+0000 7f7b97e00640 1 --2- 172.21.15.117:0/4282366007 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b98100a20 0x7f7b9819a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.698+0000 7f7b97e00640 1 -- 172.21.15.117:0/4282366007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b981a1930 con 0x7f7b981013c0 2024-03-19T03:32:47.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.698+0000 7f7b97e00640 1 --2- 172.21.15.117:0/4282366007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b981013c0 0x7f7b981a1290 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7b9000e940 tx=0x7f7b9000ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:47.701 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.698+0000 7f7b94c00640 1 -- 172.21.15.117:0/4282366007 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b90004d20 con 0x7f7b981013c0 2024-03-19T03:32:47.701 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.698+0000 7f7b94c00640 1 -- 172.21.15.117:0/4282366007 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b90005020 con 0x7f7b981013c0 2024-03-19T03:32:47.701 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.698+0000 7f7b94c00640 1 -- 172.21.15.117:0/4282366007 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b900116b0 con 0x7f7b981013c0 2024-03-19T03:32:47.701 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.698+0000 7f7b9e600640 1 -- 172.21.15.117:0/4282366007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b981a1c20 con 0x7f7b981013c0 2024-03-19T03:32:47.701 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.698+0000 7f7b9e600640 1 -- 172.21.15.117:0/4282366007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b981a2120 con 0x7f7b981013c0 2024-03-19T03:32:47.702 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.702+0000 7f7ba13d9740 1 Processor -- start 2024-03-19T03:32:47.702 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.702+0000 7f7b94c00640 1 -- 172.21.15.117:0/4282366007 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f7b900054d0 con 0x7f7b981013c0 2024-03-19T03:32:47.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.702+0000 7f7ba13d9740 1 -- start start 2024-03-19T03:32:47.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.702+0000 7f7ba13d9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e266ee83f0 0x55e266ee8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.702+0000 7f7ba13d9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee8de0 0x55e266eed8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.702+0000 7f7ba13d9740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e266eedf50 0x55e266ef03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.702+0000 7f7ba13d9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e266e4cd00 con 0x55e266ee83f0 2024-03-19T03:32:47.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.702+0000 7f7ba13d9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55e266df5b00 con 0x55e266ee8de0 2024-03-19T03:32:47.703 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.702+0000 7f7ba13d9740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x55e266e48da0 con 0x55e266eedf50 2024-03-19T03:32:47.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.703+0000 7f7b97400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee8de0 0x55e266eed8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.703+0000 7f7b8fe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e266ee83f0 0x55e266ee8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.703+0000 7f7b97400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee8de0 0x55e266eed8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:58046/0 (socket says 172.21.15.117:58046) 2024-03-19T03:32:47.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.703+0000 7f7b97400640 1 -- 172.21.15.117:0/3100604543 learned_addr learned my addr 172.21.15.117:0/3100604543 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:47.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.703+0000 7f7b8fe00640 1 --2- 172.21.15.117:0/3100604543 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e266eedf50 0x55e266ef03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.703+0000 7f7b97400640 1 -- 172.21.15.117:0/3100604543 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e266eedf50 msgr2=0x55e266ef03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:47.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.703+0000 7f7b97400640 1 --2- 172.21.15.117:0/3100604543 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e266eedf50 0x55e266ef03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.703+0000 7f7b97400640 1 -- 172.21.15.117:0/3100604543 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e266ee83f0 msgr2=0x55e266ee8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:47.706 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.703+0000 7f7b97400640 1 --2- 172.21.15.117:0/3100604543 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e266ee83f0 0x55e266ee8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.703+0000 7f7b97400640 1 -- 172.21.15.117:0/3100604543 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e266b612d0 con 0x55e266ee8de0 2024-03-19T03:32:47.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.703+0000 7f7b8fe00640 1 --2- 172.21.15.117:0/3100604543 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e266eedf50 0x55e266ef03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T03:32:47.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.703+0000 7f7b97400640 1 --2- 172.21.15.117:0/3100604543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee8de0 0x55e266eed8e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7b8802c170 tx=0x7f7b880095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=caf6ce01f6c5f342 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:47.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.704+0000 7f7b8cc00640 1 -- 172.21.15.117:0/3100604543 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b88045070 con 0x55e266ee8de0 2024-03-19T03:32:47.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.704+0000 7f7b8cc00640 1 -- 172.21.15.117:0/3100604543 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b88004040 con 0x55e266ee8de0 2024-03-19T03:32:47.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.704+0000 7f7b8cc00640 1 -- 172.21.15.117:0/3100604543 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b88004ef0 con 0x55e266ee8de0 2024-03-19T03:32:47.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.704+0000 7f7b94c00640 1 --2- 172.21.15.117:0/4282366007 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7b7c0a2880 0x7f7b7c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.704+0000 7f7ba13d9740 1 -- 172.21.15.117:0/3100604543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee8de0 msgr2=0x55e266eed8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.704+0000 7f7ba13d9740 1 --2- 172.21.15.117:0/3100604543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee8de0 0x55e266eed8e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7b8802c170 tx=0x7f7b880095e0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.704+0000 7f7b94c00640 1 -- 172.21.15.117:0/4282366007 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7b900cf220 con 0x7f7b981013c0 2024-03-19T03:32:47.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.704+0000 7f7b8fe00640 1 --2- 172.21.15.117:0/4282366007 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7b7c0a2880 0x7f7b7c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.705+0000 7f7ba13d9740 1 -- 172.21.15.117:0/3100604543 shutdown_connections 2024-03-19T03:32:47.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.705+0000 7f7ba13d9740 1 --2- 172.21.15.117:0/3100604543 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e266eedf50 0x55e266ef03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.705+0000 7f7ba13d9740 1 --2- 172.21.15.117:0/3100604543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee8de0 0x55e266eed8e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.707 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.705+0000 7f7ba13d9740 1 --2- 172.21.15.117:0/3100604543 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e266ee83f0 0x55e266ee8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.705+0000 7f7ba13d9740 1 -- 172.21.15.117:0/3100604543 >> 172.21.15.117:0/3100604543 conn(0x55e266ee3960 msgr2=0x55e266ee3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:47.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.705+0000 7f7ba13d9740 1 -- 172.21.15.117:0/3100604543 shutdown_connections 2024-03-19T03:32:47.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.705+0000 7f7ba13d9740 1 -- 172.21.15.117:0/3100604543 wait complete. 2024-03-19T03:32:47.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.705+0000 7f7ba13d9740 1 -- start start 2024-03-19T03:32:47.708 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.705+0000 7f7b8fe00640 1 --2- 172.21.15.117:0/4282366007 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7b7c0a2880 0x7f7b7c0a4d40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x55e266ee9640 tx=0x7f7b80009370 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:47.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.705+0000 7f7ba13d9740 1 Processor -- start 2024-03-19T03:32:47.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.706+0000 7f7ba13d9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e266ee83f0 0x55e266ef2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.706+0000 7f7ba13d9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee8de0 0x55e266ef2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.706+0000 7f7ba13d9740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e266eedf50 0x55e266ef9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.706+0000 7f7ba13d9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e266ef0ea0 con 0x55e266ee83f0 2024-03-19T03:32:47.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.706+0000 7f7ba13d9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55e266e08ed0 con 0x55e266ee8de0 2024-03-19T03:32:47.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.706+0000 7f7ba13d9740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x55e266ef1020 con 0x55e266eedf50 2024-03-19T03:32:47.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.706+0000 7f7ba13d9740 10 client.0 _ll_register_callbacks cb 0x55e266e06c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:32:47.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.706+0000 7f7b8fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee8de0 0x55e266ef2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.706+0000 7f7b97400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e266ee83f0 0x55e266ef2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.706+0000 7f7b8fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee8de0 0x55e266ef2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:58054/0 (socket says 172.21.15.117:58054) 2024-03-19T03:32:47.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.706+0000 7f7b8fe00640 1 -- 172.21.15.117:0/3630272719 learned_addr learned my addr 172.21.15.117:0/3630272719 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:47.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.706+0000 7f7b97e00640 1 --2- 172.21.15.117:0/3630272719 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e266eedf50 0x55e266ef9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.707+0000 7f7b8fe00640 1 -- 172.21.15.117:0/3630272719 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e266eedf50 msgr2=0x55e266ef9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:47.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.707+0000 7f7b8fe00640 1 --2- 172.21.15.117:0/3630272719 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e266eedf50 0x55e266ef9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.707+0000 7f7b8fe00640 1 -- 172.21.15.117:0/3630272719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e266ee83f0 msgr2=0x55e266ef2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:47.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.707+0000 7f7b8fe00640 1 --2- 172.21.15.117:0/3630272719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e266ee83f0 0x55e266ef2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.707+0000 7f7b8fe00640 1 -- 172.21.15.117:0/3630272719 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e266c2b6b0 con 0x55e266ee8de0 2024-03-19T03:32:47.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.707+0000 7f7b97400640 1 --2- 172.21.15.117:0/3630272719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e266ee83f0 0x55e266ef2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:32:47.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.707+0000 7f7b8fe00640 1 --2- 172.21.15.117:0/3630272719 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee8de0 0x55e266ef2dc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7b800045c0 tx=0x7f7b80004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a649fe12ee69e241 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:47.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.707+0000 7f7b85600640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-19T03:32:47.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.707+0000 7f7b85600640 1 -- 172.21.15.117:0/3630272719 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8002e060 con 0x55e266ee8de0 2024-03-19T03:32:47.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.707+0000 7f7b85600640 1 -- 172.21.15.117:0/3630272719 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7b80019070 con 0x55e266ee8de0 2024-03-19T03:32:47.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.707+0000 7f7ba13d9740 1 -- 172.21.15.117:0/3630272719 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b88004040 con 0x55e266ee8de0 2024-03-19T03:32:47.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.708+0000 7f7b85600640 1 -- 172.21.15.117:0/3630272719 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7b80035050 con 0x55e266ee8de0 2024-03-19T03:32:47.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.708+0000 7f7ba13d9740 10 client.25238 fetch_fsmap learned FSMap version 28 2024-03-19T03:32:47.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.708+0000 7f7ba13d9740 1 -- 172.21.15.117:0/3630272719 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e266efd150 con 0x55e266ee8de0 2024-03-19T03:32:47.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.708+0000 7f7b85600640 1 -- 172.21.15.117:0/3630272719 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b88004040 con 0x55e266ee8de0 2024-03-19T03:32:47.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.708+0000 7f7b85600640 1 -- 172.21.15.117:0/3630272719 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7b8001f990 con 0x55e266ee8de0 2024-03-19T03:32:47.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.708+0000 7f7ba13d9740 10 client.25238 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T03:32:47.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.708+0000 7f7ba13d9740 10 client.25238 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-19T03:32:47.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.708+0000 7f7ba13d9740 20 client.25238 populate_metadata read hostname 'smithi117' 2024-03-19T03:32:47.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.708+0000 7f7ba13d9740 1 --2- 172.21.15.117:0/3630272719 >> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] conn(0x55e266eff890 0x55e266f00d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.708+0000 7f7ba13d9740 4 client.25238 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:32:47.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.708+0000 7f7ba13d9740 1 -- 172.21.15.117:0/3630272719 --> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e266e5a240 con 0x55e266eff890 2024-03-19T03:32:47.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.708+0000 7f7b97400640 1 --2- 172.21.15.117:0/3630272719 >> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] conn(0x55e266eff890 0x55e266f00d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.709+0000 7f7b97400640 1 --2- 172.21.15.117:0/3630272719 >> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] conn(0x55e266eff890 0x55e266f00d00 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be2aa9becd349e77 server_cookie=b1e84ce64c82f1cf in_seq=0 out_seq=0 2024-03-19T03:32:47.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.709+0000 7f7b85600640 10 client.25238 ms_handle_connect on v2:172.21.15.119:6838/2383354833 2024-03-19T03:32:47.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.710+0000 7f7b85600640 1 -- 172.21.15.117:0/3630272719 <== mds.0 v2:172.21.15.119:6838/2383354833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e266e5a240 con 0x55e266eff890 2024-03-19T03:32:47.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.710+0000 7f7b85600640 10 client.25238 handle_command_reply: tid=0 2024-03-19T03:32:47.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.710+0000 7f7ba13d9740 1 client.25238 shutdown 2024-03-19T03:32:47.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.710+0000 7f7ba13d9740 1 -- 172.21.15.117:0/3630272719 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee8de0 msgr2=0x55e266ef2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.710+0000 7f7ba13d9740 1 --2- 172.21.15.117:0/3630272719 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee8de0 0x55e266ef2dc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7b800045c0 tx=0x7f7b80004730 comp rx=0 tx=0).stop 2024-03-19T03:32:47.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.710+0000 7f7ba13d9740 1 -- 172.21.15.117:0/3630272719 shutdown_connections 2024-03-19T03:32:47.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.710+0000 7f7ba13d9740 1 --2- 172.21.15.117:0/3630272719 >> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] conn(0x55e266eff890 0x55e266f00d00 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.710+0000 7f7ba13d9740 1 --2- 172.21.15.117:0/3630272719 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e266eedf50 0x55e266ef9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.710+0000 7f7ba13d9740 1 --2- 172.21.15.117:0/3630272719 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee8de0 0x55e266ef2dc0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.710+0000 7f7b85600640 0 client.25238 ms_handle_reset on v2:172.21.15.119:6838/2383354833 2024-03-19T03:32:47.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.710+0000 7f7ba13d9740 1 --2- 172.21.15.117:0/3630272719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e266ee83f0 0x55e266ef2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.710+0000 7f7ba13d9740 1 -- 172.21.15.117:0/3630272719 >> 172.21.15.117:0/3630272719 conn(0x55e266ee3960 msgr2=0x55e266ee2aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:47.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.710+0000 7f7ba13d9740 1 -- 172.21.15.117:0/3630272719 shutdown_connections 2024-03-19T03:32:47.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.710+0000 7f7ba13d9740 1 -- 172.21.15.117:0/3630272719 wait complete. 2024-03-19T03:32:47.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.710+0000 7f7ba13d9740 20 client.25238 trim_cache size 0 max 16384 2024-03-19T03:32:47.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.739+0000 7f7ba13d9740 1 Processor -- start 2024-03-19T03:32:47.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.740+0000 7f7ba13d9740 1 -- start start 2024-03-19T03:32:47.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.740+0000 7f7ba13d9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e266ee83f0 0x55e266ef2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.740+0000 7f7ba13d9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee8de0 0x55e266ef9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.740+0000 7f7ba13d9740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e266eedf50 0x55e266ef2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.740+0000 7f7ba13d9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e266eff040 con 0x55e266ee83f0 2024-03-19T03:32:47.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.740+0000 7f7ba13d9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55e266eff710 con 0x55e266ee8de0 2024-03-19T03:32:47.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.740+0000 7f7ba13d9740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x55e266effde0 con 0x55e266eedf50 2024-03-19T03:32:47.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.740+0000 7f7b97400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e266ee83f0 0x55e266ef2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.740+0000 7f7b97e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e266eedf50 0x55e266ef2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.740+0000 7f7b97e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e266eedf50 0x55e266ef2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:39056/0 (socket says 172.21.15.117:39056) 2024-03-19T03:32:47.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.740+0000 7f7b97400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e266ee83f0 0x55e266ef2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:55810/0 (socket says 172.21.15.117:55810) 2024-03-19T03:32:47.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.740+0000 7f7b97e00640 1 -- 172.21.15.117:0/2114463261 learned_addr learned my addr 172.21.15.117:0/2114463261 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:47.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.740+0000 7f7b8fe00640 1 --2- 172.21.15.117:0/2114463261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee8de0 0x55e266ef9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.741+0000 7f7b97e00640 1 -- 172.21.15.117:0/2114463261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee8de0 msgr2=0x55e266ef9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:47.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.741+0000 7f7b97e00640 1 --2- 172.21.15.117:0/2114463261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee8de0 0x55e266ef9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.741+0000 7f7b97e00640 1 -- 172.21.15.117:0/2114463261 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e266ee83f0 msgr2=0x55e266ef2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:47.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.741+0000 7f7b97e00640 1 --2- 172.21.15.117:0/2114463261 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e266ee83f0 0x55e266ef2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.741+0000 7f7b97e00640 1 -- 172.21.15.117:0/2114463261 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e266ee46d0 con 0x55e266eedf50 2024-03-19T03:32:47.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.741+0000 7f7b97400640 1 --2- 172.21.15.117:0/2114463261 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e266ee83f0 0x55e266ef2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:32:47.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.741+0000 7f7b97e00640 1 --2- 172.21.15.117:0/2114463261 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e266eedf50 0x55e266ef2d90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7b900cc980 tx=0x7f7b900058d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bfa7894621b88d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:47.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.741+0000 7f7b87e00640 1 -- 172.21.15.117:0/2114463261 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b9001f040 con 0x55e266eedf50 2024-03-19T03:32:47.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.741+0000 7f7b87e00640 1 -- 172.21.15.117:0/2114463261 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b9009b880 con 0x55e266eedf50 2024-03-19T03:32:47.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.741+0000 7f7b87e00640 1 -- 172.21.15.117:0/2114463261 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b900860b0 con 0x55e266eedf50 2024-03-19T03:32:47.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.742+0000 7f7ba13d9740 1 -- 172.21.15.117:0/2114463261 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e266eedf50 msgr2=0x55e266ef2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.742+0000 7f7ba13d9740 1 --2- 172.21.15.117:0/2114463261 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e266eedf50 0x55e266ef2d90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7b900cc980 tx=0x7f7b900058d0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.742+0000 7f7ba13d9740 1 -- 172.21.15.117:0/2114463261 shutdown_connections 2024-03-19T03:32:47.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.742+0000 7f7ba13d9740 1 --2- 172.21.15.117:0/2114463261 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e266eedf50 0x55e266ef2d90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.742+0000 7f7ba13d9740 1 --2- 172.21.15.117:0/2114463261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee8de0 0x55e266ef9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.742+0000 7f7ba13d9740 1 --2- 172.21.15.117:0/2114463261 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e266ee83f0 0x55e266ef2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.742+0000 7f7ba13d9740 1 -- 172.21.15.117:0/2114463261 >> 172.21.15.117:0/2114463261 conn(0x55e266ee3960 msgr2=0x55e266eea2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:47.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.742+0000 7f7ba13d9740 1 -- 172.21.15.117:0/2114463261 shutdown_connections 2024-03-19T03:32:47.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.742+0000 7f7ba13d9740 1 -- 172.21.15.117:0/2114463261 wait complete. 2024-03-19T03:32:47.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.743+0000 7f7ba13d9740 1 -- start start 2024-03-19T03:32:47.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.743+0000 7f7ba13d9740 1 Processor -- start 2024-03-19T03:32:47.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.743+0000 7f7ba13d9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee83f0 0x55e266ef9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.747 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.743+0000 7f7ba13d9740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e266ee8de0 0x55e266ef9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.743+0000 7f7ba13d9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e266eedf50 0x55e266ef60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.743+0000 7f7ba13d9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e266f00b80 con 0x55e266eedf50 2024-03-19T03:32:47.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.743+0000 7f7ba13d9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55e266f004b0 con 0x55e266ee83f0 2024-03-19T03:32:47.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.743+0000 7f7ba13d9740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x55e266f01250 con 0x55e266ee8de0 2024-03-19T03:32:47.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.744+0000 7f7ba13d9740 10 client.0 _ll_register_callbacks cb 0x7f7b80012e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:32:47.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.744+0000 7f7b97400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee83f0 0x55e266ef9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.744+0000 7f7b97400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee83f0 0x55e266ef9570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:58078/0 (socket says 172.21.15.117:58078) 2024-03-19T03:32:47.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.744+0000 7f7b97400640 1 -- 172.21.15.117:0/2713100365 learned_addr learned my addr 172.21.15.117:0/2713100365 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:47.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.744+0000 7f7b97e00640 1 --2- 172.21.15.117:0/2713100365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e266eedf50 0x55e266ef60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.744+0000 7f7b8fe00640 1 --2- 172.21.15.117:0/2713100365 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e266ee8de0 0x55e266ef9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.744+0000 7f7b97400640 1 -- 172.21.15.117:0/2713100365 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e266ee8de0 msgr2=0x55e266ef9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:47.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.744+0000 7f7b97400640 1 --2- 172.21.15.117:0/2713100365 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e266ee8de0 0x55e266ef9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.744+0000 7f7b97400640 1 -- 172.21.15.117:0/2713100365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e266eedf50 msgr2=0x55e266ef60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:32:47.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.744+0000 7f7b97400640 1 --2- 172.21.15.117:0/2713100365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e266eedf50 0x55e266ef60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.744+0000 7f7b97400640 1 -- 172.21.15.117:0/2713100365 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e266ef66c0 con 0x55e266ee83f0 2024-03-19T03:32:47.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.744+0000 7f7b8fe00640 1 --2- 172.21.15.117:0/2713100365 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e266ee8de0 0x55e266ef9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T03:32:47.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.744+0000 7f7b97e00640 1 --2- 172.21.15.117:0/2713100365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e266eedf50 0x55e266ef60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:32:47.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.744+0000 7f7b97400640 1 --2- 172.21.15.117:0/2713100365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee83f0 0x55e266ef9570 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7b8802c240 tx=0x7f7b88009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=419c304c63de23dd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:47.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.744+0000 7f7b85600640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-19T03:32:47.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.744+0000 7f7b85600640 1 -- 172.21.15.117:0/2713100365 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b88059060 con 0x55e266ee83f0 2024-03-19T03:32:47.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.745+0000 7f7b85600640 1 -- 172.21.15.117:0/2713100365 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7b880042f0 con 0x55e266ee83f0 2024-03-19T03:32:47.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.745+0000 7f7ba13d9740 1 -- 172.21.15.117:0/2713100365 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b9009b880 con 0x55e266ee83f0 2024-03-19T03:32:47.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.745+0000 7f7b85600640 1 -- 172.21.15.117:0/2713100365 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7b88060050 con 0x55e266ee83f0 2024-03-19T03:32:47.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.745+0000 7f7ba13d9740 10 client.25244 fetch_fsmap learned FSMap version 28 2024-03-19T03:32:47.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.745+0000 7f7ba13d9740 1 -- 172.21.15.117:0/2713100365 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e266f66160 con 0x55e266ee83f0 2024-03-19T03:32:47.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.745+0000 7f7b85600640 1 -- 172.21.15.117:0/2713100365 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b9009b880 con 0x55e266ee83f0 2024-03-19T03:32:47.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.746+0000 7f7b85600640 1 -- 172.21.15.117:0/2713100365 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7b880542e0 con 0x55e266ee83f0 2024-03-19T03:32:47.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.746+0000 7f7ba13d9740 10 client.25244 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T03:32:47.749 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.746+0000 7f7ba13d9740 10 client.25244 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-19T03:32:47.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.746+0000 7f7ba13d9740 20 client.25244 populate_metadata read hostname 'smithi117' 2024-03-19T03:32:47.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.746+0000 7f7ba13d9740 1 --2- 172.21.15.117:0/2713100365 >> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] conn(0x55e266f6a9a0 0x55e266f696b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:47.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.746+0000 7f7ba13d9740 4 client.25244 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T03:32:47.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.746+0000 7f7ba13d9740 1 -- 172.21.15.117:0/2713100365 --> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] -- command(tid 0: {"prefix": "status"}) -- 0x55e266e54590 con 0x55e266f6a9a0 2024-03-19T03:32:47.750 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.746+0000 7f7b8fe00640 1 --2- 172.21.15.117:0/2713100365 >> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] conn(0x55e266f6a9a0 0x55e266f696b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:47.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.747+0000 7f7b8fe00640 1 --2- 172.21.15.117:0/2713100365 >> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] conn(0x55e266f6a9a0 0x55e266f696b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ccf2de1b86003a9 server_cookie=d3a5c4cc81a2f233 in_seq=0 out_seq=0 2024-03-19T03:32:47.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.747+0000 7f7b85600640 10 client.25244 ms_handle_connect on v2:172.21.15.119:6838/2383354833 2024-03-19T03:32:47.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.747+0000 7f7b85600640 1 -- 172.21.15.117:0/2713100365 <== mds.0 v2:172.21.15.119:6838/2383354833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55e266e54590 con 0x55e266f6a9a0 2024-03-19T03:32:47.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.747+0000 7f7b85600640 10 client.25244 handle_command_reply: tid=0 2024-03-19T03:32:47.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.747+0000 7f7ba13d9740 1 client.25244 shutdown 2024-03-19T03:32:47.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.747+0000 7f7ba13d9740 1 -- 172.21.15.117:0/2713100365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee83f0 msgr2=0x55e266ef9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.747+0000 7f7ba13d9740 1 --2- 172.21.15.117:0/2713100365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee83f0 0x55e266ef9570 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7b8802c240 tx=0x7f7b88009260 comp rx=0 tx=0).stop 2024-03-19T03:32:47.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.747+0000 7f7ba13d9740 1 -- 172.21.15.117:0/2713100365 shutdown_connections 2024-03-19T03:32:47.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.747+0000 7f7ba13d9740 1 --2- 172.21.15.117:0/2713100365 >> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] conn(0x55e266f6a9a0 0x55e266f696b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.747+0000 7f7ba13d9740 1 --2- 172.21.15.117:0/2713100365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e266eedf50 0x55e266ef60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.747+0000 7f7ba13d9740 1 --2- 172.21.15.117:0/2713100365 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e266ee8de0 0x55e266ef9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.747+0000 7f7ba13d9740 1 --2- 172.21.15.117:0/2713100365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55e266ee83f0 0x55e266ef9570 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.747+0000 7f7ba13d9740 1 -- 172.21.15.117:0/2713100365 >> 172.21.15.117:0/2713100365 conn(0x55e266ee3960 msgr2=0x55e266eea2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:47.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.747+0000 7f7b85600640 0 client.25244 ms_handle_reset on v2:172.21.15.119:6838/2383354833 2024-03-19T03:32:47.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.748+0000 7f7ba13d9740 1 -- 172.21.15.117:0/2713100365 shutdown_connections 2024-03-19T03:32:47.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.748+0000 7f7ba13d9740 1 -- 172.21.15.117:0/2713100365 wait complete. 2024-03-19T03:32:47.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.748+0000 7f7ba13d9740 20 client.25244 trim_cache size 0 max 16384 2024-03-19T03:32:47.752 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-19T03:32:47.753 INFO:teuthology.orchestra.run.smithi117.stdout: "cluster_fsid": "a3a82456-e59f-11ee-95c9-87774f69a715", 2024-03-19T03:32:47.753 INFO:teuthology.orchestra.run.smithi117.stdout: "whoami": 0, 2024-03-19T03:32:47.753 INFO:teuthology.orchestra.run.smithi117.stdout: "id": 24599, 2024-03-19T03:32:47.753 INFO:teuthology.orchestra.run.smithi117.stdout: "want_state": "up:active", 2024-03-19T03:32:47.753 INFO:teuthology.orchestra.run.smithi117.stdout: "state": "up:active", 2024-03-19T03:32:47.753 INFO:teuthology.orchestra.run.smithi117.stdout: "fs_name": "cephfs", 2024-03-19T03:32:47.753 INFO:teuthology.orchestra.run.smithi117.stdout: "rank_uptime": 132.08045464700001, 2024-03-19T03:32:47.753 INFO:teuthology.orchestra.run.smithi117.stdout: "mdsmap_epoch": 28, 2024-03-19T03:32:47.753 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch": 93, 2024-03-19T03:32:47.753 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch_barrier": 89, 2024-03-19T03:32:47.753 INFO:teuthology.orchestra.run.smithi117.stdout: "uptime": 139.424260758 2024-03-19T03:32:47.753 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-19T03:32:47.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.749+0000 7f7b9e600640 1 -- 172.21.15.117:0/4282366007 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7b7c0a2880 msgr2=0x7f7b7c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.749+0000 7f7b9e600640 1 --2- 172.21.15.117:0/4282366007 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7b7c0a2880 0x7f7b7c0a4d40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x55e266ee9640 tx=0x7f7b80009370 comp rx=0 tx=0).stop 2024-03-19T03:32:47.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.749+0000 7f7b9e600640 1 -- 172.21.15.117:0/4282366007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b981013c0 msgr2=0x7f7b981a1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:47.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.749+0000 7f7b9e600640 1 --2- 172.21.15.117:0/4282366007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b981013c0 0x7f7b981a1290 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7b9000e940 tx=0x7f7b9000ee00 comp rx=0 tx=0).stop 2024-03-19T03:32:47.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.749+0000 7f7b9e600640 1 -- 172.21.15.117:0/4282366007 shutdown_connections 2024-03-19T03:32:47.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.749+0000 7f7b9e600640 1 --2- 172.21.15.117:0/4282366007 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7b7c0a2880 0x7f7b7c0a4d40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.749+0000 7f7b9e600640 1 --2- 172.21.15.117:0/4282366007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b981013c0 0x7f7b981a1290 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.749+0000 7f7b9e600640 1 --2- 172.21.15.117:0/4282366007 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b98100a20 0x7f7b9819a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.749+0000 7f7b9e600640 1 --2- 172.21.15.117:0/4282366007 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b981000c0 0x7f7b98199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:47.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.749+0000 7f7b9e600640 1 -- 172.21.15.117:0/4282366007 >> 172.21.15.117:0/4282366007 conn(0x7f7b980fbca0 msgr2=0x7f7b980fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:47.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.749+0000 7f7b9e600640 1 -- 172.21.15.117:0/4282366007 shutdown_connections 2024-03-19T03:32:47.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:47.749+0000 7f7b9e600640 1 -- 172.21.15.117:0/4282366007 wait complete. 2024-03-19T03:32:47.768 INFO:tasks.cephadm:Setup complete, yielding 2024-03-19T03:32:47.769 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-19T03:32:47.778 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi117.front.sepia.ceph.com 2024-03-19T03:32:47.778 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- bash -c 'ceph orch status' 2024-03-19T03:32:48.255 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:32:49.433 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:49 smithi117 ceph-mon[31838]: pgmap v388: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 461 B/s wr, 8 op/s; 7.1 KiB/s, 1 keys/s, 2 objects/s recovering 2024-03-19T03:32:49.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:49 smithi119 ceph-mon[66861]: pgmap v388: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 461 B/s wr, 8 op/s; 7.1 KiB/s, 1 keys/s, 2 objects/s recovering 2024-03-19T03:32:49.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:49 smithi178 ceph-mon[67578]: pgmap v388: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 461 B/s wr, 8 op/s; 7.1 KiB/s, 1 keys/s, 2 objects/s recovering 2024-03-19T03:32:50.439 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.437+0000 7f292da00700 1 -- 172.21.15.117:0/2396144895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2928102e30 msgr2=0x7f2928105220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:50.439 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.437+0000 7f292da00700 1 --2- 172.21.15.117:0/2396144895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2928102e30 0x7f2928105220 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2918009990 tx=0x7f2918009ca0 comp rx=0 tx=0).stop 2024-03-19T03:32:50.439 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.438+0000 7f292da00700 1 -- 172.21.15.117:0/2396144895 shutdown_connections 2024-03-19T03:32:50.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.438+0000 7f292da00700 1 --2- 172.21.15.117:0/2396144895 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2928108090 0x7f292810a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:50.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.438+0000 7f292da00700 1 --2- 172.21.15.117:0/2396144895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2928102e30 0x7f2928105220 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:50.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.438+0000 7f292da00700 1 --2- 172.21.15.117:0/2396144895 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2928105760 0x7f2928107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:50.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.438+0000 7f292da00700 1 -- 172.21.15.117:0/2396144895 >> 172.21.15.117:0/2396144895 conn(0x7f29280fc7e0 msgr2=0x7f29280fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:50.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.438+0000 7f292da00700 1 -- 172.21.15.117:0/2396144895 shutdown_connections 2024-03-19T03:32:50.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.438+0000 7f292da00700 1 -- 172.21.15.117:0/2396144895 wait complete. 2024-03-19T03:32:50.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.439+0000 7f292da00700 1 Processor -- start 2024-03-19T03:32:50.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.439+0000 7f292da00700 1 -- start start 2024-03-19T03:32:50.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.440+0000 7f292da00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2928102e30 0x7f2928070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:50.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.440+0000 7f292da00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2928105760 0x7f2928071220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:50.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.440+0000 7f292da00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2928108090 0x7f29281a8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:50.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.440+0000 7f292da00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f292810dfa0 con 0x7f2928108090 2024-03-19T03:32:50.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.440+0000 7f292da00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f292810de20 con 0x7f2928105760 2024-03-19T03:32:50.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.440+0000 7f292da00700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f292810e120 con 0x7f2928102e30 2024-03-19T03:32:50.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.440+0000 7f2926000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2928105760 0x7f2928071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:50.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.440+0000 7f2926000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2928105760 0x7f2928071220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:58110/0 (socket says 172.21.15.117:58110) 2024-03-19T03:32:50.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.440+0000 7f2926000700 1 -- 172.21.15.117:0/2363951125 learned_addr learned my addr 172.21.15.117:0/2363951125 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:50.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.440+0000 7f2926a00700 1 --2- 172.21.15.117:0/2363951125 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2928102e30 0x7f2928070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:50.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.440+0000 7f2927400700 1 --2- 172.21.15.117:0/2363951125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2928108090 0x7f29281a8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:50.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.441+0000 7f2926000700 1 -- 172.21.15.117:0/2363951125 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2928102e30 msgr2=0x7f2928070ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:50.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.441+0000 7f2926000700 1 --2- 172.21.15.117:0/2363951125 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2928102e30 0x7f2928070ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:50.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.441+0000 7f2926000700 1 -- 172.21.15.117:0/2363951125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2928108090 msgr2=0x7f29281a8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:50.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.441+0000 7f2926000700 1 --2- 172.21.15.117:0/2363951125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2928108090 0x7f29281a8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:50.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.441+0000 7f2926000700 1 -- 172.21.15.117:0/2363951125 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29281a8bd0 con 0x7f2928105760 2024-03-19T03:32:50.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.441+0000 7f2927400700 1 --2- 172.21.15.117:0/2363951125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2928108090 0x7f29281a8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T03:32:50.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.441+0000 7f2926000700 1 --2- 172.21.15.117:0/2363951125 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2928105760 0x7f2928071220 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f291400e9a0 tx=0x7f291400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:50.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.441+0000 7f291f400700 1 -- 172.21.15.117:0/2363951125 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f291400cbd0 con 0x7f2928105760 2024-03-19T03:32:50.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.441+0000 7f291f400700 1 -- 172.21.15.117:0/2363951125 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f291400cd70 con 0x7f2928105760 2024-03-19T03:32:50.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.441+0000 7f291f400700 1 -- 172.21.15.117:0/2363951125 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2914018710 con 0x7f2928105760 2024-03-19T03:32:50.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.441+0000 7f292da00700 1 -- 172.21.15.117:0/2363951125 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29281a8da0 con 0x7f2928105760 2024-03-19T03:32:50.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.441+0000 7f292da00700 1 -- 172.21.15.117:0/2363951125 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29281a9200 con 0x7f2928105760 2024-03-19T03:32:50.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.443+0000 7f292da00700 1 -- 172.21.15.117:0/2363951125 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2928068fd0 con 0x7f2928105760 2024-03-19T03:32:50.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.445+0000 7f291f400700 1 -- 172.21.15.117:0/2363951125 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f2914010430 con 0x7f2928105760 2024-03-19T03:32:50.447 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.447+0000 7f291f400700 1 --2- 172.21.15.117:0/2363951125 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f290c0a2910 0x7f290c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:50.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.447+0000 7f291f400700 1 -- 172.21.15.117:0/2363951125 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f29140cf090 con 0x7f2928105760 2024-03-19T03:32:50.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.447+0000 7f2926a00700 1 --2- 172.21.15.117:0/2363951125 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f290c0a2910 0x7f290c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:50.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.448+0000 7f2926a00700 1 --2- 172.21.15.117:0/2363951125 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f290c0a2910 0x7f290c0a4dd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f291800b7b0 tx=0x7f291800b880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:50.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.450+0000 7f291f400700 1 -- 172.21.15.117:0/2363951125 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f291409bd40 con 0x7f2928105760 2024-03-19T03:32:50.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.807+0000 7f292da00700 1 -- 172.21.15.117:0/2363951125 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f2928062750 con 0x7f290c0a2910 2024-03-19T03:32:50.809 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.809+0000 7f291f400700 1 -- 172.21.15.117:0/2363951125 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f2928062750 con 0x7f290c0a2910 2024-03-19T03:32:50.810 INFO:teuthology.orchestra.run.smithi117.stdout:Backend: cephadm 2024-03-19T03:32:50.810 INFO:teuthology.orchestra.run.smithi117.stdout:Available: Yes 2024-03-19T03:32:50.810 INFO:teuthology.orchestra.run.smithi117.stdout:Paused: No 2024-03-19T03:32:50.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.812+0000 7f292da00700 1 -- 172.21.15.117:0/2363951125 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f290c0a2910 msgr2=0x7f290c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:50.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.812+0000 7f292da00700 1 --2- 172.21.15.117:0/2363951125 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f290c0a2910 0x7f290c0a4dd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f291800b7b0 tx=0x7f291800b880 comp rx=0 tx=0).stop 2024-03-19T03:32:50.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.812+0000 7f292da00700 1 -- 172.21.15.117:0/2363951125 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2928105760 msgr2=0x7f2928071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:50.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.812+0000 7f292da00700 1 --2- 172.21.15.117:0/2363951125 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2928105760 0x7f2928071220 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f291400e9a0 tx=0x7f291400ecb0 comp rx=0 tx=0).stop 2024-03-19T03:32:50.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.812+0000 7f292da00700 1 -- 172.21.15.117:0/2363951125 shutdown_connections 2024-03-19T03:32:50.813 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.812+0000 7f292da00700 1 --2- 172.21.15.117:0/2363951125 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f290c0a2910 0x7f290c0a4dd0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:50.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.812+0000 7f292da00700 1 --2- 172.21.15.117:0/2363951125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2928108090 0x7f29281a8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:50.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.812+0000 7f292da00700 1 --2- 172.21.15.117:0/2363951125 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2928102e30 0x7f2928070ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:50.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.812+0000 7f292da00700 1 --2- 172.21.15.117:0/2363951125 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2928105760 0x7f2928071220 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:50.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.812+0000 7f292da00700 1 -- 172.21.15.117:0/2363951125 >> 172.21.15.117:0/2363951125 conn(0x7f29280fc7e0 msgr2=0x7f29281038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:50.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.812+0000 7f292da00700 1 -- 172.21.15.117:0/2363951125 shutdown_connections 2024-03-19T03:32:50.814 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:50.812+0000 7f292da00700 1 -- 172.21.15.117:0/2363951125 wait complete. 2024-03-19T03:32:51.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:51 smithi117 ceph-mon[31838]: pgmap v389: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 409 B/s wr, 10 op/s; 6.3 KiB/s, 0 keys/s, 1 objects/s recovering 2024-03-19T03:32:51.616 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-19T03:32:51.648 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:51 smithi178 ceph-mon[67578]: pgmap v389: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 409 B/s wr, 10 op/s; 6.3 KiB/s, 0 keys/s, 1 objects/s recovering 2024-03-19T03:32:51.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:51 smithi119 ceph-mon[66861]: pgmap v389: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 409 B/s wr, 10 op/s; 6.3 KiB/s, 0 keys/s, 1 objects/s recovering 2024-03-19T03:32:52.132 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:32:52.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:52 smithi117 ceph-mon[31838]: from='client.25250 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:32:52.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:52 smithi119 ceph-mon[66861]: from='client.25250 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:32:52.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:52 smithi178 ceph-mon[67578]: from='client.25250 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:32:53.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:53 smithi117 ceph-mon[31838]: pgmap v390: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 345 B/s wr, 10 op/s; 5.3 KiB/s, 0 keys/s, 1 objects/s recovering 2024-03-19T03:32:53.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:53 smithi119 ceph-mon[66861]: pgmap v390: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 345 B/s wr, 10 op/s; 5.3 KiB/s, 0 keys/s, 1 objects/s recovering 2024-03-19T03:32:53.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:53 smithi178 ceph-mon[67578]: pgmap v390: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 345 B/s wr, 10 op/s; 5.3 KiB/s, 0 keys/s, 1 objects/s recovering 2024-03-19T03:32:54.416 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.414+0000 7f9a81a00700 1 -- 172.21.15.117:0/3836546695 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a7c103e20 msgr2=0x7f9a7c10a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:54.416 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.414+0000 7f9a81a00700 1 --2- 172.21.15.117:0/3836546695 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a7c103e20 0x7f9a7c10a6d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9a7400b0d0 tx=0x7f9a7400b3e0 comp rx=0 tx=0).stop 2024-03-19T03:32:54.416 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.415+0000 7f9a81a00700 1 -- 172.21.15.117:0/3836546695 shutdown_connections 2024-03-19T03:32:54.416 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.415+0000 7f9a81a00700 1 --2- 172.21.15.117:0/3836546695 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a7c103e20 0x7f9a7c10a6d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:54.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.415+0000 7f9a81a00700 1 --2- 172.21.15.117:0/3836546695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a7c102280 0x7f9a7c1026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:54.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.415+0000 7f9a81a00700 1 --2- 172.21.15.117:0/3836546695 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9a7c103480 0x7f9a7c1038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:54.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.415+0000 7f9a81a00700 1 -- 172.21.15.117:0/3836546695 >> 172.21.15.117:0/3836546695 conn(0x7f9a7c0fd820 msgr2=0x7f9a7c0ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:54.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.415+0000 7f9a81a00700 1 -- 172.21.15.117:0/3836546695 shutdown_connections 2024-03-19T03:32:54.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.415+0000 7f9a81a00700 1 -- 172.21.15.117:0/3836546695 wait complete. 2024-03-19T03:32:54.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.416+0000 7f9a81a00700 1 Processor -- start 2024-03-19T03:32:54.417 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.417+0000 7f9a81a00700 1 -- start start 2024-03-19T03:32:54.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.417+0000 7f9a81a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a7c102280 0x7f9a7c0760b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:54.418 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.417+0000 7f9a81a00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9a7c103480 0x7f9a7c074700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:54.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.417+0000 7f9a81a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a7c103e20 0x7f9a7c074c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:54.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.417+0000 7f9a81a00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7c10dec0 con 0x7f9a7c103e20 2024-03-19T03:32:54.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.417+0000 7f9a81a00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7c10dd40 con 0x7f9a7c102280 2024-03-19T03:32:54.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.417+0000 7f9a81a00700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7c10e040 con 0x7f9a7c103480 2024-03-19T03:32:54.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.417+0000 7f9a7be00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a7c102280 0x7f9a7c0760b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:54.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.417+0000 7f9a7be00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a7c102280 0x7f9a7c0760b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:58122/0 (socket says 172.21.15.117:58122) 2024-03-19T03:32:54.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.417+0000 7f9a7be00700 1 -- 172.21.15.117:0/1078207994 learned_addr learned my addr 172.21.15.117:0/1078207994 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:54.419 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.418+0000 7f9a7b400700 1 --2- 172.21.15.117:0/1078207994 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9a7c103480 0x7f9a7c074700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:54.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.418+0000 7f9a7be00700 1 -- 172.21.15.117:0/1078207994 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9a7c103480 msgr2=0x7f9a7c074700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:54.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.418+0000 7f9a7be00700 1 --2- 172.21.15.117:0/1078207994 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9a7c103480 0x7f9a7c074700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:54.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.418+0000 7f9a7be00700 1 -- 172.21.15.117:0/1078207994 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a7c103e20 msgr2=0x7f9a7c074c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:54.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.418+0000 7f9a7be00700 1 --2- 172.21.15.117:0/1078207994 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a7c103e20 0x7f9a7c074c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:54.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.418+0000 7f9a7be00700 1 -- 172.21.15.117:0/1078207994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a7c0754f0 con 0x7f9a7c102280 2024-03-19T03:32:54.420 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.418+0000 7f9a7be00700 1 --2- 172.21.15.117:0/1078207994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a7c102280 0x7f9a7c0760b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00e970 tx=0x7f9a6c00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:54.421 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.418+0000 7f9a78c00700 1 -- 172.21.15.117:0/1078207994 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c00cbd0 con 0x7f9a7c102280 2024-03-19T03:32:54.421 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.418+0000 7f9a78c00700 1 -- 172.21.15.117:0/1078207994 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a6c004500 con 0x7f9a7c102280 2024-03-19T03:32:54.421 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.418+0000 7f9a81a00700 1 -- 172.21.15.117:0/1078207994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a7c1a2640 con 0x7f9a7c102280 2024-03-19T03:32:54.421 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.419+0000 7f9a78c00700 1 -- 172.21.15.117:0/1078207994 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c003e30 con 0x7f9a7c102280 2024-03-19T03:32:54.421 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.419+0000 7f9a81a00700 1 -- 172.21.15.117:0/1078207994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a7c1a2b80 con 0x7f9a7c102280 2024-03-19T03:32:54.421 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.420+0000 7f9a81a00700 1 -- 172.21.15.117:0/1078207994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a7c068fd0 con 0x7f9a7c102280 2024-03-19T03:32:54.422 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.422+0000 7f9a78c00700 1 -- 172.21.15.117:0/1078207994 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f9a6c004000 con 0x7f9a7c102280 2024-03-19T03:32:54.424 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.423+0000 7f9a78c00700 1 --2- 172.21.15.117:0/1078207994 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f9a600a2910 0x7f9a600a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:54.424 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.423+0000 7f9a78c00700 1 -- 172.21.15.117:0/1078207994 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f9a6c0cf130 con 0x7f9a7c102280 2024-03-19T03:32:54.424 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.423+0000 7f9a7b400700 1 --2- 172.21.15.117:0/1078207994 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f9a600a2910 0x7f9a600a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:54.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.424+0000 7f9a7b400700 1 --2- 172.21.15.117:0/1078207994 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f9a600a2910 0x7f9a600a4dd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9a68009be0 tx=0x7f9a68009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:54.431 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.430+0000 7f9a78c00700 1 -- 172.21.15.117:0/1078207994 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9a6c09bd80 con 0x7f9a7c102280 2024-03-19T03:32:54.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.672+0000 7f9a81a00700 1 -- 172.21.15.117:0/1078207994 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f9a7c062750 con 0x7f9a600a2910 2024-03-19T03:32:54.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.692+0000 7f9a78c00700 1 -- 172.21.15.117:0/1078207994 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f9a7c062750 con 0x7f9a600a2910 2024-03-19T03:32:54.694 INFO:teuthology.orchestra.run.smithi117.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-19T03:32:54.694 INFO:teuthology.orchestra.run.smithi117.stdout:mds.a smithi117 running (2m) 31s ago 2m 16.6M - 19.0.0-2206-gb6ae2491 2312c8e42d00 175e7bb21179 2024-03-19T03:32:54.694 INFO:teuthology.orchestra.run.smithi117.stdout:mds.b smithi119 running (2m) 77s ago 2m 26.3M - 19.0.0-2206-gb6ae2491 2312c8e42d00 de4c1844e911 2024-03-19T03:32:54.694 INFO:teuthology.orchestra.run.smithi117.stdout:mds.c smithi178 running (2m) 78s ago 2m 14.7M - 19.0.0-2206-gb6ae2491 2312c8e42d00 5baaacaf8c4a 2024-03-19T03:32:54.695 INFO:teuthology.orchestra.run.smithi117.stdout:mds.d smithi117 running (2m) 31s ago 2m 12.5M - 19.0.0-2206-gb6ae2491 2312c8e42d00 4dd4662cf281 2024-03-19T03:32:54.695 INFO:teuthology.orchestra.run.smithi117.stdout:mds.e smithi119 running (2m) 77s ago 2m 17.3M - 19.0.0-2206-gb6ae2491 2312c8e42d00 0a8ea9d7e2a1 2024-03-19T03:32:54.695 INFO:teuthology.orchestra.run.smithi117.stdout:mds.f smithi178 running (2m) 78s ago 2m 17.2M - 19.0.0-2206-gb6ae2491 2312c8e42d00 21115f9198fa 2024-03-19T03:32:54.695 INFO:teuthology.orchestra.run.smithi117.stdout:mds.g smithi117 running (2m) 31s ago 2m 16.4M - 19.0.0-2206-gb6ae2491 2312c8e42d00 71a8940c83dd 2024-03-19T03:32:54.695 INFO:teuthology.orchestra.run.smithi117.stdout:mds.h smithi119 running (2m) 77s ago 2m 17.0M - 19.0.0-2206-gb6ae2491 2312c8e42d00 f0cc29854e2d 2024-03-19T03:32:54.696 INFO:teuthology.orchestra.run.smithi117.stdout:mds.i smithi178 running (2m) 78s ago 2m 16.4M - 19.0.0-2206-gb6ae2491 2312c8e42d00 5ef48f5e4afc 2024-03-19T03:32:54.696 INFO:teuthology.orchestra.run.smithi117.stdout:mds.j smithi117 running (2m) 31s ago 2m 12.5M - 19.0.0-2206-gb6ae2491 2312c8e42d00 657e4491ff59 2024-03-19T03:32:54.696 INFO:teuthology.orchestra.run.smithi117.stdout:mds.k smithi119 running (3m) 77s ago 2m 16.9M - 19.0.0-2206-gb6ae2491 2312c8e42d00 0ad918646f67 2024-03-19T03:32:54.696 INFO:teuthology.orchestra.run.smithi117.stdout:mds.l smithi178 running (3m) 78s ago 3m 18.1M - 19.0.0-2206-gb6ae2491 2312c8e42d00 503c6609fe6d 2024-03-19T03:32:54.696 INFO:teuthology.orchestra.run.smithi117.stdout:mgr.x smithi117 *:9283,8765 running (11m) 31s ago 11m 514M - 19.0.0-2206-gb6ae2491 2312c8e42d00 fb84b6bcbbed 2024-03-19T03:32:54.696 INFO:teuthology.orchestra.run.smithi117.stdout:mgr.y smithi119 *:8443,8765 running (8m) 77s ago 8m 454M - 19.0.0-2206-gb6ae2491 2312c8e42d00 ed517e272106 2024-03-19T03:32:54.696 INFO:teuthology.orchestra.run.smithi117.stdout:mgr.z smithi178 *:8443,8765 running (8m) 78s ago 8m 452M - 19.0.0-2206-gb6ae2491 2312c8e42d00 7a6e85b68b39 2024-03-19T03:32:54.696 INFO:teuthology.orchestra.run.smithi117.stdout:mon.a smithi117 running (11m) 31s ago 11m 48.5M 2048M 19.0.0-2206-gb6ae2491 2312c8e42d00 e14c366900a5 2024-03-19T03:32:54.696 INFO:teuthology.orchestra.run.smithi117.stdout:mon.b smithi119 running (9m) 77s ago 9m 43.6M 2048M 19.0.0-2206-gb6ae2491 2312c8e42d00 b6d5cb3916da 2024-03-19T03:32:54.697 INFO:teuthology.orchestra.run.smithi117.stdout:mon.c smithi178 running (9m) 78s ago 9m 43.8M 2048M 19.0.0-2206-gb6ae2491 2312c8e42d00 fa156234d0fc 2024-03-19T03:32:54.697 INFO:teuthology.orchestra.run.smithi117.stdout:osd.0 smithi117 running (8m) 31s ago 8m 45.4M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 246df83a8f02 2024-03-19T03:32:54.697 INFO:teuthology.orchestra.run.smithi117.stdout:osd.1 smithi119 running (8m) 77s ago 8m 60.1M 4270M 19.0.0-2206-gb6ae2491 2312c8e42d00 daddd3056be3 2024-03-19T03:32:54.697 INFO:teuthology.orchestra.run.smithi117.stdout:osd.2 smithi178 running (7m) 78s ago 7m 46.5M 4270M 19.0.0-2206-gb6ae2491 2312c8e42d00 9838e6b75cb7 2024-03-19T03:32:54.697 INFO:teuthology.orchestra.run.smithi117.stdout:osd.3 smithi117 running (7m) 31s ago 7m 43.1M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 d52cba30da4a 2024-03-19T03:32:54.697 INFO:teuthology.orchestra.run.smithi117.stdout:osd.4 smithi119 running (6m) 77s ago 6m 42.9M 4270M 19.0.0-2206-gb6ae2491 2312c8e42d00 6794d9b7a9b4 2024-03-19T03:32:54.697 INFO:teuthology.orchestra.run.smithi117.stdout:osd.5 smithi178 running (6m) 78s ago 6m 45.4M 4270M 19.0.0-2206-gb6ae2491 2312c8e42d00 ece9b29409b4 2024-03-19T03:32:54.697 INFO:teuthology.orchestra.run.smithi117.stdout:osd.6 smithi117 running (5m) 31s ago 5m 52.8M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 8cf120542559 2024-03-19T03:32:54.697 INFO:teuthology.orchestra.run.smithi117.stdout:osd.7 smithi119 running (5m) 77s ago 5m 54.4M 4270M 19.0.0-2206-gb6ae2491 2312c8e42d00 f4e014e08895 2024-03-19T03:32:54.698 INFO:teuthology.orchestra.run.smithi117.stdout:osd.8 smithi178 running (4m) 78s ago 4m 43.3M 4270M 19.0.0-2206-gb6ae2491 2312c8e42d00 2539095efb1b 2024-03-19T03:32:54.698 INFO:teuthology.orchestra.run.smithi117.stdout:osd.9 smithi117 running (4m) 31s ago 4m 53.7M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 c78311a78edf 2024-03-19T03:32:54.698 INFO:teuthology.orchestra.run.smithi117.stdout:osd.10 smithi119 running (3m) 77s ago 3m 44.0M 4270M 19.0.0-2206-gb6ae2491 2312c8e42d00 b65964565b87 2024-03-19T03:32:54.698 INFO:teuthology.orchestra.run.smithi117.stdout:osd.11 smithi178 running (3m) 78s ago 3m 53.9M 4270M 19.0.0-2206-gb6ae2491 2312c8e42d00 ec6bb26c6b20 2024-03-19T03:32:54.699 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.695+0000 7f9a81a00700 1 -- 172.21.15.117:0/1078207994 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f9a600a2910 msgr2=0x7f9a600a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:54.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.695+0000 7f9a81a00700 1 --2- 172.21.15.117:0/1078207994 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f9a600a2910 0x7f9a600a4dd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9a68009be0 tx=0x7f9a68009260 comp rx=0 tx=0).stop 2024-03-19T03:32:54.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.695+0000 7f9a81a00700 1 -- 172.21.15.117:0/1078207994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a7c102280 msgr2=0x7f9a7c0760b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:54.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.695+0000 7f9a81a00700 1 --2- 172.21.15.117:0/1078207994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a7c102280 0x7f9a7c0760b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00e970 tx=0x7f9a6c00ec80 comp rx=0 tx=0).stop 2024-03-19T03:32:54.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.695+0000 7f9a81a00700 1 -- 172.21.15.117:0/1078207994 shutdown_connections 2024-03-19T03:32:54.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.695+0000 7f9a81a00700 1 --2- 172.21.15.117:0/1078207994 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f9a600a2910 0x7f9a600a4dd0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:54.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.695+0000 7f9a81a00700 1 --2- 172.21.15.117:0/1078207994 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a7c103e20 0x7f9a7c074c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:54.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.695+0000 7f9a81a00700 1 --2- 172.21.15.117:0/1078207994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a7c102280 0x7f9a7c0760b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:54.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.695+0000 7f9a81a00700 1 --2- 172.21.15.117:0/1078207994 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9a7c103480 0x7f9a7c074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:54.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.695+0000 7f9a81a00700 1 -- 172.21.15.117:0/1078207994 >> 172.21.15.117:0/1078207994 conn(0x7f9a7c0fd820 msgr2=0x7f9a7c1066b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:54.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.695+0000 7f9a81a00700 1 -- 172.21.15.117:0/1078207994 shutdown_connections 2024-03-19T03:32:54.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:54.695+0000 7f9a81a00700 1 -- 172.21.15.117:0/1078207994 wait complete. 2024-03-19T03:32:55.551 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-19T03:32:55.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:55 smithi119 ceph-mon[66861]: pgmap v391: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 85 B/s wr, 10 op/s; 5.3 KiB/s, 0 keys/s, 1 objects/s recovering 2024-03-19T03:32:55.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:55 smithi117 ceph-mon[31838]: pgmap v391: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 85 B/s wr, 10 op/s; 5.3 KiB/s, 0 keys/s, 1 objects/s recovering 2024-03-19T03:32:55.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:55 smithi178 ceph-mon[67578]: pgmap v391: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 85 B/s wr, 10 op/s; 5.3 KiB/s, 0 keys/s, 1 objects/s recovering 2024-03-19T03:32:56.000 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:32:56.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:56 smithi119 ceph-mon[66861]: from='client.25256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:32:56.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:56 smithi117 ceph-mon[31838]: from='client.25256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:32:56.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:56 smithi178 ceph-mon[67578]: from='client.25256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:32:57.419 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:57 smithi117 ceph-mon[31838]: pgmap v392: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:32:57.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:57 smithi119 ceph-mon[66861]: pgmap v392: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:32:57.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:57 smithi178 ceph-mon[67578]: pgmap v392: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:32:58.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.069+0000 7f12f4e00700 1 -- 172.21.15.117:0/893935995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12f0103f00 msgr2=0x7f12f010a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:58.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.069+0000 7f12f4e00700 1 --2- 172.21.15.117:0/893935995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12f0103f00 0x7f12f010a7b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f12e800b0d0 tx=0x7f12e800b3e0 comp rx=0 tx=0).stop 2024-03-19T03:32:58.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.069+0000 7f12f4e00700 1 -- 172.21.15.117:0/893935995 shutdown_connections 2024-03-19T03:32:58.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.069+0000 7f12f4e00700 1 --2- 172.21.15.117:0/893935995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12f0103f00 0x7f12f010a7b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:58.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.069+0000 7f12f4e00700 1 --2- 172.21.15.117:0/893935995 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12f0102360 0x7f12f0102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:58.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.069+0000 7f12f4e00700 1 --2- 172.21.15.117:0/893935995 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12f0103560 0x7f12f01039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:58.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.069+0000 7f12f4e00700 1 -- 172.21.15.117:0/893935995 >> 172.21.15.117:0/893935995 conn(0x7f12f00fd8c0 msgr2=0x7f12f00ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:58.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.069+0000 7f12f4e00700 1 -- 172.21.15.117:0/893935995 shutdown_connections 2024-03-19T03:32:58.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.069+0000 7f12f4e00700 1 -- 172.21.15.117:0/893935995 wait complete. 2024-03-19T03:32:58.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.071+0000 7f12f4e00700 1 Processor -- start 2024-03-19T03:32:58.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.071+0000 7f12f4e00700 1 -- start start 2024-03-19T03:32:58.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.071+0000 7f12f4e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12f0102360 0x7f12f0198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:58.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.071+0000 7f12f4e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12f0103560 0x7f12f0198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:58.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.071+0000 7f12f4e00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12f0103f00 0x7f12f019fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:58.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.071+0000 7f12f4e00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f12f010df80 con 0x7f12f0103560 2024-03-19T03:32:58.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.071+0000 7f12f4e00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f12f010de00 con 0x7f12f0102360 2024-03-19T03:32:58.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.071+0000 7f12f4e00700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f12f010e100 con 0x7f12f0103f00 2024-03-19T03:32:58.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.072+0000 7f12ede00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12f0102360 0x7f12f0198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:58.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.072+0000 7f12ede00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12f0102360 0x7f12f0198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:42282/0 (socket says 172.21.15.117:42282) 2024-03-19T03:32:58.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.072+0000 7f12ede00700 1 -- 172.21.15.117:0/3652533453 learned_addr learned my addr 172.21.15.117:0/3652533453 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:32:58.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.072+0000 7f12ee800700 1 --2- 172.21.15.117:0/3652533453 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12f0103f00 0x7f12f019fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:58.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.072+0000 7f12ed400700 1 --2- 172.21.15.117:0/3652533453 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12f0103560 0x7f12f0198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:58.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.072+0000 7f12ede00700 1 -- 172.21.15.117:0/3652533453 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12f0103f00 msgr2=0x7f12f019fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:58.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.072+0000 7f12ede00700 1 --2- 172.21.15.117:0/3652533453 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12f0103f00 0x7f12f019fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:58.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.072+0000 7f12ede00700 1 -- 172.21.15.117:0/3652533453 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12f0103560 msgr2=0x7f12f0198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:58.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.072+0000 7f12ede00700 1 --2- 172.21.15.117:0/3652533453 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12f0103560 0x7f12f0198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:58.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.072+0000 7f12ede00700 1 -- 172.21.15.117:0/3652533453 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f01a01a0 con 0x7f12f0102360 2024-03-19T03:32:58.074 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.073+0000 7f12ede00700 1 --2- 172.21.15.117:0/3652533453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12f0102360 0x7f12f0198540 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f12e000d980 tx=0x7f12e000dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:58.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.073+0000 7f12e6a00700 1 -- 172.21.15.117:0/3652533453 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e0016070 con 0x7f12f0102360 2024-03-19T03:32:58.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.073+0000 7f12e6a00700 1 -- 172.21.15.117:0/3652533453 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12e000bb60 con 0x7f12f0102360 2024-03-19T03:32:58.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.073+0000 7f12f4e00700 1 -- 172.21.15.117:0/3652533453 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12f01a0490 con 0x7f12f0102360 2024-03-19T03:32:58.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.073+0000 7f12e6a00700 1 -- 172.21.15.117:0/3652533453 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e000f620 con 0x7f12f0102360 2024-03-19T03:32:58.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.073+0000 7f12f4e00700 1 -- 172.21.15.117:0/3652533453 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12f01a0a20 con 0x7f12f0102360 2024-03-19T03:32:58.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.074+0000 7f12f4e00700 1 -- 172.21.15.117:0/3652533453 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12f0068fd0 con 0x7f12f0102360 2024-03-19T03:32:58.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.077+0000 7f12e6a00700 1 -- 172.21.15.117:0/3652533453 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f12e0019070 con 0x7f12f0102360 2024-03-19T03:32:58.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.079+0000 7f12e6a00700 1 --2- 172.21.15.117:0/3652533453 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f12d40a2910 0x7f12d40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:32:58.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.079+0000 7f12e6a00700 1 -- 172.21.15.117:0/3652533453 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f12e00cea70 con 0x7f12f0102360 2024-03-19T03:32:58.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.079+0000 7f12ed400700 1 --2- 172.21.15.117:0/3652533453 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f12d40a2910 0x7f12d40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:32:58.081 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.080+0000 7f12ed400700 1 --2- 172.21.15.117:0/3652533453 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f12d40a2910 0x7f12d40a4dd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f12dc005950 tx=0x7f12dc0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:32:58.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.087+0000 7f12e6a00700 1 -- 172.21.15.117:0/3652533453 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f12e0020030 con 0x7f12f0102360 2024-03-19T03:32:58.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.324+0000 7f12f4e00700 1 -- 172.21.15.117:0/3652533453 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f12f0062750 con 0x7f12d40a2910 2024-03-19T03:32:58.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.331+0000 7f12e6a00700 1 -- 172.21.15.117:0/3652533453 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f12f0062750 con 0x7f12d40a2910 2024-03-19T03:32:58.332 INFO:teuthology.orchestra.run.smithi117.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-19T03:32:58.332 INFO:teuthology.orchestra.run.smithi117.stdout:mds.all 12/12 81s ago 3m smithi117=a;smithi117=d;smithi117=g;smithi117=j;smithi119=b;smithi119=e;smithi119=h;smithi119=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l;count:12 2024-03-19T03:32:58.332 INFO:teuthology.orchestra.run.smithi117.stdout:mgr 3/3 81s ago 9m smithi117=x;smithi119=y;smithi178=z;count:3 2024-03-19T03:32:58.332 INFO:teuthology.orchestra.run.smithi117.stdout:mon 3/3 81s ago 9m smithi117:172.21.15.117=a;smithi119:172.21.15.119=b;smithi178:172.21.15.178=c;count:3 2024-03-19T03:32:58.332 INFO:teuthology.orchestra.run.smithi117.stdout:osd 12 81s ago - 2024-03-19T03:32:58.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.334+0000 7f12f4e00700 1 -- 172.21.15.117:0/3652533453 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f12d40a2910 msgr2=0x7f12d40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:58.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.334+0000 7f12f4e00700 1 --2- 172.21.15.117:0/3652533453 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f12d40a2910 0x7f12d40a4dd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f12dc005950 tx=0x7f12dc0058e0 comp rx=0 tx=0).stop 2024-03-19T03:32:58.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.334+0000 7f12f4e00700 1 -- 172.21.15.117:0/3652533453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12f0102360 msgr2=0x7f12f0198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:32:58.335 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.334+0000 7f12f4e00700 1 --2- 172.21.15.117:0/3652533453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12f0102360 0x7f12f0198540 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f12e000d980 tx=0x7f12e000dd40 comp rx=0 tx=0).stop 2024-03-19T03:32:58.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.334+0000 7f12f4e00700 1 -- 172.21.15.117:0/3652533453 shutdown_connections 2024-03-19T03:32:58.337 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.334+0000 7f12f4e00700 1 --2- 172.21.15.117:0/3652533453 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f12d40a2910 0x7f12d40a4dd0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:58.338 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.334+0000 7f12f4e00700 1 --2- 172.21.15.117:0/3652533453 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12f0103f00 0x7f12f019fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:58.338 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.334+0000 7f12f4e00700 1 --2- 172.21.15.117:0/3652533453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12f0102360 0x7f12f0198540 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:58.338 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.334+0000 7f12f4e00700 1 --2- 172.21.15.117:0/3652533453 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12f0103560 0x7f12f0198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:32:58.338 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.334+0000 7f12f4e00700 1 -- 172.21.15.117:0/3652533453 >> 172.21.15.117:0/3652533453 conn(0x7f12f00fd8c0 msgr2=0x7f12f0106860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:32:58.338 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.335+0000 7f12f4e00700 1 -- 172.21.15.117:0/3652533453 shutdown_connections 2024-03-19T03:32:58.338 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:32:58.335+0000 7f12f4e00700 1 -- 172.21.15.117:0/3652533453 wait complete. 2024-03-19T03:32:59.127 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-19T03:32:59.617 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:32:59.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:59 smithi119 ceph-mon[66861]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:32:59.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:32:59 smithi119 ceph-mon[66861]: pgmap v393: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:32:59.764 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:59 smithi117 ceph-mon[31838]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:32:59.764 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:32:59 smithi117 ceph-mon[31838]: pgmap v393: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:32:59.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:59 smithi178 ceph-mon[67578]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:32:59.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:32:59 smithi178 ceph-mon[67578]: pgmap v393: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:33:01.663 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:01 smithi117 ceph-mon[31838]: pgmap v394: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:01.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:01 smithi119 ceph-mon[66861]: pgmap v394: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:01.834 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.833+0000 7fcdf3400700 1 -- 172.21.15.117:0/3904746380 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdec074d10 msgr2=0x7fcdec075190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:01.835 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.833+0000 7fcdf3400700 1 --2- 172.21.15.117:0/3904746380 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdec074d10 0x7fcdec075190 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8009990 tx=0x7fcdd8009ca0 comp rx=0 tx=0).stop 2024-03-19T03:33:01.835 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.833+0000 7fcdf3400700 1 -- 172.21.15.117:0/3904746380 shutdown_connections 2024-03-19T03:33:01.835 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.833+0000 7fcdf3400700 1 --2- 172.21.15.117:0/3904746380 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcdec10a240 0x7fcdec10c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:01.835 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.833+0000 7fcdf3400700 1 --2- 172.21.15.117:0/3904746380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdec0762e0 0x7fcdec074740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:01.835 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.833+0000 7fcdf3400700 1 --2- 172.21.15.117:0/3904746380 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdec074d10 0x7fcdec075190 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:01.835 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.833+0000 7fcdf3400700 1 -- 172.21.15.117:0/3904746380 >> 172.21.15.117:0/3904746380 conn(0x7fcdec0fd8c0 msgr2=0x7fcdec0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:01.835 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.833+0000 7fcdf3400700 1 -- 172.21.15.117:0/3904746380 shutdown_connections 2024-03-19T03:33:01.835 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.833+0000 7fcdf3400700 1 -- 172.21.15.117:0/3904746380 wait complete. 2024-03-19T03:33:01.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.834+0000 7fcdf3400700 1 Processor -- start 2024-03-19T03:33:01.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.834+0000 7fcdf3400700 1 -- start start 2024-03-19T03:33:01.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.835+0000 7fcdf3400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdec074d10 0x7fcdec19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:01.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.835+0000 7fcdf3400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdec0762e0 0x7fcdec19ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:01.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.835+0000 7fcdf3400700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcdec10a240 0x7fcdec1a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:01.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.835+0000 7fcdf3400700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcdec110180 con 0x7fcdec074d10 2024-03-19T03:33:01.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.835+0000 7fcdf3400700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fcdec110000 con 0x7fcdec0762e0 2024-03-19T03:33:01.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.835+0000 7fcdf3400700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fcdec110300 con 0x7fcdec10a240 2024-03-19T03:33:01.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.835+0000 7fcdebe00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdec0762e0 0x7fcdec19ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:01.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.835+0000 7fcdf0a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdec074d10 0x7fcdec19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:01.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.835+0000 7fcdebe00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdec0762e0 0x7fcdec19ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:42290/0 (socket says 172.21.15.117:42290) 2024-03-19T03:33:01.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.835+0000 7fcdebe00700 1 -- 172.21.15.117:0/2610716961 learned_addr learned my addr 172.21.15.117:0/2610716961 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:01.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.835+0000 7fcdf1400700 1 --2- 172.21.15.117:0/2610716961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcdec10a240 0x7fcdec1a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:01.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.836+0000 7fcdebe00700 1 -- 172.21.15.117:0/2610716961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcdec10a240 msgr2=0x7fcdec1a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:01.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.836+0000 7fcdebe00700 1 --2- 172.21.15.117:0/2610716961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcdec10a240 0x7fcdec1a1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:01.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.836+0000 7fcdebe00700 1 -- 172.21.15.117:0/2610716961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdec074d10 msgr2=0x7fcdec19a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:01.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.836+0000 7fcdebe00700 1 --2- 172.21.15.117:0/2610716961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdec074d10 0x7fcdec19a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:01.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.836+0000 7fcdebe00700 1 -- 172.21.15.117:0/2610716961 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdec1a2390 con 0x7fcdec0762e0 2024-03-19T03:33:01.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.836+0000 7fcdf1400700 1 --2- 172.21.15.117:0/2610716961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcdec10a240 0x7fcdec1a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:33:01.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.836+0000 7fcdebe00700 1 --2- 172.21.15.117:0/2610716961 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdec0762e0 0x7fcdec19ac70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fcdd800b7b0 tx=0x7fcdd800bbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:01.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.836+0000 7fcde9600700 1 -- 172.21.15.117:0/2610716961 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd8027070 con 0x7fcdec0762e0 2024-03-19T03:33:01.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.836+0000 7fcde9600700 1 -- 172.21.15.117:0/2610716961 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdd8022470 con 0x7fcdec0762e0 2024-03-19T03:33:01.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.836+0000 7fcdf3400700 1 -- 172.21.15.117:0/2610716961 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdec1a2620 con 0x7fcdec0762e0 2024-03-19T03:33:01.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.836+0000 7fcde9600700 1 -- 172.21.15.117:0/2610716961 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd800f6a0 con 0x7fcdec0762e0 2024-03-19T03:33:01.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.837+0000 7fcdf3400700 1 -- 172.21.15.117:0/2610716961 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdec1a2ad0 con 0x7fcdec0762e0 2024-03-19T03:33:01.840 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.838+0000 7fcdf3400700 1 -- 172.21.15.117:0/2610716961 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdcc005360 con 0x7fcdec0762e0 2024-03-19T03:33:01.842 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.841+0000 7fcde9600700 1 -- 172.21.15.117:0/2610716961 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fcdd802a070 con 0x7fcdec0762e0 2024-03-19T03:33:01.843 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.842+0000 7fcde9600700 1 --2- 172.21.15.117:0/2610716961 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fcdd40a2910 0x7fcdd40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:01.843 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.842+0000 7fcde9600700 1 -- 172.21.15.117:0/2610716961 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fcdd80cfc80 con 0x7fcdec0762e0 2024-03-19T03:33:01.843 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.843+0000 7fcdf0a00700 1 --2- 172.21.15.117:0/2610716961 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fcdd40a2910 0x7fcdd40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:01.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.843+0000 7fcdf0a00700 1 --2- 172.21.15.117:0/2610716961 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fcdd40a2910 0x7fcdd40a4dd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fcde0007c60 tx=0x7fcde00073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:01.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:01 smithi178 ceph-mon[67578]: pgmap v394: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:01.848 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:01.848+0000 7fcde9600700 1 -- 172.21.15.117:0/2610716961 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcdd8013070 con 0x7fcdec0762e0 2024-03-19T03:33:02.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:02.091+0000 7fcdf3400700 1 -- 172.21.15.117:0/2610716961 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fcdcc000bf0 con 0x7fcdd40a2910 2024-03-19T03:33:02.094 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:02.094+0000 7fcde9600700 1 -- 172.21.15.117:0/2610716961 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fcdcc000bf0 con 0x7fcdd40a2910 2024-03-19T03:33:02.095 INFO:teuthology.orchestra.run.smithi117.stdout:HOST ADDR LABELS STATUS 2024-03-19T03:33:02.095 INFO:teuthology.orchestra.run.smithi117.stdout:smithi117 172.21.15.117 2024-03-19T03:33:02.095 INFO:teuthology.orchestra.run.smithi117.stdout:smithi119 172.21.15.119 2024-03-19T03:33:02.095 INFO:teuthology.orchestra.run.smithi117.stdout:smithi178 172.21.15.178 2024-03-19T03:33:02.095 INFO:teuthology.orchestra.run.smithi117.stdout:3 hosts in cluster 2024-03-19T03:33:02.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:02.096+0000 7fcdf3400700 1 -- 172.21.15.117:0/2610716961 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fcdd40a2910 msgr2=0x7fcdd40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:02.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:02.096+0000 7fcdf3400700 1 --2- 172.21.15.117:0/2610716961 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fcdd40a2910 0x7fcdd40a4dd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fcde0007c60 tx=0x7fcde00073d0 comp rx=0 tx=0).stop 2024-03-19T03:33:02.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:02.096+0000 7fcdf3400700 1 -- 172.21.15.117:0/2610716961 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdec0762e0 msgr2=0x7fcdec19ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:02.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:02.096+0000 7fcdf3400700 1 --2- 172.21.15.117:0/2610716961 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdec0762e0 0x7fcdec19ac70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fcdd800b7b0 tx=0x7fcdd800bbb0 comp rx=0 tx=0).stop 2024-03-19T03:33:02.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:02.097+0000 7fcdf3400700 1 -- 172.21.15.117:0/2610716961 shutdown_connections 2024-03-19T03:33:02.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:02.097+0000 7fcdf3400700 1 --2- 172.21.15.117:0/2610716961 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fcdd40a2910 0x7fcdd40a4dd0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:02.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:02.097+0000 7fcdf3400700 1 --2- 172.21.15.117:0/2610716961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcdec10a240 0x7fcdec1a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:02.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:02.097+0000 7fcdf3400700 1 --2- 172.21.15.117:0/2610716961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdec074d10 0x7fcdec19a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:02.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:02.097+0000 7fcdf3400700 1 --2- 172.21.15.117:0/2610716961 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdec0762e0 0x7fcdec19ac70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:02.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:02.097+0000 7fcdf3400700 1 -- 172.21.15.117:0/2610716961 >> 172.21.15.117:0/2610716961 conn(0x7fcdec0fd8c0 msgr2=0x7fcdec106190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:02.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:02.097+0000 7fcdf3400700 1 -- 172.21.15.117:0/2610716961 shutdown_connections 2024-03-19T03:33:02.099 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:02.097+0000 7fcdf3400700 1 -- 172.21.15.117:0/2610716961 wait complete. 2024-03-19T03:33:02.839 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-19T03:33:03.296 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:33:03.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:03 smithi119 ceph-mon[66861]: from='client.25268 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:33:03.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:03 smithi119 ceph-mon[66861]: pgmap v395: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T03:33:03.747 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:03 smithi117 ceph-mon[31838]: from='client.25268 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:33:03.747 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:03 smithi117 ceph-mon[31838]: pgmap v395: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T03:33:03.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:03 smithi178 ceph-mon[67578]: from='client.25268 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:33:03.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:03 smithi178 ceph-mon[67578]: pgmap v395: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T03:33:05.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.602+0000 7f3b09600700 1 -- 172.21.15.117:0/3967826207 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3b041056f0 msgr2=0x7f3b04107ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:05.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.602+0000 7f3b09600700 1 --2- 172.21.15.117:0/3967826207 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3b041056f0 0x7f3b04107ae0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3af0009990 tx=0x7f3af0009ca0 comp rx=0 tx=0).stop 2024-03-19T03:33:05.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.602+0000 7f3b09600700 1 -- 172.21.15.117:0/3967826207 shutdown_connections 2024-03-19T03:33:05.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.602+0000 7f3b09600700 1 --2- 172.21.15.117:0/3967826207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b04108020 0x7f3b0410a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:05.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.602+0000 7f3b09600700 1 --2- 172.21.15.117:0/3967826207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3b04102dc0 0x7f3b041051b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:05.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.602+0000 7f3b09600700 1 --2- 172.21.15.117:0/3967826207 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3b041056f0 0x7f3b04107ae0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:05.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.602+0000 7f3b09600700 1 -- 172.21.15.117:0/3967826207 >> 172.21.15.117:0/3967826207 conn(0x7f3b040fc7d0 msgr2=0x7f3b040fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:05.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.602+0000 7f3b09600700 1 -- 172.21.15.117:0/3967826207 shutdown_connections 2024-03-19T03:33:05.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.602+0000 7f3b09600700 1 -- 172.21.15.117:0/3967826207 wait complete. 2024-03-19T03:33:05.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.604+0000 7f3b09600700 1 Processor -- start 2024-03-19T03:33:05.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.604+0000 7f3b09600700 1 -- start start 2024-03-19T03:33:05.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.604+0000 7f3b09600700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3b04102dc0 0x7f3b04198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:05.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.605+0000 7f3b09600700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b041056f0 0x7f3b04198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:05.606 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.605+0000 7f3b09600700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3b04108020 0x7f3b0419fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:05.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.605+0000 7f3b09600700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0410df90 con 0x7f3b04102dc0 2024-03-19T03:33:05.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.605+0000 7f3b09600700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0410de10 con 0x7f3b041056f0 2024-03-19T03:33:05.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.605+0000 7f3b09600700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0410e110 con 0x7f3b04108020 2024-03-19T03:33:05.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.605+0000 7f3b02000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b041056f0 0x7f3b04198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:05.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.605+0000 7f3b02000700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b041056f0 0x7f3b04198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:42308/0 (socket says 172.21.15.117:42308) 2024-03-19T03:33:05.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.605+0000 7f3b02000700 1 -- 172.21.15.117:0/2620041234 learned_addr learned my addr 172.21.15.117:0/2620041234 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:05.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.605+0000 7f3b02a00700 1 --2- 172.21.15.117:0/2620041234 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3b04102dc0 0x7f3b04198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:05.607 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.605+0000 7f3b03400700 1 --2- 172.21.15.117:0/2620041234 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3b04108020 0x7f3b0419fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:05.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.605+0000 7f3b02000700 1 -- 172.21.15.117:0/2620041234 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3b04108020 msgr2=0x7f3b0419fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:05.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.605+0000 7f3b02000700 1 --2- 172.21.15.117:0/2620041234 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3b04108020 0x7f3b0419fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:05.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.605+0000 7f3b02000700 1 -- 172.21.15.117:0/2620041234 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3b04102dc0 msgr2=0x7f3b04198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:05.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.605+0000 7f3b02000700 1 --2- 172.21.15.117:0/2620041234 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3b04102dc0 0x7f3b04198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:05.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.605+0000 7f3b02000700 1 -- 172.21.15.117:0/2620041234 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b041a01f0 con 0x7f3b041056f0 2024-03-19T03:33:05.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.606+0000 7f3b02000700 1 --2- 172.21.15.117:0/2620041234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b041056f0 0x7f3b04198b70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3af000b7b0 tx=0x7f3af000bc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:05.608 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.606+0000 7f3afb400700 1 -- 172.21.15.117:0/2620041234 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af0027070 con 0x7f3b041056f0 2024-03-19T03:33:05.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.606+0000 7f3afb400700 1 -- 172.21.15.117:0/2620041234 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3af0003b20 con 0x7f3b041056f0 2024-03-19T03:33:05.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.606+0000 7f3b09600700 1 -- 172.21.15.117:0/2620041234 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b041a0480 con 0x7f3b041056f0 2024-03-19T03:33:05.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.606+0000 7f3afb400700 1 -- 172.21.15.117:0/2620041234 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af000f6c0 con 0x7f3b041056f0 2024-03-19T03:33:05.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.606+0000 7f3b09600700 1 -- 172.21.15.117:0/2620041234 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b041a0930 con 0x7f3b041056f0 2024-03-19T03:33:05.609 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.607+0000 7f3af8c00700 1 -- 172.21.15.117:0/2620041234 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3adc005330 con 0x7f3b041056f0 2024-03-19T03:33:05.611 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.610+0000 7f3afb400700 1 -- 172.21.15.117:0/2620041234 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f3af002a070 con 0x7f3b041056f0 2024-03-19T03:33:05.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.612+0000 7f3afb400700 1 --2- 172.21.15.117:0/2620041234 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f3ae80a2910 0x7f3ae80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:05.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.612+0000 7f3afb400700 1 -- 172.21.15.117:0/2620041234 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f3af00d0060 con 0x7f3b041056f0 2024-03-19T03:33:05.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.612+0000 7f3b02a00700 1 --2- 172.21.15.117:0/2620041234 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f3ae80a2910 0x7f3ae80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:05.613 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.613+0000 7f3b02a00700 1 --2- 172.21.15.117:0/2620041234 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f3ae80a2910 0x7f3ae80a4dd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3af4009bf0 tx=0x7f3af4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:05.614 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.613+0000 7f3afb400700 1 -- 172.21.15.117:0/2620041234 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3af0013070 con 0x7f3b041056f0 2024-03-19T03:33:05.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:05 smithi119 ceph-mon[66861]: pgmap v396: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:05.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:05 smithi117 ceph-mon[31838]: pgmap v396: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:05.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:05 smithi178 ceph-mon[67578]: pgmap v396: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:05.902 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.901+0000 7f3af8c00700 1 -- 172.21.15.117:0/2620041234 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f3adc000bc0 con 0x7f3ae80a2910 2024-03-19T03:33:05.905 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.904+0000 7f3afb400700 1 -- 172.21.15.117:0/2620041234 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f3adc000bc0 con 0x7f3ae80a2910 2024-03-19T03:33:05.905 INFO:teuthology.orchestra.run.smithi117.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-19T03:33:05.905 INFO:teuthology.orchestra.run.smithi117.stdout:smithi117 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400V6400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-03-19T03:33:05.905 INFO:teuthology.orchestra.run.smithi117.stdout:smithi119 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400Y1400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-19T03:33:05.905 INFO:teuthology.orchestra.run.smithi117.stdout:smithi178 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB75130029480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-03-19T03:33:05.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.906+0000 7f3af8c00700 1 -- 172.21.15.117:0/2620041234 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f3ae80a2910 msgr2=0x7f3ae80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:05.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.906+0000 7f3af8c00700 1 --2- 172.21.15.117:0/2620041234 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f3ae80a2910 0x7f3ae80a4dd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3af4009bf0 tx=0x7f3af4009260 comp rx=0 tx=0).stop 2024-03-19T03:33:05.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.906+0000 7f3af8c00700 1 -- 172.21.15.117:0/2620041234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b041056f0 msgr2=0x7f3b04198b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:05.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.906+0000 7f3af8c00700 1 --2- 172.21.15.117:0/2620041234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b041056f0 0x7f3b04198b70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3af000b7b0 tx=0x7f3af000bc60 comp rx=0 tx=0).stop 2024-03-19T03:33:05.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.907+0000 7f3af8c00700 1 -- 172.21.15.117:0/2620041234 shutdown_connections 2024-03-19T03:33:05.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.907+0000 7f3af8c00700 1 --2- 172.21.15.117:0/2620041234 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f3ae80a2910 0x7f3ae80a4dd0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:05.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.907+0000 7f3af8c00700 1 --2- 172.21.15.117:0/2620041234 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3b04108020 0x7f3b0419fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:05.908 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.907+0000 7f3af8c00700 1 --2- 172.21.15.117:0/2620041234 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3b04102dc0 0x7f3b04198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:05.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.907+0000 7f3af8c00700 1 --2- 172.21.15.117:0/2620041234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b041056f0 0x7f3b04198b70 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:05.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.907+0000 7f3af8c00700 1 -- 172.21.15.117:0/2620041234 >> 172.21.15.117:0/2620041234 conn(0x7f3b040fc7d0 msgr2=0x7f3b041036c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:05.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.907+0000 7f3af8c00700 1 -- 172.21.15.117:0/2620041234 shutdown_connections 2024-03-19T03:33:05.909 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:05.907+0000 7f3af8c00700 1 -- 172.21.15.117:0/2620041234 wait complete. 2024-03-19T03:33:06.683 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-19T03:33:06.692 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi117.front.sepia.ceph.com 2024-03-19T03:33:06.692 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-19T03:33:07.179 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:33:07.721 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:07 smithi117 ceph-mon[31838]: from='client.25274 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:33:07.721 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:07 smithi117 ceph-mon[31838]: pgmap v397: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:33:07.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:07 smithi119 ceph-mon[66861]: from='client.25274 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:33:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:07 smithi119 ceph-mon[66861]: pgmap v397: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:33:07.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:07 smithi178 ceph-mon[67578]: from='client.25274 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:33:07.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:07 smithi178 ceph-mon[67578]: pgmap v397: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:33:09.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.545+0000 7f9806800700 1 -- 172.21.15.117:0/4225200580 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9800105700 msgr2=0x7f9800107af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:09.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.545+0000 7f9806800700 1 --2- 172.21.15.117:0/4225200580 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9800105700 0x7f9800107af0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f97ec009990 tx=0x7f97ec009ca0 comp rx=0 tx=0).stop 2024-03-19T03:33:09.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.545+0000 7f9806800700 1 -- 172.21.15.117:0/4225200580 shutdown_connections 2024-03-19T03:33:09.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.545+0000 7f9806800700 1 --2- 172.21.15.117:0/4225200580 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9800108030 0x7f980010a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:09.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.545+0000 7f9806800700 1 --2- 172.21.15.117:0/4225200580 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9800102dd0 0x7f98001051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:09.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.545+0000 7f9806800700 1 --2- 172.21.15.117:0/4225200580 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9800105700 0x7f9800107af0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:09.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.545+0000 7f9806800700 1 -- 172.21.15.117:0/4225200580 >> 172.21.15.117:0/4225200580 conn(0x7f98000fc7e0 msgr2=0x7f98000fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:09.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.545+0000 7f9806800700 1 -- 172.21.15.117:0/4225200580 shutdown_connections 2024-03-19T03:33:09.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.545+0000 7f9806800700 1 -- 172.21.15.117:0/4225200580 wait complete. 2024-03-19T03:33:09.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.547+0000 7f9806800700 1 Processor -- start 2024-03-19T03:33:09.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.547+0000 7f9806800700 1 -- start start 2024-03-19T03:33:09.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.547+0000 7f9806800700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9800102dd0 0x7f9800198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:09.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.547+0000 7f9806800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9800105700 0x7f9800198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:09.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.547+0000 7f9806800700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9800108030 0x7f980019fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:09.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.547+0000 7f9806800700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f980010dfa0 con 0x7f9800105700 2024-03-19T03:33:09.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.547+0000 7f9806800700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f980010de20 con 0x7f9800102dd0 2024-03-19T03:33:09.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.547+0000 7f9806800700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f980010e120 con 0x7f9800108030 2024-03-19T03:33:09.549 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.547+0000 7f97ff400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9800102dd0 0x7f9800198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:09.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.548+0000 7f97ff400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9800102dd0 0x7f9800198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:54436/0 (socket says 172.21.15.117:54436) 2024-03-19T03:33:09.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.548+0000 7f97ff400700 1 -- 172.21.15.117:0/1233034301 learned_addr learned my addr 172.21.15.117:0/1233034301 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:09.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.548+0000 7f97ffe00700 1 --2- 172.21.15.117:0/1233034301 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9800108030 0x7f980019fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:09.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.548+0000 7f97fea00700 1 --2- 172.21.15.117:0/1233034301 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9800105700 0x7f9800198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:09.550 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.548+0000 7f97ff400700 1 -- 172.21.15.117:0/1233034301 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9800108030 msgr2=0x7f980019fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:09.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.548+0000 7f97ff400700 1 --2- 172.21.15.117:0/1233034301 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9800108030 0x7f980019fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:09.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.548+0000 7f97ff400700 1 -- 172.21.15.117:0/1233034301 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9800105700 msgr2=0x7f9800198b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:09.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.548+0000 7f97ff400700 1 --2- 172.21.15.117:0/1233034301 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9800105700 0x7f9800198b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:09.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.548+0000 7f97ff400700 1 -- 172.21.15.117:0/1233034301 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98001a0280 con 0x7f9800102dd0 2024-03-19T03:33:09.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.548+0000 7f97ffe00700 1 --2- 172.21.15.117:0/1233034301 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9800108030 0x7f980019fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:33:09.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.549+0000 7f97ff400700 1 --2- 172.21.15.117:0/1233034301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9800102dd0 0x7f9800198620 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f97f000e9c0 tx=0x7f97f000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:09.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.549+0000 7f97f7e00700 1 -- 172.21.15.117:0/1233034301 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f000cbd0 con 0x7f9800102dd0 2024-03-19T03:33:09.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.549+0000 7f97f7e00700 1 -- 172.21.15.117:0/1233034301 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97f000ee60 con 0x7f9800102dd0 2024-03-19T03:33:09.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.549+0000 7f9806800700 1 -- 172.21.15.117:0/1233034301 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98001a0570 con 0x7f9800102dd0 2024-03-19T03:33:09.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.549+0000 7f97f7e00700 1 -- 172.21.15.117:0/1233034301 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f0005680 con 0x7f9800102dd0 2024-03-19T03:33:09.551 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.549+0000 7f9806800700 1 -- 172.21.15.117:0/1233034301 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98001a0ab0 con 0x7f9800102dd0 2024-03-19T03:33:09.552 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.550+0000 7f9806800700 1 -- 172.21.15.117:0/1233034301 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97e0005360 con 0x7f9800102dd0 2024-03-19T03:33:09.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.552+0000 7f97f7e00700 1 -- 172.21.15.117:0/1233034301 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f97f0004750 con 0x7f9800102dd0 2024-03-19T03:33:09.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.553+0000 7f97f7e00700 1 --2- 172.21.15.117:0/1233034301 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f97e40a2910 0x7f97e40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:09.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.554+0000 7f97f7e00700 1 -- 172.21.15.117:0/1233034301 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f97f00cf1b0 con 0x7f9800102dd0 2024-03-19T03:33:09.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.554+0000 7f97fea00700 1 --2- 172.21.15.117:0/1233034301 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f97e40a2910 0x7f97e40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:09.555 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.554+0000 7f97fea00700 1 --2- 172.21.15.117:0/1233034301 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f97e40a2910 0x7f97e40a4dd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f97ec00b7b0 tx=0x7f97ec005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:09.560 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.560+0000 7f97f7e00700 1 -- 172.21.15.117:0/1233034301 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f97f0053740 con 0x7f9800102dd0 2024-03-19T03:33:09.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:09 smithi119 ceph-mon[66861]: pgmap v398: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:09.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:09 smithi117 ceph-mon[31838]: pgmap v398: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:09.831 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.830+0000 7f9806800700 1 -- 172.21.15.117:0/1233034301 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f97e0005920 con 0x7f9800102dd0 2024-03-19T03:33:09.832 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.831+0000 7f97f7e00700 1 -- 172.21.15.117:0/1233034301 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) ==== 76+0+3106 (secure 0 0 0) 0x7f97f009be00 con 0x7f9800102dd0 2024-03-19T03:33:09.833 INFO:teuthology.orchestra.run.smithi117.stdout:e28 2024-03-19T03:33:09.833 INFO:teuthology.orchestra.run.smithi117.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-19T03:33:09.833 INFO:teuthology.orchestra.run.smithi117.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-19T03:33:09.833 INFO:teuthology.orchestra.run.smithi117.stdout:legacy client fscid: 1 2024-03-19T03:33:09.833 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:33:09.833 INFO:teuthology.orchestra.run.smithi117.stdout:Filesystem 'cephfs' (1) 2024-03-19T03:33:09.833 INFO:teuthology.orchestra.run.smithi117.stdout:fs_name cephfs 2024-03-19T03:33:09.833 INFO:teuthology.orchestra.run.smithi117.stdout:epoch 28 2024-03-19T03:33:09.833 INFO:teuthology.orchestra.run.smithi117.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-19T03:33:09.833 INFO:teuthology.orchestra.run.smithi117.stdout:created 2024-03-19T03:30:35.081740+0000 2024-03-19T03:33:09.833 INFO:teuthology.orchestra.run.smithi117.stdout:modified 2024-03-19T03:31:09.995133+0000 2024-03-19T03:33:09.833 INFO:teuthology.orchestra.run.smithi117.stdout:tableserver 0 2024-03-19T03:33:09.834 INFO:teuthology.orchestra.run.smithi117.stdout:root 0 2024-03-19T03:33:09.834 INFO:teuthology.orchestra.run.smithi117.stdout:session_timeout 300 2024-03-19T03:33:09.834 INFO:teuthology.orchestra.run.smithi117.stdout:session_autoclose 300 2024-03-19T03:33:09.834 INFO:teuthology.orchestra.run.smithi117.stdout:max_file_size 1099511627776 2024-03-19T03:33:09.834 INFO:teuthology.orchestra.run.smithi117.stdout:max_xattr_size 65536 2024-03-19T03:33:09.834 INFO:teuthology.orchestra.run.smithi117.stdout:required_client_features {} 2024-03-19T03:33:09.834 INFO:teuthology.orchestra.run.smithi117.stdout:last_failure 0 2024-03-19T03:33:09.834 INFO:teuthology.orchestra.run.smithi117.stdout:last_failure_osd_epoch 0 2024-03-19T03:33:09.834 INFO:teuthology.orchestra.run.smithi117.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-19T03:33:09.834 INFO:teuthology.orchestra.run.smithi117.stdout:max_mds 5 2024-03-19T03:33:09.834 INFO:teuthology.orchestra.run.smithi117.stdout:in 0,1,2,3,4 2024-03-19T03:33:09.834 INFO:teuthology.orchestra.run.smithi117.stdout:up {0=24599,1=24581,2=24569,3=24545,4=24467} 2024-03-19T03:33:09.834 INFO:teuthology.orchestra.run.smithi117.stdout:failed 2024-03-19T03:33:09.834 INFO:teuthology.orchestra.run.smithi117.stdout:damaged 2024-03-19T03:33:09.835 INFO:teuthology.orchestra.run.smithi117.stdout:stopped 2024-03-19T03:33:09.835 INFO:teuthology.orchestra.run.smithi117.stdout:data_pools [3,4] 2024-03-19T03:33:09.835 INFO:teuthology.orchestra.run.smithi117.stdout:metadata_pool 2 2024-03-19T03:33:09.835 INFO:teuthology.orchestra.run.smithi117.stdout:inline_data disabled 2024-03-19T03:33:09.835 INFO:teuthology.orchestra.run.smithi117.stdout:balancer 2024-03-19T03:33:09.835 INFO:teuthology.orchestra.run.smithi117.stdout:bal_rank_mask -1 2024-03-19T03:33:09.835 INFO:teuthology.orchestra.run.smithi117.stdout:standby_count_wanted 0 2024-03-19T03:33:09.835 INFO:teuthology.orchestra.run.smithi117.stdout:qdb_cluster leader: 24599 members: 24581,24599,24569,24545,24467 2024-03-19T03:33:09.835 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.b{0:24599} state up:active seq 3 addr [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T03:33:09.835 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.i{0:24541} state up:standby-replay seq 1 addr [v2:172.21.15.178:6838/2655593383,v1:172.21.15.178:6839/2655593383] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T03:33:09.835 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.a{1:24581} state up:active seq 12 addr [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T03:33:09.835 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.c{1:24520} state up:standby-replay seq 1 addr [v2:172.21.15.178:6836/1770913538,v1:172.21.15.178:6837/1770913538] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T03:33:09.836 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.g{2:24569} state up:active seq 13 addr [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T03:33:09.836 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.e{2:24515} state up:standby-replay seq 1 addr [v2:172.21.15.119:6834/1686938941,v1:172.21.15.119:6835/1686938941] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T03:33:09.836 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.h{3:24545} state up:active seq 15 addr [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T03:33:09.836 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.f{3:24508} state up:standby-replay seq 1 addr [v2:172.21.15.178:6834/1579860804,v1:172.21.15.178:6835/1579860804] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T03:33:09.836 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.k{4:24467} state up:active seq 20 addr [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T03:33:09.836 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.l{4:24442} state up:standby-replay seq 1 addr [v2:172.21.15.178:6832/3751720259,v1:172.21.15.178:6833/3751720259] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T03:33:09.836 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:33:09.836 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:33:09.836 INFO:teuthology.orchestra.run.smithi117.stdout:Standby daemons: 2024-03-19T03:33:09.836 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:33:09.836 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.j{-1:14397} state up:standby seq 1 addr [v2:172.21.15.117:6834/54638259,v1:172.21.15.117:6835/54638259] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T03:33:09.836 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.d{-1:14409} state up:standby seq 1 addr [v2:172.21.15.117:6836/3608185039,v1:172.21.15.117:6837/3608185039] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T03:33:09.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.834+0000 7f9806800700 1 -- 172.21.15.117:0/1233034301 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f97e40a2910 msgr2=0x7f97e40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:09.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.834+0000 7f9806800700 1 --2- 172.21.15.117:0/1233034301 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f97e40a2910 0x7f97e40a4dd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f97ec00b7b0 tx=0x7f97ec005fd0 comp rx=0 tx=0).stop 2024-03-19T03:33:09.837 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.834+0000 7f9806800700 1 -- 172.21.15.117:0/1233034301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9800102dd0 msgr2=0x7f9800198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:09.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.834+0000 7f9806800700 1 --2- 172.21.15.117:0/1233034301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9800102dd0 0x7f9800198620 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f97f000e9c0 tx=0x7f97f000ecd0 comp rx=0 tx=0).stop 2024-03-19T03:33:09.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.834+0000 7f9806800700 1 -- 172.21.15.117:0/1233034301 shutdown_connections 2024-03-19T03:33:09.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.834+0000 7f9806800700 1 --2- 172.21.15.117:0/1233034301 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f97e40a2910 0x7f97e40a4dd0 secure :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f97ec00b7b0 tx=0x7f97ec005fd0 comp rx=0 tx=0).stop 2024-03-19T03:33:09.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.834+0000 7f9806800700 1 --2- 172.21.15.117:0/1233034301 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9800108030 0x7f980019fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:09.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.834+0000 7f9806800700 1 --2- 172.21.15.117:0/1233034301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9800102dd0 0x7f9800198620 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:09.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.834+0000 7f9806800700 1 --2- 172.21.15.117:0/1233034301 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9800105700 0x7f9800198b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:09.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.834+0000 7f9806800700 1 -- 172.21.15.117:0/1233034301 >> 172.21.15.117:0/1233034301 conn(0x7f98000fc7e0 msgr2=0x7f9800103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:09.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.835+0000 7f9806800700 1 -- 172.21.15.117:0/1233034301 shutdown_connections 2024-03-19T03:33:09.838 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:09.835+0000 7f9806800700 1 -- 172.21.15.117:0/1233034301 wait complete. 2024-03-19T03:33:09.838 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 28 2024-03-19T03:33:09.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:09 smithi178 ceph-mon[67578]: pgmap v398: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:10.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:10 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1233034301' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-19T03:33:10.733 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-19T03:33:10.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:10 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1233034301' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-19T03:33:10.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:10 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1233034301' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-19T03:33:11.206 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:33:11.706 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:11 smithi117 ceph-mon[31838]: pgmap v399: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:11.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:11 smithi119 ceph-mon[66861]: pgmap v399: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:11.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:11 smithi178 ceph-mon[67578]: pgmap v399: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:13.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.465+0000 7efc93e00700 1 -- 172.21.15.117:0/2996775407 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc8c103f00 msgr2=0x7efc8c10a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:13.466 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.465+0000 7efc93e00700 1 --2- 172.21.15.117:0/2996775407 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc8c103f00 0x7efc8c10a7b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7efc84009fc0 tx=0x7efc8400b330 comp rx=0 tx=0).stop 2024-03-19T03:33:13.466 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.465+0000 7efc93e00700 1 -- 172.21.15.117:0/2996775407 shutdown_connections 2024-03-19T03:33:13.466 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.465+0000 7efc93e00700 1 --2- 172.21.15.117:0/2996775407 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc8c103f00 0x7efc8c10a7b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:13.466 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.465+0000 7efc93e00700 1 --2- 172.21.15.117:0/2996775407 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efc8c102360 0x7efc8c102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:13.466 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.465+0000 7efc93e00700 1 --2- 172.21.15.117:0/2996775407 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc8c103560 0x7efc8c1039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:13.466 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.465+0000 7efc93e00700 1 -- 172.21.15.117:0/2996775407 >> 172.21.15.117:0/2996775407 conn(0x7efc8c0fd920 msgr2=0x7efc8c0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:13.466 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.465+0000 7efc93e00700 1 -- 172.21.15.117:0/2996775407 shutdown_connections 2024-03-19T03:33:13.466 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.465+0000 7efc93e00700 1 -- 172.21.15.117:0/2996775407 wait complete. 2024-03-19T03:33:13.466 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.467+0000 7efc93e00700 1 Processor -- start 2024-03-19T03:33:13.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.467+0000 7efc93e00700 1 -- start start 2024-03-19T03:33:13.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.467+0000 7efc93e00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc8c102360 0x7efc8c198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:13.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.467+0000 7efc93e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efc8c103560 0x7efc8c198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:13.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.467+0000 7efc93e00700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc8c103f00 0x7efc8c19fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:13.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.467+0000 7efc93e00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7efc8c10df90 con 0x7efc8c103560 2024-03-19T03:33:13.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.467+0000 7efc93e00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7efc8c10de10 con 0x7efc8c102360 2024-03-19T03:33:13.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.467+0000 7efc93e00700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7efc8c10e110 con 0x7efc8c103f00 2024-03-19T03:33:13.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.468+0000 7efc91400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc8c102360 0x7efc8c198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:13.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.468+0000 7efc91400700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc8c102360 0x7efc8c198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:54448/0 (socket says 172.21.15.117:54448) 2024-03-19T03:33:13.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.468+0000 7efc91400700 1 -- 172.21.15.117:0/1375953529 learned_addr learned my addr 172.21.15.117:0/1375953529 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:13.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.468+0000 7efc90a00700 1 --2- 172.21.15.117:0/1375953529 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efc8c103560 0x7efc8c198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:13.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.468+0000 7efc91e00700 1 --2- 172.21.15.117:0/1375953529 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc8c103f00 0x7efc8c19fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:13.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.468+0000 7efc91400700 1 -- 172.21.15.117:0/1375953529 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc8c103f00 msgr2=0x7efc8c19fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:13.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.468+0000 7efc91400700 1 --2- 172.21.15.117:0/1375953529 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc8c103f00 0x7efc8c19fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:13.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.468+0000 7efc91400700 1 -- 172.21.15.117:0/1375953529 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efc8c103560 msgr2=0x7efc8c198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:13.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.468+0000 7efc91400700 1 --2- 172.21.15.117:0/1375953529 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efc8c103560 0x7efc8c198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:13.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.468+0000 7efc91400700 1 -- 172.21.15.117:0/1375953529 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc8c074540 con 0x7efc8c102360 2024-03-19T03:33:13.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.468+0000 7efc90a00700 1 --2- 172.21.15.117:0/1375953529 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efc8c103560 0x7efc8c198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:33:13.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.468+0000 7efc91400700 1 --2- 172.21.15.117:0/1375953529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc8c102360 0x7efc8c198510 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7efc8000ebf0 tx=0x7efc8000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:13.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.469+0000 7efc8a000700 1 -- 172.21.15.117:0/1375953529 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8000ce10 con 0x7efc8c102360 2024-03-19T03:33:13.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.469+0000 7efc8a000700 1 -- 172.21.15.117:0/1375953529 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc80010910 con 0x7efc8c102360 2024-03-19T03:33:13.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.469+0000 7efc93e00700 1 -- 172.21.15.117:0/1375953529 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc8c074830 con 0x7efc8c102360 2024-03-19T03:33:13.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.469+0000 7efc8a000700 1 -- 172.21.15.117:0/1375953529 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc80018aa0 con 0x7efc8c102360 2024-03-19T03:33:13.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.469+0000 7efc93e00700 1 -- 172.21.15.117:0/1375953529 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc8c074dc0 con 0x7efc8c102360 2024-03-19T03:33:13.470 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.470+0000 7efc93e00700 1 -- 172.21.15.117:0/1375953529 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc8c068fd0 con 0x7efc8c102360 2024-03-19T03:33:13.472 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.472+0000 7efc8a000700 1 -- 172.21.15.117:0/1375953529 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7efc8001f020 con 0x7efc8c102360 2024-03-19T03:33:13.473 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.473+0000 7efc8a000700 1 --2- 172.21.15.117:0/1375953529 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7efc740a2910 0x7efc740a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:13.473 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.474+0000 7efc8a000700 1 -- 172.21.15.117:0/1375953529 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7efc800cfb60 con 0x7efc8c102360 2024-03-19T03:33:13.474 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.474+0000 7efc90a00700 1 --2- 172.21.15.117:0/1375953529 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7efc740a2910 0x7efc740a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:13.474 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.474+0000 7efc90a00700 1 --2- 172.21.15.117:0/1375953529 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7efc740a2910 0x7efc740a4dd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7efc78009cb0 tx=0x7efc78009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:13.475 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.475+0000 7efc8a000700 1 -- 172.21.15.117:0/1375953529 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efc80026030 con 0x7efc8c102360 2024-03-19T03:33:13.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:13 smithi119 ceph-mon[66861]: pgmap v400: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T03:33:13.734 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.735+0000 7efc93e00700 1 -- 172.21.15.117:0/1375953529 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7efc8c076260 con 0x7efc8c102360 2024-03-19T03:33:13.735 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:13 smithi117 ceph-mon[31838]: pgmap v400: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T03:33:13.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.736+0000 7efc8a000700 1 -- 172.21.15.117:0/1375953529 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) ==== 56+0+5769 (secure 0 0 0) 0x7efc8009c7b0 con 0x7efc8c102360 2024-03-19T03:33:13.737 INFO:teuthology.orchestra.run.smithi117.stdout:epoch 93 2024-03-19T03:33:13.737 INFO:teuthology.orchestra.run.smithi117.stdout:fsid a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:33:13.737 INFO:teuthology.orchestra.run.smithi117.stdout:created 2024-03-19T03:21:30.427136+0000 2024-03-19T03:33:13.738 INFO:teuthology.orchestra.run.smithi117.stdout:modified 2024-03-19T03:32:40.453890+0000 2024-03-19T03:33:13.738 INFO:teuthology.orchestra.run.smithi117.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-19T03:33:13.738 INFO:teuthology.orchestra.run.smithi117.stdout:crush_version 27 2024-03-19T03:33:13.738 INFO:teuthology.orchestra.run.smithi117.stdout:full_ratio 0.95 2024-03-19T03:33:13.738 INFO:teuthology.orchestra.run.smithi117.stdout:backfillfull_ratio 0.9 2024-03-19T03:33:13.738 INFO:teuthology.orchestra.run.smithi117.stdout:nearfull_ratio 0.85 2024-03-19T03:33:13.738 INFO:teuthology.orchestra.run.smithi117.stdout:require_min_compat_client luminous 2024-03-19T03:33:13.738 INFO:teuthology.orchestra.run.smithi117.stdout:min_compat_client luminous 2024-03-19T03:33:13.738 INFO:teuthology.orchestra.run.smithi117.stdout:require_osd_release squid 2024-03-19T03:33:13.738 INFO:teuthology.orchestra.run.smithi117.stdout:stretch_mode_enabled false 2024-03-19T03:33:13.738 INFO:teuthology.orchestra.run.smithi117.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-19T03:33:13.738 INFO:teuthology.orchestra.run.smithi117.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-19T03:33:13.738 INFO:teuthology.orchestra.run.smithi117.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-19T03:33:13.739 INFO:teuthology.orchestra.run.smithi117.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-19T03:33:13.739 INFO:teuthology.orchestra.run.smithi117.stdout:max_osd 12 2024-03-19T03:33:13.739 INFO:teuthology.orchestra.run.smithi117.stdout:osd.0 up in weight 1 up_from 8 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6802/3014934900,v1:172.21.15.117:6803/3014934900] [v2:172.21.15.117:6804/3014934900,v1:172.21.15.117:6805/3014934900] exists,up b32eb0ee-9d4b-4748-87ec-eda02c907415 2024-03-19T03:33:13.739 INFO:teuthology.orchestra.run.smithi117.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6800/2425927341,v1:172.21.15.119:6801/2425927341] [v2:172.21.15.119:6802/2425927341,v1:172.21.15.119:6803/2425927341] exists,up e07585be-c016-4466-b07c-abbc22a58de4 2024-03-19T03:33:13.739 INFO:teuthology.orchestra.run.smithi117.stdout:osd.2 up in weight 1 up_from 18 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.178:6800/3733812989,v1:172.21.15.178:6801/3733812989] [v2:172.21.15.178:6802/3733812989,v1:172.21.15.178:6803/3733812989] exists,up ede9edda-3b9f-41ca-ad04-5dcb738ab757 2024-03-19T03:33:13.739 INFO:teuthology.orchestra.run.smithi117.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6810/2178310135,v1:172.21.15.117:6811/2178310135] [v2:172.21.15.117:6812/2178310135,v1:172.21.15.117:6813/2178310135] exists,up f81c41e1-3a69-40db-a90f-58151286b081 2024-03-19T03:33:13.739 INFO:teuthology.orchestra.run.smithi117.stdout:osd.4 up in weight 1 up_from 33 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6808/3455398086,v1:172.21.15.119:6809/3455398086] [v2:172.21.15.119:6810/3455398086,v1:172.21.15.119:6811/3455398086] exists,up caf7ce86-caf0-4eaf-bbab-bd2d93e63c33 2024-03-19T03:33:13.739 INFO:teuthology.orchestra.run.smithi117.stdout:osd.5 up in weight 1 up_from 38 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.178:6808/1259631912,v1:172.21.15.178:6809/1259631912] [v2:172.21.15.178:6810/1259631912,v1:172.21.15.178:6811/1259631912] exists,up 20ba0b8a-2e0f-4ea7-b435-430e44766200 2024-03-19T03:33:13.739 INFO:teuthology.orchestra.run.smithi117.stdout:osd.6 up in weight 1 up_from 46 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6818/864146757,v1:172.21.15.117:6819/864146757] [v2:172.21.15.117:6820/864146757,v1:172.21.15.117:6821/864146757] exists,up 75c2ddd7-f837-42e0-ac47-8cfbcf97a52a 2024-03-19T03:33:13.739 INFO:teuthology.orchestra.run.smithi117.stdout:osd.7 up in weight 1 up_from 51 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6816/1273365545,v1:172.21.15.119:6817/1273365545] [v2:172.21.15.119:6818/1273365545,v1:172.21.15.119:6819/1273365545] exists,up 929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c 2024-03-19T03:33:13.739 INFO:teuthology.orchestra.run.smithi117.stdout:osd.8 up in weight 1 up_from 59 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.178:6816/1059587501,v1:172.21.15.178:6817/1059587501] [v2:172.21.15.178:6818/1059587501,v1:172.21.15.178:6819/1059587501] exists,up fd371461-a538-4781-92e1-5eead4ca3864 2024-03-19T03:33:13.739 INFO:teuthology.orchestra.run.smithi117.stdout:osd.9 up in weight 1 up_from 67 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6826/3653875600,v1:172.21.15.117:6827/3653875600] [v2:172.21.15.117:6828/3653875600,v1:172.21.15.117:6829/3653875600] exists,up 0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8 2024-03-19T03:33:13.740 INFO:teuthology.orchestra.run.smithi117.stdout:osd.10 up in weight 1 up_from 72 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.119:6824/1050976194,v1:172.21.15.119:6825/1050976194] [v2:172.21.15.119:6826/1050976194,v1:172.21.15.119:6827/1050976194] exists,up de2049da-d19e-4386-afbf-fd6b5fddbcba 2024-03-19T03:33:13.740 INFO:teuthology.orchestra.run.smithi117.stdout:osd.11 up in weight 1 up_from 80 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.178:6824/2915080368,v1:172.21.15.178:6825/2915080368] [v2:172.21.15.178:6826/2915080368,v1:172.21.15.178:6827/2915080368] exists,up 3cccc0f9-d273-4b6b-9b8e-c926371472dd 2024-03-19T03:33:13.740 INFO:teuthology.orchestra.run.smithi117.stdout:pg_upmap_items 2.5 [8,11] 2024-03-19T03:33:13.740 INFO:teuthology.orchestra.run.smithi117.stdout:pg_upmap_items 2.b [2,4] 2024-03-19T03:33:13.740 INFO:teuthology.orchestra.run.smithi117.stdout:pg_upmap_items 2.14 [2,4] 2024-03-19T03:33:13.740 INFO:teuthology.orchestra.run.smithi117.stdout:pg_upmap_items 2.1d [8,11] 2024-03-19T03:33:13.740 INFO:teuthology.orchestra.run.smithi117.stdout:pg_upmap_items 3.e [4,6] 2024-03-19T03:33:13.740 INFO:teuthology.orchestra.run.smithi117.stdout:pg_upmap_items 3.14 [1,9] 2024-03-19T03:33:13.740 INFO:teuthology.orchestra.run.smithi117.stdout:pg_upmap_items 3.17 [4,7,5,8] 2024-03-19T03:33:13.740 INFO:teuthology.orchestra.run.smithi117.stdout:pg_upmap_items 3.1d [4,8] 2024-03-19T03:33:13.740 INFO:teuthology.orchestra.run.smithi117.stdout:pg_upmap_items 3.1e [3,9] 2024-03-19T03:33:13.740 INFO:teuthology.orchestra.run.smithi117.stdout:pg_upmap_items 3.27 [3,9] 2024-03-19T03:33:13.740 INFO:teuthology.orchestra.run.smithi117.stdout:pg_upmap_items 3.39 [5,6] 2024-03-19T03:33:13.740 INFO:teuthology.orchestra.run.smithi117.stdout:pg_upmap_items 4.5 [5,8] 2024-03-19T03:33:13.740 INFO:teuthology.orchestra.run.smithi117.stdout:pg_upmap_items 4.f [9,10] 2024-03-19T03:33:13.741 INFO:teuthology.orchestra.run.smithi117.stdout:pg_upmap_items 4.16 [9,8] 2024-03-19T03:33:13.741 INFO:teuthology.orchestra.run.smithi117.stdout:pg_upmap_items 4.17 [9,8] 2024-03-19T03:33:13.741 INFO:teuthology.orchestra.run.smithi117.stdout:blocklist 172.21.15.117:0/185684447 expires 2024-03-20T03:22:38.134578+0000 2024-03-19T03:33:13.741 INFO:teuthology.orchestra.run.smithi117.stdout:blocklist 172.21.15.117:0/361367101 expires 2024-03-20T03:22:38.134578+0000 2024-03-19T03:33:13.741 INFO:teuthology.orchestra.run.smithi117.stdout:blocklist 172.21.15.117:6801/2788067603 expires 2024-03-20T03:22:38.134578+0000 2024-03-19T03:33:13.741 INFO:teuthology.orchestra.run.smithi117.stdout:blocklist 172.21.15.117:6801/1570899077 expires 2024-03-20T03:22:03.905461+0000 2024-03-19T03:33:13.741 INFO:teuthology.orchestra.run.smithi117.stdout:blocklist 172.21.15.117:0/3506200675 expires 2024-03-20T03:22:38.134578+0000 2024-03-19T03:33:13.741 INFO:teuthology.orchestra.run.smithi117.stdout:blocklist 172.21.15.117:6800/1570899077 expires 2024-03-20T03:22:03.905461+0000 2024-03-19T03:33:13.741 INFO:teuthology.orchestra.run.smithi117.stdout:blocklist 172.21.15.117:0/3283471541 expires 2024-03-20T03:22:03.905461+0000 2024-03-19T03:33:13.741 INFO:teuthology.orchestra.run.smithi117.stdout:blocklist 172.21.15.117:0/3750920169 expires 2024-03-20T03:22:03.905461+0000 2024-03-19T03:33:13.741 INFO:teuthology.orchestra.run.smithi117.stdout:blocklist 172.21.15.117:0/2780813298 expires 2024-03-20T03:22:03.905461+0000 2024-03-19T03:33:13.741 INFO:teuthology.orchestra.run.smithi117.stdout:blocklist 172.21.15.117:6800/2788067603 expires 2024-03-20T03:22:38.134578+0000 2024-03-19T03:33:13.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.740+0000 7efc93e00700 1 -- 172.21.15.117:0/1375953529 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7efc740a2910 msgr2=0x7efc740a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:13.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.740+0000 7efc93e00700 1 --2- 172.21.15.117:0/1375953529 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7efc740a2910 0x7efc740a4dd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7efc78009cb0 tx=0x7efc78009260 comp rx=0 tx=0).stop 2024-03-19T03:33:13.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.740+0000 7efc93e00700 1 -- 172.21.15.117:0/1375953529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc8c102360 msgr2=0x7efc8c198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:13.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.740+0000 7efc93e00700 1 --2- 172.21.15.117:0/1375953529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc8c102360 0x7efc8c198510 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7efc8000ebf0 tx=0x7efc8000c2d0 comp rx=0 tx=0).stop 2024-03-19T03:33:13.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.741+0000 7efc93e00700 1 -- 172.21.15.117:0/1375953529 shutdown_connections 2024-03-19T03:33:13.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.741+0000 7efc93e00700 1 --2- 172.21.15.117:0/1375953529 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7efc740a2910 0x7efc740a4dd0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:13.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.741+0000 7efc93e00700 1 --2- 172.21.15.117:0/1375953529 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc8c103f00 0x7efc8c19fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:13.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.741+0000 7efc93e00700 1 --2- 172.21.15.117:0/1375953529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc8c102360 0x7efc8c198510 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:13.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.741+0000 7efc93e00700 1 --2- 172.21.15.117:0/1375953529 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efc8c103560 0x7efc8c198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:13.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.741+0000 7efc93e00700 1 -- 172.21.15.117:0/1375953529 >> 172.21.15.117:0/1375953529 conn(0x7efc8c0fd920 msgr2=0x7efc8c106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:13.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.741+0000 7efc93e00700 1 -- 172.21.15.117:0/1375953529 shutdown_connections 2024-03-19T03:33:13.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:13.741+0000 7efc93e00700 1 -- 172.21.15.117:0/1375953529 wait complete. 2024-03-19T03:33:13.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:13 smithi178 ceph-mon[67578]: pgmap v400: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T03:33:14.530 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-19T03:33:14.543 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:33:14.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:14 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1375953529' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-19T03:33:14.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.752+0000 7f2b0e000640 1 -- 172.21.15.117:0/3720717610 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b08102a30 msgr2=0x7f2b08102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:14.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.752+0000 7f2b0e000640 1 --2- 172.21.15.117:0/3720717610 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b08102a30 0x7f2b08102eb0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2af4009900 tx=0x7f2af402bc30 comp rx=0 tx=0).stop 2024-03-19T03:33:14.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.752+0000 7f2b0e000640 1 -- 172.21.15.117:0/3720717610 shutdown_connections 2024-03-19T03:33:14.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.752+0000 7f2b0e000640 1 --2- 172.21.15.117:0/3720717610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b081033f0 0x7f2b08109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:14.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.752+0000 7f2b0e000640 1 --2- 172.21.15.117:0/3720717610 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b08102a30 0x7f2b08102eb0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:14.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.752+0000 7f2b0e000640 1 --2- 172.21.15.117:0/3720717610 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b08101840 0x7f2b08101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:14.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.752+0000 7f2b0e000640 1 -- 172.21.15.117:0/3720717610 >> 172.21.15.117:0/3720717610 conn(0x7f2b080fcdd0 msgr2=0x7f2b080ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:14.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.752+0000 7f2b0e000640 1 -- 172.21.15.117:0/3720717610 shutdown_connections 2024-03-19T03:33:14.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.752+0000 7f2b0e000640 1 -- 172.21.15.117:0/3720717610 wait complete. 2024-03-19T03:33:14.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.753+0000 7f2b0e000640 1 Processor -- start 2024-03-19T03:33:14.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.754+0000 7f2b0e000640 1 -- start start 2024-03-19T03:33:14.756 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.754+0000 7f2b0e000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b08101840 0x7f2b08197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:14.756 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.754+0000 7f2b0e000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b08102a30 0x7f2b08198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:14.756 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.754+0000 7f2b0e000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b081033f0 0x7f2b0819f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:14.756 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.754+0000 7f2b0e000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0810d4c0 con 0x7f2b081033f0 2024-03-19T03:33:14.756 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.754+0000 7f2b0e000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0810d340 con 0x7f2b08102a30 2024-03-19T03:33:14.756 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.754+0000 7f2b0e000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0810d640 con 0x7f2b08101840 2024-03-19T03:33:14.756 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.754+0000 7f2b06a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b08102a30 0x7f2b08198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:14.756 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.754+0000 7f2b07400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b08101840 0x7f2b08197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:14.756 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.754+0000 7f2b06a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b08102a30 0x7f2b08198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:54474/0 (socket says 172.21.15.117:54474) 2024-03-19T03:33:14.756 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.754+0000 7f2b07400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b08101840 0x7f2b08197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:58910/0 (socket says 172.21.15.117:58910) 2024-03-19T03:33:14.756 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.754+0000 7f2b06a00640 1 -- 172.21.15.117:0/2649926143 learned_addr learned my addr 172.21.15.117:0/2649926143 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:14.756 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.754+0000 7f2b07e00640 1 --2- 172.21.15.117:0/2649926143 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b081033f0 0x7f2b0819f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:14.756 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.755+0000 7f2b07400640 1 -- 172.21.15.117:0/2649926143 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b08102a30 msgr2=0x7f2b08198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:14.757 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.755+0000 7f2b07400640 1 --2- 172.21.15.117:0/2649926143 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b08102a30 0x7f2b08198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:14.757 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.755+0000 7f2b07400640 1 -- 172.21.15.117:0/2649926143 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b081033f0 msgr2=0x7f2b0819f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:14.757 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.755+0000 7f2b07400640 1 --2- 172.21.15.117:0/2649926143 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b081033f0 0x7f2b0819f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:14.757 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.755+0000 7f2b07400640 1 -- 172.21.15.117:0/2649926143 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b0819f750 con 0x7f2b08101840 2024-03-19T03:33:14.757 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.755+0000 7f2b07e00640 1 --2- 172.21.15.117:0/2649926143 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b081033f0 0x7f2b0819f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:33:14.757 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.755+0000 7f2b07400640 1 --2- 172.21.15.117:0/2649926143 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b08101840 0x7f2b08197b30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2af800e9c0 tx=0x7f2af800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:14.757 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.755+0000 7f2affe00640 1 -- 172.21.15.117:0/2649926143 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af800ccc0 con 0x7f2b08101840 2024-03-19T03:33:14.757 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.755+0000 7f2affe00640 1 -- 172.21.15.117:0/2649926143 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2af80035f0 con 0x7f2b08101840 2024-03-19T03:33:14.757 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.755+0000 7f2affe00640 1 -- 172.21.15.117:0/2649926143 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af800ce60 con 0x7f2b08101840 2024-03-19T03:33:14.757 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.755+0000 7f2b0e000640 1 -- 172.21.15.117:0/2649926143 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b0819fa40 con 0x7f2b08101840 2024-03-19T03:33:14.757 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.756+0000 7f2b0e000640 1 -- 172.21.15.117:0/2649926143 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b0819ff10 con 0x7f2b08101840 2024-03-19T03:33:14.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.757+0000 7f2b0e000640 1 -- 172.21.15.117:0/2649926143 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b08068a60 con 0x7f2b08101840 2024-03-19T03:33:14.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.758+0000 7f2affe00640 1 -- 172.21.15.117:0/2649926143 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f2af80037a0 con 0x7f2b08101840 2024-03-19T03:33:14.760 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.761+0000 7f2affe00640 1 --2- 172.21.15.117:0/2649926143 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f2aec0a2800 0x7f2aec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:14.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.761+0000 7f2affe00640 1 -- 172.21.15.117:0/2649926143 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f2af80cf470 con 0x7f2b08101840 2024-03-19T03:33:14.761 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.761+0000 7f2b06a00640 1 --2- 172.21.15.117:0/2649926143 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f2aec0a2800 0x7f2aec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:14.762 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.762+0000 7f2b06a00640 1 --2- 172.21.15.117:0/2649926143 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f2aec0a2800 0x7f2aec0a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2af4000c00 tx=0x7f2af402c880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:14.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.764+0000 7f2affe00640 1 -- 172.21.15.117:0/2649926143 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2af809c0c0 con 0x7f2b08101840 2024-03-19T03:33:14.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:14 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1375953529' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-19T03:33:14.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:14 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1375953529' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-19T03:33:14.985 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.985+0000 7f2b0e000640 1 -- 172.21.15.117:0/2649926143 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2b08062420 con 0x7f2b08101840 2024-03-19T03:33:14.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.988+0000 7f2affe00640 1 -- 172.21.15.117:0/2649926143 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24989 (secure 0 0 0) 0x7f2af809b9c0 con 0x7f2b08101840 2024-03-19T03:33:14.988 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:33:14.988 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":93,"fsid":"a3a82456-e59f-11ee-95c9-87774f69a715","created":"2024-03-19T03:21:30.427136+0000","modified":"2024-03-19T03:32:40.453890+0000","last_up_change":"2024-03-19T03:29:32.967505+0000","last_in_change":"2024-03-19T03:29:12.848740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:25:18.824270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T03:30:33.137170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T03:30:34.598838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T03:30:38.230023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b32eb0ee-9d4b-4748-87ec-eda02c907415","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6803","nonce":3014934900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6805","nonce":3014934900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6809","nonce":3014934900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6807","nonce":3014934900}]},"public_addr":"172.21.15.117:6803/3014934900","cluster_addr":"172.21.15.117:6805/3014934900","heartbeat_back_addr":"172.21.15.117:6809/3014934900","heartbeat_front_addr":"172.21.15.117:6807/3014934900","state":["exists","up"]},{"osd":1,"uuid":"e07585be-c016-4466-b07c-abbc22a58de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6801","nonce":2425927341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6803","nonce":2425927341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6807","nonce":2425927341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6805","nonce":2425927341}]},"public_addr":"172.21.15.119:6801/2425927341","cluster_addr":"172.21.15.119:6803/2425927341","heartbeat_back_addr":"172.21.15.119:6807/2425927341","heartbeat_front_addr":"172.21.15.119:6805/2425927341","state":["exists","up"]},{"osd":2,"uuid":"ede9edda-3b9f-41ca-ad04-5dcb738ab757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6801","nonce":3733812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6803","nonce":3733812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6807","nonce":3733812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6805","nonce":3733812989}]},"public_addr":"172.21.15.178:6801/3733812989","cluster_addr":"172.21.15.178:6803/3733812989","heartbeat_back_addr":"172.21.15.178:6807/3733812989","heartbeat_front_addr":"172.21.15.178:6805/3733812989","state":["exists","up"]},{"osd":3,"uuid":"f81c41e1-3a69-40db-a90f-58151286b081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6811","nonce":2178310135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6813","nonce":2178310135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6817","nonce":2178310135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6815","nonce":2178310135}]},"public_addr":"172.21.15.117:6811/2178310135","cluster_addr":"172.21.15.117:6813/2178310135","heartbeat_back_addr":"172.21.15.117:6817/2178310135","heartbeat_front_addr":"172.21.15.117:6815/2178310135","state":["exists","up"]},{"osd":4,"uuid":"caf7ce86-caf0-4eaf-bbab-bd2d93e63c33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6809","nonce":3455398086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6811","nonce":3455398086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6815","nonce":3455398086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6813","nonce":3455398086}]},"public_addr":"172.21.15.119:6809/3455398086","cluster_addr":"172.21.15.119:6811/3455398086","heartbeat_back_addr":"172.21.15.119:6815/3455398086","heartbeat_front_addr":"172.21.15.119:6813/3455398086","state":["exists","up"]},{"osd":5,"uuid":"20ba0b8a-2e0f-4ea7-b435-430e44766200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6809","nonce":1259631912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6811","nonce":1259631912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6815","nonce":1259631912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6813","nonce":1259631912}]},"public_addr":"172.21.15.178:6809/1259631912","cluster_addr":"172.21.15.178:6811/1259631912","heartbeat_back_addr":"172.21.15.178:6815/1259631912","heartbeat_front_addr":"172.21.15.178:6813/1259631912","state":["exists","up"]},{"osd":6,"uuid":"75c2ddd7-f837-42e0-ac47-8cfbcf97a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6819","nonce":864146757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6821","nonce":864146757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6825","nonce":864146757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6823","nonce":864146757}]},"public_addr":"172.21.15.117:6819/864146757","cluster_addr":"172.21.15.117:6821/864146757","heartbeat_back_addr":"172.21.15.117:6825/864146757","heartbeat_front_addr":"172.21.15.117:6823/864146757","state":["exists","up"]},{"osd":7,"uuid":"929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6817","nonce":1273365545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6819","nonce":1273365545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6823","nonce":1273365545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6821","nonce":1273365545}]},"public_addr":"172.21.15.119:6817/1273365545","cluster_addr":"172.21.15.119:6819/1273365545","heartbeat_back_addr":"172.21.15.119:6823/1273365545","heartbeat_front_addr":"172.21.15.119:6821/1273365545","state":["exists","up"]},{"osd":8,"uuid":"fd371461-a538-4781-92e1-5eead4ca3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6817","nonce":1059587501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6819","nonce":1059587501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6823","nonce":1059587501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6821","nonce":1059587501}]},"public_addr":"172.21.15.178:6817/1059587501","cluster_addr":"172.21.15.178:6819/1059587501","heartbeat_back_addr":"172.21.15.178:6823/1059587501","heartbeat_front_addr":"172.21.15.178:6821/1059587501","state":["exists","up"]},{"osd":9,"uuid":"0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6827","nonce":3653875600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6829","nonce":3653875600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6833","nonce":3653875600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6831","nonce":3653875600}]},"public_addr":"172.21.15.117:6827/3653875600","cluster_addr":"172.21.15.117:6829/3653875600","heartbeat_back_addr":"172.21.15.117:6833/3653875600","heartbeat_front_addr":"172.21.15.117:6831/3653875600","state":["exists","up"]},{"osd":10,"uuid":"de2049da-d19e-4386-afbf-fd6b5fddbcba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6825","nonce":1050976194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6827","nonce":1050976194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6831","nonce":1050976194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6829","nonce":1050976194}]},"public_addr":"172.21.15.119:6825/1050976194","cluster_addr":"172.21.15.119:6827/1050976194","heartbeat_back_addr":"172.21.15.119:6831/1050976194","heartbeat_front_addr":"172.21.15.119:6829/1050976194","state":["exists","up"]},{"osd":11,"uuid":"3cccc0f9-d273-4b6b-9b8e-c926371472dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6825","nonce":2915080368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6827","nonce":2915080368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6831","nonce":2915080368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6829","nonce":2915080368}]},"public_addr":"172.21.15.178:6825/2915080368","cluster_addr":"172.21.15.178:6827/2915080368","heartbeat_back_addr":"172.21.15.178:6831/2915080368","heartbeat_front_addr":"172.21.15.178:6829/2915080368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:27.876274+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:53.094706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:16.598813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:48.160419+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:14.048925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:38.814304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:12.542328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:38.942747+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:04.488963+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:38.196053+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:05.254728+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:30.874915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7},{"from":5,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.117:0/185684447":"2024-03-20T03:22:38.134578+0000","172.21.15.117:0/361367101":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/2788067603":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3506200675":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6800/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3283471541":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3750920169":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/2780813298":"2024-03-20T03:22:03.905461+0000","172.21.15.117:6800/2788067603":"2024-03-20T03:22:38.134578+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:33:14.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.990+0000 7f2b0e000640 1 -- 172.21.15.117:0/2649926143 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f2aec0a2800 msgr2=0x7f2aec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:14.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.990+0000 7f2b0e000640 1 --2- 172.21.15.117:0/2649926143 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f2aec0a2800 0x7f2aec0a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2af4000c00 tx=0x7f2af402c880 comp rx=0 tx=0).stop 2024-03-19T03:33:14.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.990+0000 7f2b0e000640 1 -- 172.21.15.117:0/2649926143 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b08101840 msgr2=0x7f2b08197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:14.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.990+0000 7f2b0e000640 1 --2- 172.21.15.117:0/2649926143 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b08101840 0x7f2b08197b30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2af800e9c0 tx=0x7f2af800ee80 comp rx=0 tx=0).stop 2024-03-19T03:33:14.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.990+0000 7f2b0e000640 1 -- 172.21.15.117:0/2649926143 shutdown_connections 2024-03-19T03:33:14.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.990+0000 7f2b0e000640 1 --2- 172.21.15.117:0/2649926143 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b081033f0 0x7f2b0819f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:14.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.990+0000 7f2b0e000640 1 --2- 172.21.15.117:0/2649926143 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f2aec0a2800 0x7f2aec0a4cc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:14.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.990+0000 7f2b0e000640 1 --2- 172.21.15.117:0/2649926143 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b08102a30 0x7f2b08198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:14.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.990+0000 7f2b0e000640 1 --2- 172.21.15.117:0/2649926143 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b08101840 0x7f2b08197b30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:14.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.990+0000 7f2b0e000640 1 -- 172.21.15.117:0/2649926143 >> 172.21.15.117:0/2649926143 conn(0x7f2b080fcdd0 msgr2=0x7f2b08105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:14.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.990+0000 7f2b0e000640 1 -- 172.21.15.117:0/2649926143 shutdown_connections 2024-03-19T03:33:14.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:14.990+0000 7f2b0e000640 1 -- 172.21.15.117:0/2649926143 wait complete. 2024-03-19T03:33:15.019 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T03:25:18.824270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T03:30:33.137170+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T03:30:34.598838+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T03:30:38.230023+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T03:33:15.019 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T03:33:15.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.237+0000 7fc6b9600640 1 -- 172.21.15.117:0/1085785094 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6b41017f0 msgr2=0x7fc6b4101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:15.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.237+0000 7fc6b9600640 1 --2- 172.21.15.117:0/1085785094 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6b41017f0 0x7fc6b4101c70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0009900 tx=0x7fc6a002bc30 comp rx=0 tx=0).stop 2024-03-19T03:33:15.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.237+0000 7fc6b9600640 1 -- 172.21.15.117:0/1085785094 shutdown_connections 2024-03-19T03:33:15.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.237+0000 7fc6b9600640 1 --2- 172.21.15.117:0/1085785094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6b41021b0 0x7fc6b4108a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:15.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.237+0000 7fc6b9600640 1 --2- 172.21.15.117:0/1085785094 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6b41017f0 0x7fc6b4101c70 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:15.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.237+0000 7fc6b9600640 1 --2- 172.21.15.117:0/1085785094 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6b4100600 0x7fc6b4100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:15.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.237+0000 7fc6b9600640 1 -- 172.21.15.117:0/1085785094 >> 172.21.15.117:0/1085785094 conn(0x7fc6b40fbb90 msgr2=0x7fc6b40fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:15.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.237+0000 7fc6b9600640 1 -- 172.21.15.117:0/1085785094 shutdown_connections 2024-03-19T03:33:15.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.237+0000 7fc6b9600640 1 -- 172.21.15.117:0/1085785094 wait complete. 2024-03-19T03:33:15.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.239+0000 7fc6b9600640 1 Processor -- start 2024-03-19T03:33:15.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.239+0000 7fc6b9600640 1 -- start start 2024-03-19T03:33:15.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.239+0000 7fc6b9600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6b4100600 0x7fc6b4197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:15.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.239+0000 7fc6b9600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6b41017f0 0x7fc6b4197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:15.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.239+0000 7fc6b9600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6b41021b0 0x7fc6b419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:15.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.239+0000 7fc6b9600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b410c260 con 0x7fc6b4100600 2024-03-19T03:33:15.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.239+0000 7fc6b9600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b410c0e0 con 0x7fc6b41021b0 2024-03-19T03:33:15.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.239+0000 7fc6b9600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b410c3e0 con 0x7fc6b41017f0 2024-03-19T03:33:15.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.240+0000 7fc6b3e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6b4100600 0x7fc6b4197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:15.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.240+0000 7fc6b8c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6b41021b0 0x7fc6b419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:15.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.240+0000 7fc6b3e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6b4100600 0x7fc6b4197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:44320/0 (socket says 172.21.15.117:44320) 2024-03-19T03:33:15.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.240+0000 7fc6b3e00640 1 -- 172.21.15.117:0/2066274221 learned_addr learned my addr 172.21.15.117:0/2066274221 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:15.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.240+0000 7fc6b3400640 1 --2- 172.21.15.117:0/2066274221 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6b41017f0 0x7fc6b4197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:15.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.240+0000 7fc6b3e00640 1 -- 172.21.15.117:0/2066274221 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6b41017f0 msgr2=0x7fc6b4197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:15.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.240+0000 7fc6b3e00640 1 --2- 172.21.15.117:0/2066274221 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6b41017f0 0x7fc6b4197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:15.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.240+0000 7fc6b3e00640 1 -- 172.21.15.117:0/2066274221 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6b41021b0 msgr2=0x7fc6b419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:15.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.240+0000 7fc6b3e00640 1 --2- 172.21.15.117:0/2066274221 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6b41021b0 0x7fc6b419efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:15.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.240+0000 7fc6b3e00640 1 -- 172.21.15.117:0/2066274221 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6b419f620 con 0x7fc6b4100600 2024-03-19T03:33:15.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.240+0000 7fc6b8c00640 1 --2- 172.21.15.117:0/2066274221 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6b41021b0 0x7fc6b419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:33:15.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.240+0000 7fc6b3e00640 1 --2- 172.21.15.117:0/2066274221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6b4100600 0x7fc6b4197a00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc6a400d950 tx=0x7fc6a400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:15.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.241+0000 7fc6b0c00640 1 -- 172.21.15.117:0/2066274221 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a4017020 con 0x7fc6b4100600 2024-03-19T03:33:15.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.241+0000 7fc6b0c00640 1 -- 172.21.15.117:0/2066274221 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6a4009c30 con 0x7fc6b4100600 2024-03-19T03:33:15.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.241+0000 7fc6b9600640 1 -- 172.21.15.117:0/2066274221 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6b419f910 con 0x7fc6b4100600 2024-03-19T03:33:15.242 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.241+0000 7fc6b0c00640 1 -- 172.21.15.117:0/2066274221 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a400f720 con 0x7fc6b4100600 2024-03-19T03:33:15.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.241+0000 7fc6b9600640 1 -- 172.21.15.117:0/2066274221 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6b419fe30 con 0x7fc6b4100600 2024-03-19T03:33:15.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.242+0000 7fc6b9600640 1 -- 172.21.15.117:0/2066274221 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6b4068530 con 0x7fc6b4100600 2024-03-19T03:33:15.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.245+0000 7fc6b0c00640 1 -- 172.21.15.117:0/2066274221 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fc6a401a020 con 0x7fc6b4100600 2024-03-19T03:33:15.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.247+0000 7fc6b0c00640 1 --2- 172.21.15.117:0/2066274221 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc6980a2800 0x7fc6980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:15.247 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.247+0000 7fc6b0c00640 1 -- 172.21.15.117:0/2066274221 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fc6a40cf100 con 0x7fc6b4100600 2024-03-19T03:33:15.247 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.247+0000 7fc6b3400640 1 --2- 172.21.15.117:0/2066274221 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc6980a2800 0x7fc6980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:15.247 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.248+0000 7fc6b3400640 1 --2- 172.21.15.117:0/2066274221 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc6980a2800 0x7fc6980a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc6a002c7b0 tx=0x7fc6a0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:15.249 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.249+0000 7fc6b0c00640 1 -- 172.21.15.117:0/2066274221 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc6a4021020 con 0x7fc6b4100600 2024-03-19T03:33:15.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.455+0000 7fc6b9600640 1 -- 172.21.15.117:0/2066274221 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc6b41a0060 con 0x7fc6b4100600 2024-03-19T03:33:15.455 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.456+0000 7fc6b0c00640 1 -- 172.21.15.117:0/2066274221 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fc6a40bb0a0 con 0x7fc6b4100600 2024-03-19T03:33:15.456 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 1 2024-03-19T03:33:15.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.458+0000 7fc6b9600640 1 -- 172.21.15.117:0/2066274221 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc6980a2800 msgr2=0x7fc6980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:15.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.458+0000 7fc6b9600640 1 --2- 172.21.15.117:0/2066274221 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc6980a2800 0x7fc6980a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc6a002c7b0 tx=0x7fc6a0005fd0 comp rx=0 tx=0).stop 2024-03-19T03:33:15.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.458+0000 7fc6b9600640 1 -- 172.21.15.117:0/2066274221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6b4100600 msgr2=0x7fc6b4197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:15.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.458+0000 7fc6b9600640 1 --2- 172.21.15.117:0/2066274221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6b4100600 0x7fc6b4197a00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc6a400d950 tx=0x7fc6a400de10 comp rx=0 tx=0).stop 2024-03-19T03:33:15.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.458+0000 7fc6b9600640 1 -- 172.21.15.117:0/2066274221 shutdown_connections 2024-03-19T03:33:15.458 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.458+0000 7fc6b9600640 1 --2- 172.21.15.117:0/2066274221 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc6980a2800 0x7fc6980a4cc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:15.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.458+0000 7fc6b9600640 1 --2- 172.21.15.117:0/2066274221 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6b41021b0 0x7fc6b419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:15.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.458+0000 7fc6b9600640 1 --2- 172.21.15.117:0/2066274221 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6b41017f0 0x7fc6b4197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:15.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.458+0000 7fc6b9600640 1 --2- 172.21.15.117:0/2066274221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6b4100600 0x7fc6b4197a00 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:15.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.458+0000 7fc6b9600640 1 -- 172.21.15.117:0/2066274221 >> 172.21.15.117:0/2066274221 conn(0x7fc6b40fbb90 msgr2=0x7fc6b40fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:15.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.458+0000 7fc6b9600640 1 -- 172.21.15.117:0/2066274221 shutdown_connections 2024-03-19T03:33:15.459 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.458+0000 7fc6b9600640 1 -- 172.21.15.117:0/2066274221 wait complete. 2024-03-19T03:33:15.480 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T03:33:15.682 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.682+0000 7fe84fe00640 1 -- 172.21.15.117:0/2437906648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8500ff660 msgr2=0x7fe8500ffa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:15.682 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.682+0000 7fe84fe00640 1 --2- 172.21.15.117:0/2437906648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8500ff660 0x7fe8500ffa80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe840009900 tx=0x7fe84002bc50 comp rx=0 tx=0).stop 2024-03-19T03:33:15.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.682+0000 7fe84fe00640 1 -- 172.21.15.117:0/2437906648 shutdown_connections 2024-03-19T03:33:15.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.682+0000 7fe84fe00640 1 --2- 172.21.15.117:0/2437906648 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe850101070 0x7fe850101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:15.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.682+0000 7fe84fe00640 1 --2- 172.21.15.117:0/2437906648 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe8501006d0 0x7fe850100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:15.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.682+0000 7fe84fe00640 1 --2- 172.21.15.117:0/2437906648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8500ff660 0x7fe8500ffa80 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:15.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.682+0000 7fe84fe00640 1 -- 172.21.15.117:0/2437906648 >> 172.21.15.117:0/2437906648 conn(0x7fe8500faea0 msgr2=0x7fe8500fd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:15.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.682+0000 7fe84fe00640 1 -- 172.21.15.117:0/2437906648 shutdown_connections 2024-03-19T03:33:15.683 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.682+0000 7fe84fe00640 1 -- 172.21.15.117:0/2437906648 wait complete. 2024-03-19T03:33:15.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.683+0000 7fe84fe00640 1 Processor -- start 2024-03-19T03:33:15.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.683+0000 7fe84fe00640 1 -- start start 2024-03-19T03:33:15.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.684+0000 7fe84fe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8500ff660 0x7fe850199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:15.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.684+0000 7fe84fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8501006d0 0x7fe85019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:15.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.684+0000 7fe84fe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe850101070 0x7fe8501a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:15.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.684+0000 7fe84fe00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe85010f5d0 con 0x7fe8500ff660 2024-03-19T03:33:15.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.684+0000 7fe84fe00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe85010f450 con 0x7fe8501006d0 2024-03-19T03:33:15.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.684+0000 7fe84fe00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fe85010f750 con 0x7fe850101070 2024-03-19T03:33:15.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.684+0000 7fe84e000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8501006d0 0x7fe85019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:15.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.684+0000 7fe84e000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8501006d0 0x7fe85019a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:54518/0 (socket says 172.21.15.117:54518) 2024-03-19T03:33:15.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.684+0000 7fe84e000640 1 -- 172.21.15.117:0/1952864997 learned_addr learned my addr 172.21.15.117:0/1952864997 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:15.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.684+0000 7fe84ea00640 1 --2- 172.21.15.117:0/1952864997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8500ff660 0x7fe850199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:15.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.684+0000 7fe84f400640 1 --2- 172.21.15.117:0/1952864997 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe850101070 0x7fe8501a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:15.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.685+0000 7fe84e000640 1 -- 172.21.15.117:0/1952864997 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe850101070 msgr2=0x7fe8501a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:15.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.685+0000 7fe84e000640 1 --2- 172.21.15.117:0/1952864997 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe850101070 0x7fe8501a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:15.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.685+0000 7fe84e000640 1 -- 172.21.15.117:0/1952864997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8500ff660 msgr2=0x7fe850199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:15.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.685+0000 7fe84e000640 1 --2- 172.21.15.117:0/1952864997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8500ff660 0x7fe850199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:15.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.685+0000 7fe84e000640 1 -- 172.21.15.117:0/1952864997 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8501a1810 con 0x7fe8501006d0 2024-03-19T03:33:15.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.685+0000 7fe84e000640 1 --2- 172.21.15.117:0/1952864997 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8501006d0 0x7fe85019a130 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00e9f0 tx=0x7fe83c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:15.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.685+0000 7fe847400640 1 -- 172.21.15.117:0/1952864997 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c00cd80 con 0x7fe8501006d0 2024-03-19T03:33:15.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.685+0000 7fe847400640 1 -- 172.21.15.117:0/1952864997 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe83c005500 con 0x7fe8501006d0 2024-03-19T03:33:15.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.685+0000 7fe84fe00640 1 -- 172.21.15.117:0/1952864997 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8501a1b00 con 0x7fe8501006d0 2024-03-19T03:33:15.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.685+0000 7fe847400640 1 -- 172.21.15.117:0/1952864997 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c010690 con 0x7fe8501006d0 2024-03-19T03:33:15.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.686+0000 7fe84fe00640 1 -- 172.21.15.117:0/1952864997 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8501a2050 con 0x7fe8501006d0 2024-03-19T03:33:15.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.687+0000 7fe84fe00640 1 -- 172.21.15.117:0/1952864997 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe85010bb40 con 0x7fe8501006d0 2024-03-19T03:33:15.690 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.690+0000 7fe847400640 1 -- 172.21.15.117:0/1952864997 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fe83c0046c0 con 0x7fe8501006d0 2024-03-19T03:33:15.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.693+0000 7fe847400640 1 --2- 172.21.15.117:0/1952864997 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe8340a2880 0x7fe8340a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:15.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.693+0000 7fe847400640 1 -- 172.21.15.117:0/1952864997 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe83c0cf210 con 0x7fe8501006d0 2024-03-19T03:33:15.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.693+0000 7fe84ea00640 1 --2- 172.21.15.117:0/1952864997 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe8340a2880 0x7fe8340a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:15.693 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.694+0000 7fe84ea00640 1 --2- 172.21.15.117:0/1952864997 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe8340a2880 0x7fe8340a4d40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe84002c260 tx=0x7fe840000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:15.700 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.700+0000 7fe847400640 1 -- 172.21.15.117:0/1952864997 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe83c09be60 con 0x7fe8501006d0 2024-03-19T03:33:15.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:15 smithi119 ceph-mon[66861]: pgmap v401: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:15.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:15 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2649926143' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:33:15.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:15 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2066274221' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:33:15.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:15 smithi117 ceph-mon[31838]: pgmap v401: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:15.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:15 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2649926143' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:33:15.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:15 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2066274221' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:33:15.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:15 smithi178 ceph-mon[67578]: pgmap v401: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:15.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:15 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2649926143' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:33:15.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:15 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2066274221' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:33:15.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.945+0000 7fe84fe00640 1 -- 172.21.15.117:0/1952864997 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe850062420 con 0x7fe8501006d0 2024-03-19T03:33:15.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.946+0000 7fe847400640 1 -- 172.21.15.117:0/1952864997 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fe83c09b760 con 0x7fe8501006d0 2024-03-19T03:33:15.946 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 32 2024-03-19T03:33:15.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.949+0000 7fe84fe00640 1 -- 172.21.15.117:0/1952864997 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe8340a2880 msgr2=0x7fe8340a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:15.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.949+0000 7fe84fe00640 1 --2- 172.21.15.117:0/1952864997 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe8340a2880 0x7fe8340a4d40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe84002c260 tx=0x7fe840000bc0 comp rx=0 tx=0).stop 2024-03-19T03:33:15.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.949+0000 7fe84fe00640 1 -- 172.21.15.117:0/1952864997 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8501006d0 msgr2=0x7fe85019a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:15.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.949+0000 7fe84fe00640 1 --2- 172.21.15.117:0/1952864997 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8501006d0 0x7fe85019a130 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00e9f0 tx=0x7fe83c00edf0 comp rx=0 tx=0).stop 2024-03-19T03:33:15.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.949+0000 7fe84fe00640 1 -- 172.21.15.117:0/1952864997 shutdown_connections 2024-03-19T03:33:15.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.949+0000 7fe84fe00640 1 --2- 172.21.15.117:0/1952864997 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe850101070 0x7fe8501a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:15.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.949+0000 7fe84fe00640 1 --2- 172.21.15.117:0/1952864997 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe8340a2880 0x7fe8340a4d40 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:15.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.949+0000 7fe84fe00640 1 --2- 172.21.15.117:0/1952864997 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe8501006d0 0x7fe85019a130 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:15.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.949+0000 7fe84fe00640 1 --2- 172.21.15.117:0/1952864997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8500ff660 0x7fe850199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:15.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.949+0000 7fe84fe00640 1 -- 172.21.15.117:0/1952864997 >> 172.21.15.117:0/1952864997 conn(0x7fe8500faea0 msgr2=0x7fe8500fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:15.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.950+0000 7fe84fe00640 1 -- 172.21.15.117:0/1952864997 shutdown_connections 2024-03-19T03:33:15.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:15.950+0000 7fe84fe00640 1 -- 172.21.15.117:0/1952864997 wait complete. 2024-03-19T03:33:15.969 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T03:33:16.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.202+0000 7fe3ea200640 1 -- 172.21.15.117:0/731948871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3e4103a60 msgr2=0x7fe3e4103e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:16.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.202+0000 7fe3ea200640 1 --2- 172.21.15.117:0/731948871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3e4103a60 0x7fe3e4103e80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4009900 tx=0x7fe3d402bc30 comp rx=0 tx=0).stop 2024-03-19T03:33:16.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.202+0000 7fe3ea200640 1 -- 172.21.15.117:0/731948871 shutdown_connections 2024-03-19T03:33:16.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.202+0000 7fe3ea200640 1 --2- 172.21.15.117:0/731948871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3e4105610 0x7fe3e410bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:16.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.202+0000 7fe3ea200640 1 --2- 172.21.15.117:0/731948871 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3e4104c50 0x7fe3e41050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:16.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.202+0000 7fe3ea200640 1 --2- 172.21.15.117:0/731948871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3e4103a60 0x7fe3e4103e80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:16.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.202+0000 7fe3ea200640 1 -- 172.21.15.117:0/731948871 >> 172.21.15.117:0/731948871 conn(0x7fe3e40fefd0 msgr2=0x7fe3e4101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:16.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.202+0000 7fe3ea200640 1 -- 172.21.15.117:0/731948871 shutdown_connections 2024-03-19T03:33:16.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.202+0000 7fe3ea200640 1 -- 172.21.15.117:0/731948871 wait complete. 2024-03-19T03:33:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.203+0000 7fe3ea200640 1 Processor -- start 2024-03-19T03:33:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.203+0000 7fe3ea200640 1 -- start start 2024-03-19T03:33:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.204+0000 7fe3ea200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3e4103a60 0x7fe3e4199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.204+0000 7fe3ea200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3e4104c50 0x7fe3e419a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.204+0000 7fe3ea200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3e4105610 0x7fe3e41a1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.204+0000 7fe3ea200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e410f720 con 0x7fe3e4105610 2024-03-19T03:33:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.204+0000 7fe3ea200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e410f5a0 con 0x7fe3e4104c50 2024-03-19T03:33:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.204+0000 7fe3ea200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e410f8a0 con 0x7fe3e4103a60 2024-03-19T03:33:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.204+0000 7fe3dbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3e4104c50 0x7fe3e419a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.204+0000 7fe3dbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3e4104c50 0x7fe3e419a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52548/0 (socket says 172.21.15.117:52548) 2024-03-19T03:33:16.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.204+0000 7fe3dbe00640 1 -- 172.21.15.117:0/2128231819 learned_addr learned my addr 172.21.15.117:0/2128231819 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.204+0000 7fe3e3e00640 1 --2- 172.21.15.117:0/2128231819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3e4105610 0x7fe3e41a1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.204+0000 7fe3e3400640 1 --2- 172.21.15.117:0/2128231819 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3e4103a60 0x7fe3e4199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.204+0000 7fe3dbe00640 1 -- 172.21.15.117:0/2128231819 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3e4103a60 msgr2=0x7fe3e4199dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.204+0000 7fe3dbe00640 1 --2- 172.21.15.117:0/2128231819 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3e4103a60 0x7fe3e4199dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.204+0000 7fe3dbe00640 1 -- 172.21.15.117:0/2128231819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3e4105610 msgr2=0x7fe3e41a1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.204+0000 7fe3dbe00640 1 --2- 172.21.15.117:0/2128231819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3e4105610 0x7fe3e41a1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.204+0000 7fe3dbe00640 1 -- 172.21.15.117:0/2128231819 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3e41a19e0 con 0x7fe3e4104c50 2024-03-19T03:33:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.205+0000 7fe3dbe00640 1 --2- 172.21.15.117:0/2128231819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3e4104c50 0x7fe3e419a300 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc00e9f0 tx=0x7fe3cc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.205+0000 7fe3e0c00640 1 -- 172.21.15.117:0/2128231819 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc00cd80 con 0x7fe3e4104c50 2024-03-19T03:33:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.205+0000 7fe3e0c00640 1 -- 172.21.15.117:0/2128231819 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3cc005500 con 0x7fe3e4104c50 2024-03-19T03:33:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.205+0000 7fe3ea200640 1 -- 172.21.15.117:0/2128231819 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3e41a1cd0 con 0x7fe3e4104c50 2024-03-19T03:33:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.205+0000 7fe3e0c00640 1 -- 172.21.15.117:0/2128231819 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc010690 con 0x7fe3e4104c50 2024-03-19T03:33:16.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.205+0000 7fe3ea200640 1 -- 172.21.15.117:0/2128231819 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3e41a21d0 con 0x7fe3e4104c50 2024-03-19T03:33:16.207 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.206+0000 7fe3ea200640 1 -- 172.21.15.117:0/2128231819 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3e410ad80 con 0x7fe3e4104c50 2024-03-19T03:33:16.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.209+0000 7fe3e0c00640 1 -- 172.21.15.117:0/2128231819 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fe3cc0046c0 con 0x7fe3e4104c50 2024-03-19T03:33:16.210 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.211+0000 7fe3e0c00640 1 --2- 172.21.15.117:0/2128231819 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe3c80a2880 0x7fe3c80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:16.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.211+0000 7fe3e0c00640 1 -- 172.21.15.117:0/2128231819 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fe3cc0cf1b0 con 0x7fe3e4104c50 2024-03-19T03:33:16.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.211+0000 7fe3e3400640 1 --2- 172.21.15.117:0/2128231819 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe3c80a2880 0x7fe3c80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:16.211 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.212+0000 7fe3e3400640 1 --2- 172.21.15.117:0/2128231819 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe3c80a2880 0x7fe3c80a4d40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe3d402c170 tx=0x7fe3d402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:16.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.219+0000 7fe3e0c00640 1 -- 172.21.15.117:0/2128231819 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe3cc09be60 con 0x7fe3e4104c50 2024-03-19T03:33:16.475 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.475+0000 7fe3ea200640 1 -- 172.21.15.117:0/2128231819 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe3e4062420 con 0x7fe3e4104c50 2024-03-19T03:33:16.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.479+0000 7fe3e0c00640 1 -- 172.21.15.117:0/2128231819 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fe3cc09b760 con 0x7fe3e4104c50 2024-03-19T03:33:16.479 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 64 2024-03-19T03:33:16.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.482+0000 7fe3ea200640 1 -- 172.21.15.117:0/2128231819 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe3c80a2880 msgr2=0x7fe3c80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:16.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.482+0000 7fe3ea200640 1 --2- 172.21.15.117:0/2128231819 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe3c80a2880 0x7fe3c80a4d40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe3d402c170 tx=0x7fe3d402df90 comp rx=0 tx=0).stop 2024-03-19T03:33:16.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.482+0000 7fe3ea200640 1 -- 172.21.15.117:0/2128231819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3e4104c50 msgr2=0x7fe3e419a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:16.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.482+0000 7fe3ea200640 1 --2- 172.21.15.117:0/2128231819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3e4104c50 0x7fe3e419a300 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc00e9f0 tx=0x7fe3cc00edf0 comp rx=0 tx=0).stop 2024-03-19T03:33:16.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.482+0000 7fe3ea200640 1 -- 172.21.15.117:0/2128231819 shutdown_connections 2024-03-19T03:33:16.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.482+0000 7fe3ea200640 1 --2- 172.21.15.117:0/2128231819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3e4105610 0x7fe3e41a1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:16.482 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.482+0000 7fe3ea200640 1 --2- 172.21.15.117:0/2128231819 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fe3c80a2880 0x7fe3c80a4d40 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:16.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.482+0000 7fe3ea200640 1 --2- 172.21.15.117:0/2128231819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3e4104c50 0x7fe3e419a300 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:16.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.482+0000 7fe3ea200640 1 --2- 172.21.15.117:0/2128231819 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3e4103a60 0x7fe3e4199dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:16.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.482+0000 7fe3ea200640 1 -- 172.21.15.117:0/2128231819 >> 172.21.15.117:0/2128231819 conn(0x7fe3e40fefd0 msgr2=0x7fe3e4107f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:16.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.482+0000 7fe3ea200640 1 -- 172.21.15.117:0/2128231819 shutdown_connections 2024-03-19T03:33:16.483 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.482+0000 7fe3ea200640 1 -- 172.21.15.117:0/2128231819 wait complete. 2024-03-19T03:33:16.504 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T03:33:16.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.726+0000 7f663b200640 1 -- 172.21.15.117:0/3252006378 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66341022e0 msgr2=0x7f6634108b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:16.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.726+0000 7f663b200640 1 --2- 172.21.15.117:0/3252006378 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66341022e0 0x7f6634108b70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f662c009ca0 tx=0x7f662c02c0f0 comp rx=0 tx=0).stop 2024-03-19T03:33:16.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.726+0000 7f663b200640 1 -- 172.21.15.117:0/3252006378 shutdown_connections 2024-03-19T03:33:16.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.726+0000 7f663b200640 1 --2- 172.21.15.117:0/3252006378 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66341022e0 0x7f6634108b70 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:16.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.726+0000 7f663b200640 1 --2- 172.21.15.117:0/3252006378 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6634101920 0x7f6634101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:16.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.726+0000 7f663b200640 1 --2- 172.21.15.117:0/3252006378 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6634100730 0x7f6634100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:16.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.726+0000 7f663b200640 1 -- 172.21.15.117:0/3252006378 >> 172.21.15.117:0/3252006378 conn(0x7f66340fbcc0 msgr2=0x7f66340fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:16.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.726+0000 7f663b200640 1 -- 172.21.15.117:0/3252006378 shutdown_connections 2024-03-19T03:33:16.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.726+0000 7f663b200640 1 -- 172.21.15.117:0/3252006378 wait complete. 2024-03-19T03:33:16.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.728+0000 7f663b200640 1 Processor -- start 2024-03-19T03:33:16.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.728+0000 7f663b200640 1 -- start start 2024-03-19T03:33:16.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.728+0000 7f663b200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6634100730 0x7f663410d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:16.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.728+0000 7f663b200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6634101920 0x7f663410bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:16.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.728+0000 7f663b200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66341022e0 0x7f663410c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:16.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.728+0000 7f663b200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f663406a840 con 0x7f6634101920 2024-03-19T03:33:16.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.728+0000 7f663b200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f663406a6c0 con 0x7f66341022e0 2024-03-19T03:33:16.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.728+0000 7f663b200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f663406a9c0 con 0x7f6634100730 2024-03-19T03:33:16.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.729+0000 7f662be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6634101920 0x7f663410bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:16.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.729+0000 7f6639200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66341022e0 0x7f663410c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:16.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.729+0000 7f6639200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66341022e0 0x7f663410c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52564/0 (socket says 172.21.15.117:52564) 2024-03-19T03:33:16.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.729+0000 7f6639200640 1 -- 172.21.15.117:0/1492314708 learned_addr learned my addr 172.21.15.117:0/1492314708 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:16.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.729+0000 7f6633e00640 1 --2- 172.21.15.117:0/1492314708 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6634100730 0x7f663410d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:16.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.729+0000 7f6639200640 1 -- 172.21.15.117:0/1492314708 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6634100730 msgr2=0x7f663410d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:16.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.729+0000 7f6639200640 1 --2- 172.21.15.117:0/1492314708 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6634100730 0x7f663410d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:16.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.729+0000 7f6639200640 1 -- 172.21.15.117:0/1492314708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6634101920 msgr2=0x7f663410bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:16.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.729+0000 7f6639200640 1 --2- 172.21.15.117:0/1492314708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6634101920 0x7f663410bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:16.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.729+0000 7f6639200640 1 -- 172.21.15.117:0/1492314708 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f663410c8f0 con 0x7f66341022e0 2024-03-19T03:33:16.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.729+0000 7f6639200640 1 --2- 172.21.15.117:0/1492314708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66341022e0 0x7f663410c060 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f662c037040 tx=0x7f662c032f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:16.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.730+0000 7f6631600640 1 -- 172.21.15.117:0/1492314708 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662c041020 con 0x7f66341022e0 2024-03-19T03:33:16.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.730+0000 7f6631600640 1 -- 172.21.15.117:0/1492314708 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f662c034cd0 con 0x7f66341022e0 2024-03-19T03:33:16.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.730+0000 7f6631600640 1 -- 172.21.15.117:0/1492314708 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662c039040 con 0x7f66341022e0 2024-03-19T03:33:16.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.730+0000 7f663b200640 1 -- 172.21.15.117:0/1492314708 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66341a1bb0 con 0x7f66341022e0 2024-03-19T03:33:16.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.730+0000 7f663b200640 1 -- 172.21.15.117:0/1492314708 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66341a2100 con 0x7f66341022e0 2024-03-19T03:33:16.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.731+0000 7f663b200640 1 -- 172.21.15.117:0/1492314708 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6634107aa0 con 0x7f66341022e0 2024-03-19T03:33:16.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:16 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1952864997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:33:16.734 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.734+0000 7f6631600640 1 -- 172.21.15.117:0/1492314708 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f662c044020 con 0x7f66341022e0 2024-03-19T03:33:16.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.736+0000 7f6631600640 1 --2- 172.21.15.117:0/1492314708 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f66180a2800 0x7f66180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:16.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.736+0000 7f6631600640 1 -- 172.21.15.117:0/1492314708 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f662c0f0d10 con 0x7f66341022e0 2024-03-19T03:33:16.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.736+0000 7f6633e00640 1 --2- 172.21.15.117:0/1492314708 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f66180a2800 0x7f66180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:16.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.737+0000 7f6633e00640 1 --2- 172.21.15.117:0/1492314708 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f66180a2800 0x7f66180a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6634101780 tx=0x7f6624005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:16.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.739+0000 7f6631600640 1 -- 172.21.15.117:0/1492314708 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f662c04b020 con 0x7f66341022e0 2024-03-19T03:33:16.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:16 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1952864997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:33:16.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:16 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1952864997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:33:16.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.937+0000 7f663b200640 1 -- 172.21.15.117:0/1492314708 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f663402d1f0 con 0x7f66341022e0 2024-03-19T03:33:16.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.938+0000 7f6631600640 1 -- 172.21.15.117:0/1492314708 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f662c0bda50 con 0x7f66341022e0 2024-03-19T03:33:16.938 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 32 2024-03-19T03:33:16.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.940+0000 7f663b200640 1 -- 172.21.15.117:0/1492314708 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f66180a2800 msgr2=0x7f66180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:16.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.940+0000 7f663b200640 1 --2- 172.21.15.117:0/1492314708 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f66180a2800 0x7f66180a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6634101780 tx=0x7f6624005f90 comp rx=0 tx=0).stop 2024-03-19T03:33:16.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.940+0000 7f663b200640 1 -- 172.21.15.117:0/1492314708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66341022e0 msgr2=0x7f663410c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:16.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.940+0000 7f663b200640 1 --2- 172.21.15.117:0/1492314708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66341022e0 0x7f663410c060 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f662c037040 tx=0x7f662c032f40 comp rx=0 tx=0).stop 2024-03-19T03:33:16.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.941+0000 7f663b200640 1 -- 172.21.15.117:0/1492314708 shutdown_connections 2024-03-19T03:33:16.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.941+0000 7f663b200640 1 --2- 172.21.15.117:0/1492314708 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f66180a2800 0x7f66180a4cc0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:16.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.941+0000 7f663b200640 1 --2- 172.21.15.117:0/1492314708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66341022e0 0x7f663410c060 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:16.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.941+0000 7f663b200640 1 --2- 172.21.15.117:0/1492314708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6634101920 0x7f663410bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:16.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.941+0000 7f663b200640 1 --2- 172.21.15.117:0/1492314708 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6634100730 0x7f663410d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:16.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.941+0000 7f663b200640 1 -- 172.21.15.117:0/1492314708 >> 172.21.15.117:0/1492314708 conn(0x7f66340fbcc0 msgr2=0x7f6634104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:16.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.941+0000 7f663b200640 1 -- 172.21.15.117:0/1492314708 shutdown_connections 2024-03-19T03:33:16.941 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:16.941+0000 7f663b200640 1 -- 172.21.15.117:0/1492314708 wait complete. 2024-03-19T03:33:16.961 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:33:17.192 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.192+0000 7f0a7c200640 1 -- 172.21.15.117:0/4021208701 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a74100740 msgr2=0x7f0a74100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:17.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.192+0000 7f0a7c200640 1 --2- 172.21.15.117:0/4021208701 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a74100740 0x7f0a74100b60 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0a68009900 tx=0x7f0a6802bc30 comp rx=0 tx=0).stop 2024-03-19T03:33:17.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.192+0000 7f0a7c200640 1 -- 172.21.15.117:0/4021208701 shutdown_connections 2024-03-19T03:33:17.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.192+0000 7f0a7c200640 1 --2- 172.21.15.117:0/4021208701 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a741022f0 0x7f0a74108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:17.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.192+0000 7f0a7c200640 1 --2- 172.21.15.117:0/4021208701 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a74101930 0x7f0a74101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:17.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.192+0000 7f0a7c200640 1 --2- 172.21.15.117:0/4021208701 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a74100740 0x7f0a74100b60 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:17.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.192+0000 7f0a7c200640 1 -- 172.21.15.117:0/4021208701 >> 172.21.15.117:0/4021208701 conn(0x7f0a740fbcd0 msgr2=0x7f0a740fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:17.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.193+0000 7f0a7c200640 1 -- 172.21.15.117:0/4021208701 shutdown_connections 2024-03-19T03:33:17.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.193+0000 7f0a7c200640 1 -- 172.21.15.117:0/4021208701 wait complete. 2024-03-19T03:33:17.195 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.194+0000 7f0a7c200640 1 Processor -- start 2024-03-19T03:33:17.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.194+0000 7f0a7c200640 1 -- start start 2024-03-19T03:33:17.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.194+0000 7f0a7c200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a74100740 0x7f0a74199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:17.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.194+0000 7f0a7c200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a74101930 0x7f0a7419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:17.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.194+0000 7f0a7c200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a741022f0 0x7f0a741a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:17.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.194+0000 7f0a7c200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7410d540 con 0x7f0a74100740 2024-03-19T03:33:17.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.194+0000 7f0a7c200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7410d3c0 con 0x7f0a74101930 2024-03-19T03:33:17.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.194+0000 7f0a7c200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7410d6c0 con 0x7f0a741022f0 2024-03-19T03:33:17.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.194+0000 7f0a78e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a74101930 0x7f0a7419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:17.196 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.194+0000 7f0a79800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a74100740 0x7f0a74199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:17.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.194+0000 7f0a78e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a74101930 0x7f0a7419a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52574/0 (socket says 172.21.15.117:52574) 2024-03-19T03:33:17.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.194+0000 7f0a78e00640 1 -- 172.21.15.117:0/236555512 learned_addr learned my addr 172.21.15.117:0/236555512 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:17.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.194+0000 7f0a7a200640 1 --2- 172.21.15.117:0/236555512 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a741022f0 0x7f0a741a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:17.198 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.195+0000 7f0a78e00640 1 -- 172.21.15.117:0/236555512 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a741022f0 msgr2=0x7f0a741a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:17.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.195+0000 7f0a78e00640 1 --2- 172.21.15.117:0/236555512 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a741022f0 0x7f0a741a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:17.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.195+0000 7f0a78e00640 1 -- 172.21.15.117:0/236555512 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a74100740 msgr2=0x7f0a74199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:17.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.195+0000 7f0a78e00640 1 --2- 172.21.15.117:0/236555512 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a74100740 0x7f0a74199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:17.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.195+0000 7f0a78e00640 1 -- 172.21.15.117:0/236555512 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a741a19c0 con 0x7f0a74101930 2024-03-19T03:33:17.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.195+0000 7f0a79800640 1 --2- 172.21.15.117:0/236555512 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a74100740 0x7f0a74199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:33:17.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.195+0000 7f0a78e00640 1 --2- 172.21.15.117:0/236555512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a74101930 0x7f0a7419a2f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0a6000e9f0 tx=0x7f0a6000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:17.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.195+0000 7f0a72000640 1 -- 172.21.15.117:0/236555512 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6000cd80 con 0x7f0a74101930 2024-03-19T03:33:17.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.195+0000 7f0a72000640 1 -- 172.21.15.117:0/236555512 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a60010910 con 0x7f0a74101930 2024-03-19T03:33:17.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.195+0000 7f0a72000640 1 -- 172.21.15.117:0/236555512 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a60018990 con 0x7f0a74101930 2024-03-19T03:33:17.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.195+0000 7f0a7c200640 1 -- 172.21.15.117:0/236555512 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a741a1cb0 con 0x7f0a74101930 2024-03-19T03:33:17.199 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.196+0000 7f0a7c200640 1 -- 172.21.15.117:0/236555512 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a741a2180 con 0x7f0a74101930 2024-03-19T03:33:17.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.197+0000 7f0a7c200640 1 -- 172.21.15.117:0/236555512 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a741a2350 con 0x7f0a74101930 2024-03-19T03:33:17.200 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.199+0000 7f0a72000640 1 -- 172.21.15.117:0/236555512 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f0a60010ac0 con 0x7f0a74101930 2024-03-19T03:33:17.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.201+0000 7f0a72000640 1 --2- 172.21.15.117:0/236555512 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0a5c0a2880 0x7f0a5c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:17.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.202+0000 7f0a72000640 1 -- 172.21.15.117:0/236555512 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f0a600cf190 con 0x7f0a74101930 2024-03-19T03:33:17.201 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.202+0000 7f0a79800640 1 --2- 172.21.15.117:0/236555512 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0a5c0a2880 0x7f0a5c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:17.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.202+0000 7f0a79800640 1 --2- 172.21.15.117:0/236555512 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0a5c0a2880 0x7f0a5c0a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0a6802c170 tx=0x7f0a6802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:17.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.204+0000 7f0a72000640 1 -- 172.21.15.117:0/236555512 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0a6009b7f0 con 0x7f0a74101930 2024-03-19T03:33:17.474 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.474+0000 7f0a7c200640 1 -- 172.21.15.117:0/236555512 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a741a25c0 con 0x7f0a74101930 2024-03-19T03:33:17.474 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.475+0000 7f0a72000640 1 -- 172.21.15.117:0/236555512 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11953 (secure 0 0 0) 0x7f0a6009f890 con 0x7f0a74101930 2024-03-19T03:33:17.475 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:33:17.475 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/54638259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":54638259},{"type":"v1","addr":"172.21.15.117:6835","nonce":54638259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3608185039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3608185039},{"type":"v1","addr":"172.21.15.117:6837","nonce":3608185039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T03:30:35.081740+0000","modified":"2024-03-19T03:31:09.995133+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24599,"mds_1":24581,"mds_2":24569,"mds_3":24545,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":20,"addr":"172.21.15.119:6833/1291833214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1291833214},{"type":"v1","addr":"172.21.15.119:6833","nonce":1291833214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6835/1579860804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1579860804},{"type":"v1","addr":"172.21.15.178:6835","nonce":1579860804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6835/1686938941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1686938941},{"type":"v1","addr":"172.21.15.119:6835","nonce":1686938941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6837/1770913538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":1770913538},{"type":"v1","addr":"172.21.15.178:6837","nonce":1770913538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6839/2655593383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2655593383},{"type":"v1","addr":"172.21.15.178:6839","nonce":2655593383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.119:6837/2054992738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2054992738},{"type":"v1","addr":"172.21.15.119:6837","nonce":2054992738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"g","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.117:6839/684335078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":684335078},{"type":"v1","addr":"172.21.15.117:6839","nonce":684335078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.117:6841/1329933892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6840","nonce":1329933892},{"type":"v1","addr":"172.21.15.117:6841","nonce":1329933892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/2383354833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2383354833},{"type":"v1","addr":"172.21.15.119:6839","nonce":2383354833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24581,24599,24569,24545,24467]},"id":1}]} 2024-03-19T03:33:17.475 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 28 2024-03-19T03:33:17.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.478+0000 7f0a7c200640 1 -- 172.21.15.117:0/236555512 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0a5c0a2880 msgr2=0x7f0a5c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:17.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.478+0000 7f0a7c200640 1 --2- 172.21.15.117:0/236555512 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0a5c0a2880 0x7f0a5c0a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0a6802c170 tx=0x7f0a6802df90 comp rx=0 tx=0).stop 2024-03-19T03:33:17.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.478+0000 7f0a7c200640 1 -- 172.21.15.117:0/236555512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a74101930 msgr2=0x7f0a7419a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:17.477 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.478+0000 7f0a7c200640 1 --2- 172.21.15.117:0/236555512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a74101930 0x7f0a7419a2f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0a6000e9f0 tx=0x7f0a6000eeb0 comp rx=0 tx=0).stop 2024-03-19T03:33:17.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.478+0000 7f0a7c200640 1 -- 172.21.15.117:0/236555512 shutdown_connections 2024-03-19T03:33:17.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.478+0000 7f0a7c200640 1 --2- 172.21.15.117:0/236555512 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a741022f0 0x7f0a741a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:17.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.478+0000 7f0a7c200640 1 --2- 172.21.15.117:0/236555512 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0a5c0a2880 0x7f0a5c0a4d40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:17.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.478+0000 7f0a7c200640 1 --2- 172.21.15.117:0/236555512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a74101930 0x7f0a7419a2f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:17.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.478+0000 7f0a7c200640 1 --2- 172.21.15.117:0/236555512 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a74100740 0x7f0a74199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:17.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.478+0000 7f0a7c200640 1 -- 172.21.15.117:0/236555512 >> 172.21.15.117:0/236555512 conn(0x7f0a740fbcd0 msgr2=0x7f0a74109540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:17.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.478+0000 7f0a7c200640 1 -- 172.21.15.117:0/236555512 shutdown_connections 2024-03-19T03:33:17.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.478+0000 7f0a7c200640 1 -- 172.21.15.117:0/236555512 wait complete. 2024-03-19T03:33:17.497 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:33:17.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.715+0000 7ffb05400640 1 -- 172.21.15.117:0/3041415644 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb001073b0 msgr2=0x7ffb001097a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:17.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.715+0000 7ffb05400640 1 --2- 172.21.15.117:0/3041415644 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb001073b0 0x7ffb001097a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8009ca0 tx=0x7ffaf802c0f0 comp rx=0 tx=0).stop 2024-03-19T03:33:17.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.716+0000 7ffb05400640 1 -- 172.21.15.117:0/3041415644 shutdown_connections 2024-03-19T03:33:17.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.716+0000 7ffb05400640 1 --2- 172.21.15.117:0/3041415644 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb001073b0 0x7ffb001097a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:17.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.716+0000 7ffb05400640 1 --2- 172.21.15.117:0/3041415644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb00104a80 0x7ffb00106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:17.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.716+0000 7ffb05400640 1 --2- 172.21.15.117:0/3041415644 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffb00102150 0x7ffb00104540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:17.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.716+0000 7ffb05400640 1 -- 172.21.15.117:0/3041415644 >> 172.21.15.117:0/3041415644 conn(0x7ffb000fbb00 msgr2=0x7ffb000fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:17.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.716+0000 7ffb05400640 1 -- 172.21.15.117:0/3041415644 shutdown_connections 2024-03-19T03:33:17.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.716+0000 7ffb05400640 1 -- 172.21.15.117:0/3041415644 wait complete. 2024-03-19T03:33:17.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.717+0000 7ffb05400640 1 Processor -- start 2024-03-19T03:33:17.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.717+0000 7ffb05400640 1 -- start start 2024-03-19T03:33:17.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.717+0000 7ffb05400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb00102150 0x7ffb00199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:17.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.717+0000 7ffb05400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffb00104a80 0x7ffb0019a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:17.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.717+0000 7ffb05400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb001073b0 0x7ffb001a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:17.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.717+0000 7ffb05400640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0010d310 con 0x7ffb00102150 2024-03-19T03:33:17.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.717+0000 7ffb05400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0010d190 con 0x7ffb001073b0 2024-03-19T03:33:17.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.717+0000 7ffb05400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0010d490 con 0x7ffb00104a80 2024-03-19T03:33:17.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.718+0000 7ffb04a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb001073b0 0x7ffb001a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:17.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.718+0000 7ffaffe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb00102150 0x7ffb00199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:17.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.718+0000 7ffb04a00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb001073b0 0x7ffb001a10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52600/0 (socket says 172.21.15.117:52600) 2024-03-19T03:33:17.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.718+0000 7ffb04a00640 1 -- 172.21.15.117:0/2271138459 learned_addr learned my addr 172.21.15.117:0/2271138459 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:17.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.718+0000 7ffaff400640 1 --2- 172.21.15.117:0/2271138459 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffb00104a80 0x7ffb0019a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:17.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.718+0000 7ffb04a00640 1 -- 172.21.15.117:0/2271138459 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffb00104a80 msgr2=0x7ffb0019a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:17.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.718+0000 7ffb04a00640 1 --2- 172.21.15.117:0/2271138459 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffb00104a80 0x7ffb0019a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:17.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.718+0000 7ffb04a00640 1 -- 172.21.15.117:0/2271138459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb00102150 msgr2=0x7ffb00199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:17.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.718+0000 7ffb04a00640 1 --2- 172.21.15.117:0/2271138459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb00102150 0x7ffb00199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:17.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.718+0000 7ffb04a00640 1 -- 172.21.15.117:0/2271138459 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb001a1740 con 0x7ffb001073b0 2024-03-19T03:33:17.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.718+0000 7ffb04a00640 1 --2- 172.21.15.117:0/2271138459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb001073b0 0x7ffb001a10d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ffaf802c630 tx=0x7ffaf8032ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:17.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.719+0000 7ffafcc00640 1 -- 172.21.15.117:0/2271138459 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf8041020 con 0x7ffb001073b0 2024-03-19T03:33:17.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.719+0000 7ffb05400640 1 -- 172.21.15.117:0/2271138459 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb001a19d0 con 0x7ffb001073b0 2024-03-19T03:33:17.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.719+0000 7ffafcc00640 1 -- 172.21.15.117:0/2271138459 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffaf8034cd0 con 0x7ffb001073b0 2024-03-19T03:33:17.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.719+0000 7ffafcc00640 1 -- 172.21.15.117:0/2271138459 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf8039040 con 0x7ffb001073b0 2024-03-19T03:33:17.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.719+0000 7ffb05400640 1 -- 172.21.15.117:0/2271138459 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb001a1ec0 con 0x7ffb001073b0 2024-03-19T03:33:17.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.721+0000 7ffb05400640 1 -- 172.21.15.117:0/2271138459 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb00108ea0 con 0x7ffb001073b0 2024-03-19T03:33:17.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.722+0000 7ffafcc00640 1 -- 172.21.15.117:0/2271138459 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7ffaf8044020 con 0x7ffb001073b0 2024-03-19T03:33:17.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.723+0000 7ffafcc00640 1 --2- 172.21.15.117:0/2271138459 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7ffae40a2800 0x7ffae40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:17.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.723+0000 7ffafcc00640 1 -- 172.21.15.117:0/2271138459 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7ffaf80f0d80 con 0x7ffb001073b0 2024-03-19T03:33:17.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.723+0000 7ffaffe00640 1 --2- 172.21.15.117:0/2271138459 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7ffae40a2800 0x7ffae40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:17.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.724+0000 7ffaffe00640 1 --2- 172.21.15.117:0/2271138459 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7ffae40a2800 0x7ffae40a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ffaf0005fd0 tx=0x7ffaf000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:17.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.732+0000 7ffafcc00640 1 -- 172.21.15.117:0/2271138459 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ffaf80f4030 con 0x7ffb001073b0 2024-03-19T03:33:17.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:17 smithi119 ceph-mon[66861]: pgmap v402: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:33:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:17 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2128231819' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:33:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:17 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1492314708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:33:17.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:17 smithi117 ceph-mon[31838]: pgmap v402: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:33:17.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:17 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2128231819' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:33:17.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:17 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1492314708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:33:17.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:17 smithi178 ceph-mon[67578]: pgmap v402: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:33:17.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:17 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2128231819' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:33:17.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:17 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1492314708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:33:17.955 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.955+0000 7ffb05400640 1 -- 172.21.15.117:0/2271138459 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffb00062420 con 0x7ffb001073b0 2024-03-19T03:33:17.956 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.956+0000 7ffafcc00640 1 -- 172.21.15.117:0/2271138459 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24989 (secure 0 0 0) 0x7ffaf804b020 con 0x7ffb001073b0 2024-03-19T03:33:17.957 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:33:17.957 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":93,"fsid":"a3a82456-e59f-11ee-95c9-87774f69a715","created":"2024-03-19T03:21:30.427136+0000","modified":"2024-03-19T03:32:40.453890+0000","last_up_change":"2024-03-19T03:29:32.967505+0000","last_in_change":"2024-03-19T03:29:12.848740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:25:18.824270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T03:30:33.137170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T03:30:34.598838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T03:30:38.230023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b32eb0ee-9d4b-4748-87ec-eda02c907415","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6803","nonce":3014934900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6805","nonce":3014934900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6809","nonce":3014934900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6807","nonce":3014934900}]},"public_addr":"172.21.15.117:6803/3014934900","cluster_addr":"172.21.15.117:6805/3014934900","heartbeat_back_addr":"172.21.15.117:6809/3014934900","heartbeat_front_addr":"172.21.15.117:6807/3014934900","state":["exists","up"]},{"osd":1,"uuid":"e07585be-c016-4466-b07c-abbc22a58de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6801","nonce":2425927341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6803","nonce":2425927341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6807","nonce":2425927341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6805","nonce":2425927341}]},"public_addr":"172.21.15.119:6801/2425927341","cluster_addr":"172.21.15.119:6803/2425927341","heartbeat_back_addr":"172.21.15.119:6807/2425927341","heartbeat_front_addr":"172.21.15.119:6805/2425927341","state":["exists","up"]},{"osd":2,"uuid":"ede9edda-3b9f-41ca-ad04-5dcb738ab757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6801","nonce":3733812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6803","nonce":3733812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6807","nonce":3733812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6805","nonce":3733812989}]},"public_addr":"172.21.15.178:6801/3733812989","cluster_addr":"172.21.15.178:6803/3733812989","heartbeat_back_addr":"172.21.15.178:6807/3733812989","heartbeat_front_addr":"172.21.15.178:6805/3733812989","state":["exists","up"]},{"osd":3,"uuid":"f81c41e1-3a69-40db-a90f-58151286b081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6811","nonce":2178310135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6813","nonce":2178310135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6817","nonce":2178310135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6815","nonce":2178310135}]},"public_addr":"172.21.15.117:6811/2178310135","cluster_addr":"172.21.15.117:6813/2178310135","heartbeat_back_addr":"172.21.15.117:6817/2178310135","heartbeat_front_addr":"172.21.15.117:6815/2178310135","state":["exists","up"]},{"osd":4,"uuid":"caf7ce86-caf0-4eaf-bbab-bd2d93e63c33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6809","nonce":3455398086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6811","nonce":3455398086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6815","nonce":3455398086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6813","nonce":3455398086}]},"public_addr":"172.21.15.119:6809/3455398086","cluster_addr":"172.21.15.119:6811/3455398086","heartbeat_back_addr":"172.21.15.119:6815/3455398086","heartbeat_front_addr":"172.21.15.119:6813/3455398086","state":["exists","up"]},{"osd":5,"uuid":"20ba0b8a-2e0f-4ea7-b435-430e44766200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6809","nonce":1259631912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6811","nonce":1259631912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6815","nonce":1259631912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6813","nonce":1259631912}]},"public_addr":"172.21.15.178:6809/1259631912","cluster_addr":"172.21.15.178:6811/1259631912","heartbeat_back_addr":"172.21.15.178:6815/1259631912","heartbeat_front_addr":"172.21.15.178:6813/1259631912","state":["exists","up"]},{"osd":6,"uuid":"75c2ddd7-f837-42e0-ac47-8cfbcf97a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6819","nonce":864146757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6821","nonce":864146757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6825","nonce":864146757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6823","nonce":864146757}]},"public_addr":"172.21.15.117:6819/864146757","cluster_addr":"172.21.15.117:6821/864146757","heartbeat_back_addr":"172.21.15.117:6825/864146757","heartbeat_front_addr":"172.21.15.117:6823/864146757","state":["exists","up"]},{"osd":7,"uuid":"929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6817","nonce":1273365545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6819","nonce":1273365545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6823","nonce":1273365545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6821","nonce":1273365545}]},"public_addr":"172.21.15.119:6817/1273365545","cluster_addr":"172.21.15.119:6819/1273365545","heartbeat_back_addr":"172.21.15.119:6823/1273365545","heartbeat_front_addr":"172.21.15.119:6821/1273365545","state":["exists","up"]},{"osd":8,"uuid":"fd371461-a538-4781-92e1-5eead4ca3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6817","nonce":1059587501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6819","nonce":1059587501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6823","nonce":1059587501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6821","nonce":1059587501}]},"public_addr":"172.21.15.178:6817/1059587501","cluster_addr":"172.21.15.178:6819/1059587501","heartbeat_back_addr":"172.21.15.178:6823/1059587501","heartbeat_front_addr":"172.21.15.178:6821/1059587501","state":["exists","up"]},{"osd":9,"uuid":"0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6827","nonce":3653875600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6829","nonce":3653875600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6833","nonce":3653875600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6831","nonce":3653875600}]},"public_addr":"172.21.15.117:6827/3653875600","cluster_addr":"172.21.15.117:6829/3653875600","heartbeat_back_addr":"172.21.15.117:6833/3653875600","heartbeat_front_addr":"172.21.15.117:6831/3653875600","state":["exists","up"]},{"osd":10,"uuid":"de2049da-d19e-4386-afbf-fd6b5fddbcba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6825","nonce":1050976194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6827","nonce":1050976194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6831","nonce":1050976194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6829","nonce":1050976194}]},"public_addr":"172.21.15.119:6825/1050976194","cluster_addr":"172.21.15.119:6827/1050976194","heartbeat_back_addr":"172.21.15.119:6831/1050976194","heartbeat_front_addr":"172.21.15.119:6829/1050976194","state":["exists","up"]},{"osd":11,"uuid":"3cccc0f9-d273-4b6b-9b8e-c926371472dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6825","nonce":2915080368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6827","nonce":2915080368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6831","nonce":2915080368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6829","nonce":2915080368}]},"public_addr":"172.21.15.178:6825/2915080368","cluster_addr":"172.21.15.178:6827/2915080368","heartbeat_back_addr":"172.21.15.178:6831/2915080368","heartbeat_front_addr":"172.21.15.178:6829/2915080368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:27.876274+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:53.094706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:16.598813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:48.160419+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:14.048925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:38.814304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:12.542328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:38.942747+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:04.488963+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:38.196053+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:05.254728+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:30.874915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7},{"from":5,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.117:0/185684447":"2024-03-20T03:22:38.134578+0000","172.21.15.117:0/361367101":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/2788067603":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3506200675":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6800/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3283471541":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3750920169":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/2780813298":"2024-03-20T03:22:03.905461+0000","172.21.15.117:6800/2788067603":"2024-03-20T03:22:38.134578+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:33:17.960 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.960+0000 7ffb05400640 1 -- 172.21.15.117:0/2271138459 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7ffae40a2800 msgr2=0x7ffae40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:17.960 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.960+0000 7ffb05400640 1 --2- 172.21.15.117:0/2271138459 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7ffae40a2800 0x7ffae40a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ffaf0005fd0 tx=0x7ffaf000b390 comp rx=0 tx=0).stop 2024-03-19T03:33:17.960 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.960+0000 7ffb05400640 1 -- 172.21.15.117:0/2271138459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb001073b0 msgr2=0x7ffb001a10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:17.960 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.960+0000 7ffb05400640 1 --2- 172.21.15.117:0/2271138459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb001073b0 0x7ffb001a10d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ffaf802c630 tx=0x7ffaf8032ea0 comp rx=0 tx=0).stop 2024-03-19T03:33:17.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.960+0000 7ffb05400640 1 -- 172.21.15.117:0/2271138459 shutdown_connections 2024-03-19T03:33:17.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.960+0000 7ffb05400640 1 --2- 172.21.15.117:0/2271138459 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7ffae40a2800 0x7ffae40a4cc0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:17.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.960+0000 7ffb05400640 1 --2- 172.21.15.117:0/2271138459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb001073b0 0x7ffb001a10d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:17.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.960+0000 7ffb05400640 1 --2- 172.21.15.117:0/2271138459 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffb00104a80 0x7ffb0019a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:17.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.960+0000 7ffb05400640 1 --2- 172.21.15.117:0/2271138459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb00102150 0x7ffb00199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:17.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.960+0000 7ffb05400640 1 -- 172.21.15.117:0/2271138459 >> 172.21.15.117:0/2271138459 conn(0x7ffb000fbb00 msgr2=0x7ffb001017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:17.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.960+0000 7ffb05400640 1 -- 172.21.15.117:0/2271138459 shutdown_connections 2024-03-19T03:33:17.961 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:17.960+0000 7ffb05400640 1 -- 172.21.15.117:0/2271138459 wait complete. 2024-03-19T03:33:17.977 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T03:25:18.824270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T03:30:33.137170+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T03:30:34.598838+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T03:30:38.230023+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T03:33:17.978 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T03:33:18.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.196+0000 7f3a42000640 1 -- 172.21.15.117:0/552852976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a3c104ca0 msgr2=0x7f3a3c107090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:18.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.196+0000 7f3a42000640 1 --2- 172.21.15.117:0/552852976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a3c104ca0 0x7f3a3c107090 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009860 tx=0x7f3a2802bb70 comp rx=0 tx=0).stop 2024-03-19T03:33:18.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.197+0000 7f3a42000640 1 -- 172.21.15.117:0/552852976 shutdown_connections 2024-03-19T03:33:18.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.197+0000 7f3a42000640 1 --2- 172.21.15.117:0/552852976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a3c1075d0 0x7f3a3c1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:18.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.197+0000 7f3a42000640 1 --2- 172.21.15.117:0/552852976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a3c104ca0 0x7f3a3c107090 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:18.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.197+0000 7f3a42000640 1 --2- 172.21.15.117:0/552852976 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a3c102370 0x7f3a3c104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:18.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.197+0000 7f3a42000640 1 -- 172.21.15.117:0/552852976 >> 172.21.15.117:0/552852976 conn(0x7f3a3c0fbcc0 msgr2=0x7f3a3c0fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:18.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.197+0000 7f3a42000640 1 -- 172.21.15.117:0/552852976 shutdown_connections 2024-03-19T03:33:18.197 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.197+0000 7f3a42000640 1 -- 172.21.15.117:0/552852976 wait complete. 2024-03-19T03:33:18.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.198+0000 7f3a42000640 1 Processor -- start 2024-03-19T03:33:18.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.198+0000 7f3a42000640 1 -- start start 2024-03-19T03:33:18.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.198+0000 7f3a42000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a3c102370 0x7f3a3c101c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:18.202 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.198+0000 7f3a42000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a3c104ca0 0x7f3a3c100320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:18.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.199+0000 7f3a42000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a3c1075d0 0x7f3a3c100860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:18.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.199+0000 7f3a42000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3c10d490 con 0x7f3a3c102370 2024-03-19T03:33:18.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.199+0000 7f3a42000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3c10d310 con 0x7f3a3c104ca0 2024-03-19T03:33:18.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.199+0000 7f3a42000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3c10d610 con 0x7f3a3c1075d0 2024-03-19T03:33:18.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.199+0000 7f3a3b400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a3c102370 0x7f3a3c101c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:18.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.199+0000 7f3a3aa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a3c104ca0 0x7f3a3c100320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:18.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.199+0000 7f3a3b400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a3c102370 0x7f3a3c101c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42150/0 (socket says 172.21.15.117:42150) 2024-03-19T03:33:18.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.199+0000 7f3a3b400640 1 -- 172.21.15.117:0/4131138259 learned_addr learned my addr 172.21.15.117:0/4131138259 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:18.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.199+0000 7f3a3aa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a3c104ca0 0x7f3a3c100320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52616/0 (socket says 172.21.15.117:52616) 2024-03-19T03:33:18.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.199+0000 7f3a3be00640 1 --2- 172.21.15.117:0/4131138259 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a3c1075d0 0x7f3a3c100860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:18.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.199+0000 7f3a3aa00640 1 -- 172.21.15.117:0/4131138259 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a3c1075d0 msgr2=0x7f3a3c100860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:18.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.199+0000 7f3a3aa00640 1 --2- 172.21.15.117:0/4131138259 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a3c1075d0 0x7f3a3c100860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:18.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.199+0000 7f3a3aa00640 1 -- 172.21.15.117:0/4131138259 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a3c102370 msgr2=0x7f3a3c101c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:18.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.199+0000 7f3a3aa00640 1 --2- 172.21.15.117:0/4131138259 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a3c102370 0x7f3a3c101c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:18.203 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.199+0000 7f3a3aa00640 1 -- 172.21.15.117:0/4131138259 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a3c101120 con 0x7f3a3c104ca0 2024-03-19T03:33:18.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.199+0000 7f3a3be00640 1 --2- 172.21.15.117:0/4131138259 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a3c1075d0 0x7f3a3c100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T03:33:18.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.200+0000 7f3a3aa00640 1 --2- 172.21.15.117:0/4131138259 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a3c104ca0 0x7f3a3c100320 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009780 tx=0x7f3a28032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:18.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.200+0000 7f3a33e00640 1 -- 172.21.15.117:0/4131138259 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a28041020 con 0x7f3a3c104ca0 2024-03-19T03:33:18.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.200+0000 7f3a33e00640 1 -- 172.21.15.117:0/4131138259 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a2802c9f0 con 0x7f3a3c104ca0 2024-03-19T03:33:18.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.200+0000 7f3a42000640 1 -- 172.21.15.117:0/4131138259 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a3c1a1cf0 con 0x7f3a3c104ca0 2024-03-19T03:33:18.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.200+0000 7f3a33e00640 1 -- 172.21.15.117:0/4131138259 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a28039040 con 0x7f3a3c104ca0 2024-03-19T03:33:18.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.200+0000 7f3a42000640 1 -- 172.21.15.117:0/4131138259 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a3c1a21b0 con 0x7f3a3c104ca0 2024-03-19T03:33:18.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.201+0000 7f3a42000640 1 -- 172.21.15.117:0/4131138259 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a3c0fd7b0 con 0x7f3a3c104ca0 2024-03-19T03:33:18.204 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.204+0000 7f3a33e00640 1 -- 172.21.15.117:0/4131138259 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f3a28044020 con 0x7f3a3c104ca0 2024-03-19T03:33:18.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.205+0000 7f3a33e00640 1 --2- 172.21.15.117:0/4131138259 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f3a200a2800 0x7f3a200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:18.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.206+0000 7f3a33e00640 1 -- 172.21.15.117:0/4131138259 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f3a280f0740 con 0x7f3a3c104ca0 2024-03-19T03:33:18.205 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.206+0000 7f3a3b400640 1 --2- 172.21.15.117:0/4131138259 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f3a200a2800 0x7f3a200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:18.206 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.206+0000 7f3a3b400640 1 --2- 172.21.15.117:0/4131138259 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f3a200a2800 0x7f3a200a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c0095f0 tx=0x7f3a2c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:18.209 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.209+0000 7f3a33e00640 1 -- 172.21.15.117:0/4131138259 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3a2804b020 con 0x7f3a3c104ca0 2024-03-19T03:33:18.409 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.409+0000 7f3a42000640 1 -- 172.21.15.117:0/4131138259 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3a3c062420 con 0x7f3a3c104ca0 2024-03-19T03:33:18.410 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.410+0000 7f3a33e00640 1 -- 172.21.15.117:0/4131138259 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f3a280bd390 con 0x7f3a3c104ca0 2024-03-19T03:33:18.410 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 1 2024-03-19T03:33:18.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.413+0000 7f3a42000640 1 -- 172.21.15.117:0/4131138259 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f3a200a2800 msgr2=0x7f3a200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:18.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.413+0000 7f3a42000640 1 --2- 172.21.15.117:0/4131138259 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f3a200a2800 0x7f3a200a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c0095f0 tx=0x7f3a2c005f90 comp rx=0 tx=0).stop 2024-03-19T03:33:18.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.413+0000 7f3a42000640 1 -- 172.21.15.117:0/4131138259 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a3c104ca0 msgr2=0x7f3a3c100320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:18.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.413+0000 7f3a42000640 1 --2- 172.21.15.117:0/4131138259 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a3c104ca0 0x7f3a3c100320 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009780 tx=0x7f3a28032430 comp rx=0 tx=0).stop 2024-03-19T03:33:18.413 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.413+0000 7f3a42000640 1 -- 172.21.15.117:0/4131138259 shutdown_connections 2024-03-19T03:33:18.414 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.413+0000 7f3a42000640 1 --2- 172.21.15.117:0/4131138259 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a3c1075d0 0x7f3a3c100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:18.414 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.413+0000 7f3a42000640 1 --2- 172.21.15.117:0/4131138259 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f3a200a2800 0x7f3a200a4cc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:18.414 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.413+0000 7f3a42000640 1 --2- 172.21.15.117:0/4131138259 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a3c104ca0 0x7f3a3c100320 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:18.414 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.413+0000 7f3a42000640 1 --2- 172.21.15.117:0/4131138259 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a3c102370 0x7f3a3c101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:18.414 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.413+0000 7f3a42000640 1 -- 172.21.15.117:0/4131138259 >> 172.21.15.117:0/4131138259 conn(0x7f3a3c0fbcc0 msgr2=0x7f3a3c102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:18.414 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.413+0000 7f3a42000640 1 -- 172.21.15.117:0/4131138259 shutdown_connections 2024-03-19T03:33:18.414 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.414+0000 7f3a42000640 1 -- 172.21.15.117:0/4131138259 wait complete. 2024-03-19T03:33:18.433 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T03:33:18.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.637+0000 7f6484600640 1 -- 172.21.15.117:0/3277227977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f647c074220 msgr2=0x7f647c074680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:18.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.637+0000 7f6484600640 1 --2- 172.21.15.117:0/3277227977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f647c074220 0x7f647c074680 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6468009900 tx=0x7f646802bcc0 comp rx=0 tx=0).stop 2024-03-19T03:33:18.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.637+0000 7f6484600640 1 -- 172.21.15.117:0/3277227977 shutdown_connections 2024-03-19T03:33:18.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.637+0000 7f6484600640 1 --2- 172.21.15.117:0/3277227977 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f647c1096c0 0x7f647c10bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:18.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.637+0000 7f6484600640 1 --2- 172.21.15.117:0/3277227977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f647c074220 0x7f647c074680 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:18.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.637+0000 7f6484600640 1 --2- 172.21.15.117:0/3277227977 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f647c075800 0x7f647c073c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:18.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.637+0000 7f6484600640 1 -- 172.21.15.117:0/3277227977 >> 172.21.15.117:0/3277227977 conn(0x7f647c0fcdc0 msgr2=0x7f647c0ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:18.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.638+0000 7f6484600640 1 -- 172.21.15.117:0/3277227977 shutdown_connections 2024-03-19T03:33:18.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.638+0000 7f6484600640 1 -- 172.21.15.117:0/3277227977 wait complete. 2024-03-19T03:33:18.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.638+0000 7f6484600640 1 Processor -- start 2024-03-19T03:33:18.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.639+0000 7f6484600640 1 -- start start 2024-03-19T03:33:18.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.639+0000 7f6484600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f647c074220 0x7f647c199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:18.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.639+0000 7f6484600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f647c075800 0x7f647c19a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:18.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.639+0000 7f6484600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f647c1096c0 0x7f647c1a1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:18.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.639+0000 7f6484600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f647c10f700 con 0x7f647c074220 2024-03-19T03:33:18.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.639+0000 7f6484600640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f647c10f580 con 0x7f647c1096c0 2024-03-19T03:33:18.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.639+0000 7f6484600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f647c10f880 con 0x7f647c075800 2024-03-19T03:33:18.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.639+0000 7f6481c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f647c074220 0x7f647c199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:18.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.639+0000 7f6482600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f647c1096c0 0x7f647c1a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:18.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.639+0000 7f6481c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f647c074220 0x7f647c199dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42180/0 (socket says 172.21.15.117:42180) 2024-03-19T03:33:18.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.639+0000 7f6481c00640 1 -- 172.21.15.117:0/3318062174 learned_addr learned my addr 172.21.15.117:0/3318062174 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:18.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.640+0000 7f6481200640 1 --2- 172.21.15.117:0/3318062174 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f647c075800 0x7f647c19a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:18.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.640+0000 7f6481c00640 1 -- 172.21.15.117:0/3318062174 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f647c075800 msgr2=0x7f647c19a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:18.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.640+0000 7f6481c00640 1 --2- 172.21.15.117:0/3318062174 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f647c075800 0x7f647c19a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:18.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.640+0000 7f6481c00640 1 -- 172.21.15.117:0/3318062174 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f647c1096c0 msgr2=0x7f647c1a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:18.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.640+0000 7f6481c00640 1 --2- 172.21.15.117:0/3318062174 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f647c1096c0 0x7f647c1a1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:18.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.640+0000 7f6481c00640 1 -- 172.21.15.117:0/3318062174 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f647c1a19d0 con 0x7f647c074220 2024-03-19T03:33:18.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.640+0000 7f6482600640 1 --2- 172.21.15.117:0/3318062174 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f647c1096c0 0x7f647c1a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:33:18.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.640+0000 7f6481c00640 1 --2- 172.21.15.117:0/3318062174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f647c074220 0x7f647c199dc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f647000e9f0 tx=0x7f647000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:18.642 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.640+0000 7f647a000640 1 -- 172.21.15.117:0/3318062174 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647000ccc0 con 0x7f647c074220 2024-03-19T03:33:18.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.641+0000 7f647a000640 1 -- 172.21.15.117:0/3318062174 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f647000ce60 con 0x7f647c074220 2024-03-19T03:33:18.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.641+0000 7f6484600640 1 -- 172.21.15.117:0/3318062174 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f647c1a1cc0 con 0x7f647c074220 2024-03-19T03:33:18.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.641+0000 7f647a000640 1 -- 172.21.15.117:0/3318062174 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6470010690 con 0x7f647c074220 2024-03-19T03:33:18.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.641+0000 7f6484600640 1 -- 172.21.15.117:0/3318062174 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f647c1a21c0 con 0x7f647c074220 2024-03-19T03:33:18.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.642+0000 7f6484600640 1 -- 172.21.15.117:0/3318062174 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f647c108480 con 0x7f647c074220 2024-03-19T03:33:18.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.644+0000 7f647a000640 1 -- 172.21.15.117:0/3318062174 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f64700046c0 con 0x7f647c074220 2024-03-19T03:33:18.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.645+0000 7f647a000640 1 --2- 172.21.15.117:0/3318062174 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f64640a2800 0x7f64640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:18.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.645+0000 7f647a000640 1 -- 172.21.15.117:0/3318062174 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f64700cf290 con 0x7f647c074220 2024-03-19T03:33:18.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.645+0000 7f6481200640 1 --2- 172.21.15.117:0/3318062174 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f64640a2800 0x7f64640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:18.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.646+0000 7f6481200640 1 --2- 172.21.15.117:0/3318062174 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f64640a2800 0x7f64640a4cc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f646802c2d0 tx=0x7f646802c910 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:18.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.651+0000 7f647a000640 1 -- 172.21.15.117:0/3318062174 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f647009bf40 con 0x7f647c074220 2024-03-19T03:33:18.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:18 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/236555512' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:33:18.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:18 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2271138459' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:33:18.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:18 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/4131138259' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:33:18.780 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:18 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/236555512' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:33:18.780 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:18 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2271138459' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:33:18.780 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:18 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/4131138259' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:33:18.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:18 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/236555512' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:33:18.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:18 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2271138459' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:33:18.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:18 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/4131138259' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:33:18.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.855+0000 7f6484600640 1 -- 172.21.15.117:0/3318062174 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f647c0624c0 con 0x7f647c074220 2024-03-19T03:33:18.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.855+0000 7f647a000640 1 -- 172.21.15.117:0/3318062174 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f647009b840 con 0x7f647c074220 2024-03-19T03:33:18.856 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 32 2024-03-19T03:33:18.858 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.858+0000 7f6484600640 1 -- 172.21.15.117:0/3318062174 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f64640a2800 msgr2=0x7f64640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:18.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.858+0000 7f6484600640 1 --2- 172.21.15.117:0/3318062174 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f64640a2800 0x7f64640a4cc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f646802c2d0 tx=0x7f646802c910 comp rx=0 tx=0).stop 2024-03-19T03:33:18.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.858+0000 7f6484600640 1 -- 172.21.15.117:0/3318062174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f647c074220 msgr2=0x7f647c199dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:18.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.858+0000 7f6484600640 1 --2- 172.21.15.117:0/3318062174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f647c074220 0x7f647c199dc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f647000e9f0 tx=0x7f647000edf0 comp rx=0 tx=0).stop 2024-03-19T03:33:18.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.858+0000 7f6484600640 1 -- 172.21.15.117:0/3318062174 shutdown_connections 2024-03-19T03:33:18.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.858+0000 7f6484600640 1 --2- 172.21.15.117:0/3318062174 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f64640a2800 0x7f64640a4cc0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:18.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.858+0000 7f6484600640 1 --2- 172.21.15.117:0/3318062174 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f647c1096c0 0x7f647c1a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:18.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.858+0000 7f6484600640 1 --2- 172.21.15.117:0/3318062174 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f647c075800 0x7f647c19a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:18.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.858+0000 7f6484600640 1 --2- 172.21.15.117:0/3318062174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f647c074220 0x7f647c199dc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:18.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.858+0000 7f6484600640 1 -- 172.21.15.117:0/3318062174 >> 172.21.15.117:0/3318062174 conn(0x7f647c0fcdc0 msgr2=0x7f647c105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:18.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.859+0000 7f6484600640 1 -- 172.21.15.117:0/3318062174 shutdown_connections 2024-03-19T03:33:18.859 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:18.859+0000 7f6484600640 1 -- 172.21.15.117:0/3318062174 wait complete. 2024-03-19T03:33:18.877 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T03:33:19.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.069+0000 7f479e200640 1 -- 172.21.15.117:0/2076658043 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4798102910 msgr2=0x7f4798102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:19.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.069+0000 7f479e200640 1 --2- 172.21.15.117:0/2076658043 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4798102910 0x7f4798102d90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4784009900 tx=0x7f478402bc30 comp rx=0 tx=0).stop 2024-03-19T03:33:19.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.069+0000 7f479e200640 1 -- 172.21.15.117:0/2076658043 shutdown_connections 2024-03-19T03:33:19.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.069+0000 7f479e200640 1 --2- 172.21.15.117:0/2076658043 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f47981032d0 0x7f4798109b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.069+0000 7f479e200640 1 --2- 172.21.15.117:0/2076658043 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4798102910 0x7f4798102d90 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.069+0000 7f479e200640 1 --2- 172.21.15.117:0/2076658043 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4798101720 0x7f4798101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.069+0000 7f479e200640 1 -- 172.21.15.117:0/2076658043 >> 172.21.15.117:0/2076658043 conn(0x7f47980fcc90 msgr2=0x7f47980ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:19.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.069+0000 7f479e200640 1 -- 172.21.15.117:0/2076658043 shutdown_connections 2024-03-19T03:33:19.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.069+0000 7f479e200640 1 -- 172.21.15.117:0/2076658043 wait complete. 2024-03-19T03:33:19.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.070+0000 7f479e200640 1 Processor -- start 2024-03-19T03:33:19.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.070+0000 7f479e200640 1 -- start start 2024-03-19T03:33:19.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.070+0000 7f479e200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4798101720 0x7f4798199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:19.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.070+0000 7f479e200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4798102910 0x7f479819a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:19.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.070+0000 7f479e200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f47981032d0 0x7f47981a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:19.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.070+0000 7f479e200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f479810d3a0 con 0x7f4798101720 2024-03-19T03:33:19.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.070+0000 7f479e200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f479810d220 con 0x7f4798102910 2024-03-19T03:33:19.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.070+0000 7f479e200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f479810d520 con 0x7f47981032d0 2024-03-19T03:33:19.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.070+0000 7f479ce00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4798101720 0x7f4798199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:19.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.070+0000 7f4797e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4798102910 0x7f479819a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:19.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.070+0000 7f479ce00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4798101720 0x7f4798199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42196/0 (socket says 172.21.15.117:42196) 2024-03-19T03:33:19.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.070+0000 7f4797e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4798102910 0x7f479819a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52648/0 (socket says 172.21.15.117:52648) 2024-03-19T03:33:19.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.070+0000 7f479ce00640 1 -- 172.21.15.117:0/1347252122 learned_addr learned my addr 172.21.15.117:0/1347252122 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:19.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.071+0000 7f479d800640 1 --2- 172.21.15.117:0/1347252122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f47981032d0 0x7f47981a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:19.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.071+0000 7f479ce00640 1 -- 172.21.15.117:0/1347252122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f47981032d0 msgr2=0x7f47981a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:19.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.071+0000 7f479ce00640 1 --2- 172.21.15.117:0/1347252122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f47981032d0 0x7f47981a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.071+0000 7f479ce00640 1 -- 172.21.15.117:0/1347252122 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4798102910 msgr2=0x7f479819a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:19.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.071+0000 7f479ce00640 1 --2- 172.21.15.117:0/1347252122 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4798102910 0x7f479819a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.071+0000 7f479ce00640 1 -- 172.21.15.117:0/1347252122 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47981a1880 con 0x7f4798101720 2024-03-19T03:33:19.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.071+0000 7f479ce00640 1 --2- 172.21.15.117:0/1347252122 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4798101720 0x7f4798199c70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f478c00d950 tx=0x7f478c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:19.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.072+0000 7f4795600640 1 -- 172.21.15.117:0/1347252122 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c017070 con 0x7f4798101720 2024-03-19T03:33:19.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.072+0000 7f4795600640 1 -- 172.21.15.117:0/1347252122 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f478c009c30 con 0x7f4798101720 2024-03-19T03:33:19.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.072+0000 7f479e200640 1 -- 172.21.15.117:0/1347252122 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47981a1b70 con 0x7f4798101720 2024-03-19T03:33:19.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.072+0000 7f4795600640 1 -- 172.21.15.117:0/1347252122 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c00f600 con 0x7f4798101720 2024-03-19T03:33:19.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.072+0000 7f479e200640 1 -- 172.21.15.117:0/1347252122 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47981a2070 con 0x7f4798101720 2024-03-19T03:33:19.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.073+0000 7f479e200640 1 -- 172.21.15.117:0/1347252122 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47981a2240 con 0x7f4798101720 2024-03-19T03:33:19.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.076+0000 7f4795600640 1 -- 172.21.15.117:0/1347252122 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f478c01a070 con 0x7f4798101720 2024-03-19T03:33:19.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.078+0000 7f4795600640 1 --2- 172.21.15.117:0/1347252122 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f47800a2800 0x7f47800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:19.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.078+0000 7f4795600640 1 -- 172.21.15.117:0/1347252122 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f478c0ce8d0 con 0x7f4798101720 2024-03-19T03:33:19.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.078+0000 7f4797e00640 1 --2- 172.21.15.117:0/1347252122 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f47800a2800 0x7f47800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:19.079 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.079+0000 7f4797e00640 1 --2- 172.21.15.117:0/1347252122 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f47800a2800 0x7f47800a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f478402c7b0 tx=0x7f4784005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:19.080 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.081+0000 7f4795600640 1 -- 172.21.15.117:0/1347252122 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f478c009de0 con 0x7f4798101720 2024-03-19T03:33:19.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.281+0000 7f479e200640 1 -- 172.21.15.117:0/1347252122 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f47981a2410 con 0x7f4798101720 2024-03-19T03:33:19.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.281+0000 7f4795600640 1 -- 172.21.15.117:0/1347252122 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f478c09aed0 con 0x7f4798101720 2024-03-19T03:33:19.281 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 64 2024-03-19T03:33:19.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.284+0000 7f479e200640 1 -- 172.21.15.117:0/1347252122 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f47800a2800 msgr2=0x7f47800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:19.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.284+0000 7f479e200640 1 --2- 172.21.15.117:0/1347252122 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f47800a2800 0x7f47800a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f478402c7b0 tx=0x7f4784005fd0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.284+0000 7f479e200640 1 -- 172.21.15.117:0/1347252122 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4798101720 msgr2=0x7f4798199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:19.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.284+0000 7f479e200640 1 --2- 172.21.15.117:0/1347252122 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4798101720 0x7f4798199c70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f478c00d950 tx=0x7f478c00de10 comp rx=0 tx=0).stop 2024-03-19T03:33:19.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.284+0000 7f479e200640 1 -- 172.21.15.117:0/1347252122 shutdown_connections 2024-03-19T03:33:19.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.284+0000 7f479e200640 1 --2- 172.21.15.117:0/1347252122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f47981032d0 0x7f47981a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.284+0000 7f479e200640 1 --2- 172.21.15.117:0/1347252122 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f47800a2800 0x7f47800a4cc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.284+0000 7f479e200640 1 --2- 172.21.15.117:0/1347252122 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4798102910 0x7f479819a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.284+0000 7f479e200640 1 --2- 172.21.15.117:0/1347252122 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4798101720 0x7f4798199c70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.284+0000 7f479e200640 1 -- 172.21.15.117:0/1347252122 >> 172.21.15.117:0/1347252122 conn(0x7f47980fcc90 msgr2=0x7f4798073a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:19.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.284+0000 7f479e200640 1 -- 172.21.15.117:0/1347252122 shutdown_connections 2024-03-19T03:33:19.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.284+0000 7f479e200640 1 -- 172.21.15.117:0/1347252122 wait complete. 2024-03-19T03:33:19.305 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T03:33:19.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.508+0000 7f8e3b200640 1 -- 172.21.15.117:0/453139539 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e3c103920 msgr2=0x7f8e3c103d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:19.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.508+0000 7f8e3b200640 1 --2- 172.21.15.117:0/453139539 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e3c103920 0x7f8e3c103d40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c009900 tx=0x7f8e2c02bc30 comp rx=0 tx=0).stop 2024-03-19T03:33:19.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.508+0000 7f8e3b200640 1 -- 172.21.15.117:0/453139539 shutdown_connections 2024-03-19T03:33:19.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.508+0000 7f8e3b200640 1 --2- 172.21.15.117:0/453139539 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e3c1054d0 0x7f8e3c10bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.508+0000 7f8e3b200640 1 --2- 172.21.15.117:0/453139539 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8e3c104b10 0x7f8e3c104f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.508+0000 7f8e3b200640 1 --2- 172.21.15.117:0/453139539 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e3c103920 0x7f8e3c103d40 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.508+0000 7f8e3b200640 1 -- 172.21.15.117:0/453139539 >> 172.21.15.117:0/453139539 conn(0x7f8e3c0fee90 msgr2=0x7f8e3c1012d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:19.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.508+0000 7f8e3b200640 1 -- 172.21.15.117:0/453139539 shutdown_connections 2024-03-19T03:33:19.509 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.508+0000 7f8e3b200640 1 -- 172.21.15.117:0/453139539 wait complete. 2024-03-19T03:33:19.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.510+0000 7f8e3b200640 1 Processor -- start 2024-03-19T03:33:19.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.510+0000 7f8e3b200640 1 -- start start 2024-03-19T03:33:19.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.510+0000 7f8e3b200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8e3c103920 0x7f8e3c199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:19.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.510+0000 7f8e3b200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e3c104b10 0x7f8e3c19a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:19.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.510+0000 7f8e3b200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e3c1054d0 0x7f8e3c1a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:19.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.511+0000 7f8e3b200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3c10f590 con 0x7f8e3c104b10 2024-03-19T03:33:19.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.511+0000 7f8e3b200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3c10f410 con 0x7f8e3c1054d0 2024-03-19T03:33:19.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.511+0000 7f8e3b200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3c10f710 con 0x7f8e3c103920 2024-03-19T03:33:19.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.511+0000 7f8e39400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e3c104b10 0x7f8e3c19a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:19.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.511+0000 7f8e39400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e3c104b10 0x7f8e3c19a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42208/0 (socket says 172.21.15.117:42208) 2024-03-19T03:33:19.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.511+0000 7f8e39e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8e3c103920 0x7f8e3c199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:19.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.511+0000 7f8e3a800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e3c1054d0 0x7f8e3c1a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:19.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.511+0000 7f8e39400640 1 -- 172.21.15.117:0/3913760341 learned_addr learned my addr 172.21.15.117:0/3913760341 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:19.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.511+0000 7f8e39400640 1 -- 172.21.15.117:0/3913760341 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8e3c103920 msgr2=0x7f8e3c199c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:19.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.511+0000 7f8e39400640 1 --2- 172.21.15.117:0/3913760341 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8e3c103920 0x7f8e3c199c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.511+0000 7f8e39400640 1 -- 172.21.15.117:0/3913760341 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e3c1054d0 msgr2=0x7f8e3c1a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:19.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.511+0000 7f8e39400640 1 --2- 172.21.15.117:0/3913760341 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e3c1054d0 0x7f8e3c1a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.511+0000 7f8e39400640 1 -- 172.21.15.117:0/3913760341 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e3c1a1860 con 0x7f8e3c104b10 2024-03-19T03:33:19.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.511+0000 7f8e3a800640 1 --2- 172.21.15.117:0/3913760341 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e3c1054d0 0x7f8e3c1a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:33:19.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.512+0000 7f8e39400640 1 --2- 172.21.15.117:0/3913760341 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e3c104b10 0x7f8e3c19a190 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8e2800e9f0 tx=0x7f8e2800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:19.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.512+0000 7f8e32a00640 1 -- 172.21.15.117:0/3913760341 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2800cd80 con 0x7f8e3c104b10 2024-03-19T03:33:19.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.512+0000 7f8e3b200640 1 -- 172.21.15.117:0/3913760341 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e3c1a1b50 con 0x7f8e3c104b10 2024-03-19T03:33:19.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.512+0000 7f8e3b200640 1 -- 172.21.15.117:0/3913760341 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e3c1a2070 con 0x7f8e3c104b10 2024-03-19T03:33:19.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.512+0000 7f8e32a00640 1 -- 172.21.15.117:0/3913760341 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e28005500 con 0x7f8e3c104b10 2024-03-19T03:33:19.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.512+0000 7f8e32a00640 1 -- 172.21.15.117:0/3913760341 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e28010690 con 0x7f8e3c104b10 2024-03-19T03:33:19.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.513+0000 7f8e3b200640 1 -- 172.21.15.117:0/3913760341 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e1c0052c0 con 0x7f8e3c104b10 2024-03-19T03:33:19.517 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.517+0000 7f8e32a00640 1 -- 172.21.15.117:0/3913760341 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f8e280046c0 con 0x7f8e3c104b10 2024-03-19T03:33:19.520 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.520+0000 7f8e32a00640 1 --2- 172.21.15.117:0/3913760341 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8e200a2880 0x7f8e200a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:19.520 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.520+0000 7f8e32a00640 1 -- 172.21.15.117:0/3913760341 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8e280cf370 con 0x7f8e3c104b10 2024-03-19T03:33:19.520 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.520+0000 7f8e39e00640 1 --2- 172.21.15.117:0/3913760341 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8e200a2880 0x7f8e200a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:19.521 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.521+0000 7f8e32a00640 1 -- 172.21.15.117:0/3913760341 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8e28053740 con 0x7f8e3c104b10 2024-03-19T03:33:19.522 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.521+0000 7f8e39e00640 1 --2- 172.21.15.117:0/3913760341 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8e200a2880 0x7f8e200a4d40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c000c00 tx=0x7f8e2c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:19.586 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:19 smithi117 ceph-mon[31838]: pgmap v403: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:19.586 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:19 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3318062174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:33:19.586 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:19 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1347252122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:33:19.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:19 smithi119 ceph-mon[66861]: pgmap v403: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:19.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:19 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3318062174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:33:19.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:19 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1347252122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:33:19.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.755+0000 7f8e3b200640 1 -- 172.21.15.117:0/3913760341 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8e1c005880 con 0x7f8e3c104b10 2024-03-19T03:33:19.756 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.756+0000 7f8e32a00640 1 -- 172.21.15.117:0/3913760341 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f8e2809bfc0 con 0x7f8e3c104b10 2024-03-19T03:33:19.756 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 32 2024-03-19T03:33:19.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.758+0000 7f8e3b200640 1 -- 172.21.15.117:0/3913760341 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8e200a2880 msgr2=0x7f8e200a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:19.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.758+0000 7f8e3b200640 1 --2- 172.21.15.117:0/3913760341 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8e200a2880 0x7f8e200a4d40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c000c00 tx=0x7f8e2c005fd0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.758+0000 7f8e3b200640 1 -- 172.21.15.117:0/3913760341 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e3c104b10 msgr2=0x7f8e3c19a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:19.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.758+0000 7f8e3b200640 1 --2- 172.21.15.117:0/3913760341 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e3c104b10 0x7f8e3c19a190 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8e2800e9f0 tx=0x7f8e2800edf0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.758+0000 7f8e3b200640 1 -- 172.21.15.117:0/3913760341 shutdown_connections 2024-03-19T03:33:19.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.758+0000 7f8e3b200640 1 --2- 172.21.15.117:0/3913760341 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8e200a2880 0x7f8e200a4d40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.758+0000 7f8e3b200640 1 --2- 172.21.15.117:0/3913760341 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e3c1054d0 0x7f8e3c1a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.758+0000 7f8e3b200640 1 --2- 172.21.15.117:0/3913760341 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e3c104b10 0x7f8e3c19a190 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.758+0000 7f8e3b200640 1 --2- 172.21.15.117:0/3913760341 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8e3c103920 0x7f8e3c199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.758+0000 7f8e3b200640 1 -- 172.21.15.117:0/3913760341 >> 172.21.15.117:0/3913760341 conn(0x7f8e3c0fee90 msgr2=0x7f8e3c107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:19.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.759+0000 7f8e3b200640 1 -- 172.21.15.117:0/3913760341 shutdown_connections 2024-03-19T03:33:19.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.759+0000 7f8e3b200640 1 -- 172.21.15.117:0/3913760341 wait complete. 2024-03-19T03:33:19.780 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:33:19.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:19 smithi178 ceph-mon[67578]: pgmap v403: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:19.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:19 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3318062174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:33:19.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:19 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1347252122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:33:19.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.987+0000 7fcec4c00640 1 -- 172.21.15.117:0/797812103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcec0103a60 msgr2=0x7fcec0103e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:19.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.987+0000 7fcec4c00640 1 --2- 172.21.15.117:0/797812103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcec0103a60 0x7fcec0103e80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fceb0009900 tx=0x7fceb002bc40 comp rx=0 tx=0).stop 2024-03-19T03:33:19.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.988+0000 7fcec4c00640 1 -- 172.21.15.117:0/797812103 shutdown_connections 2024-03-19T03:33:19.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.988+0000 7fcec4c00640 1 --2- 172.21.15.117:0/797812103 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcec0105610 0x7fcec010bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.988+0000 7fcec4c00640 1 --2- 172.21.15.117:0/797812103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcec0104c50 0x7fcec01050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.988+0000 7fcec4c00640 1 --2- 172.21.15.117:0/797812103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcec0103a60 0x7fcec0103e80 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.988+0000 7fcec4c00640 1 -- 172.21.15.117:0/797812103 >> 172.21.15.117:0/797812103 conn(0x7fcec00ff010 msgr2=0x7fcec0101430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:19.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.988+0000 7fcec4c00640 1 -- 172.21.15.117:0/797812103 shutdown_connections 2024-03-19T03:33:19.988 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.988+0000 7fcec4c00640 1 -- 172.21.15.117:0/797812103 wait complete. 2024-03-19T03:33:19.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.989+0000 7fcec4c00640 1 Processor -- start 2024-03-19T03:33:19.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.989+0000 7fcec4c00640 1 -- start start 2024-03-19T03:33:19.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.989+0000 7fcec4c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcec0103a60 0x7fcec0199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:19.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.990+0000 7fcec4c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcec0104c50 0x7fcec019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:19.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.990+0000 7fcec4c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcec0105610 0x7fcec01a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:19.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.990+0000 7fcec4c00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcec010f730 con 0x7fcec0103a60 2024-03-19T03:33:19.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.990+0000 7fcec4c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fcec010f5b0 con 0x7fcec0105610 2024-03-19T03:33:19.990 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.990+0000 7fcec4c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fcec010f8b0 con 0x7fcec0104c50 2024-03-19T03:33:19.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.990+0000 7fcebde00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcec0103a60 0x7fcec0199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:19.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.990+0000 7fcebe800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcec0105610 0x7fcec01a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:19.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.990+0000 7fcebde00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcec0103a60 0x7fcec0199d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42216/0 (socket says 172.21.15.117:42216) 2024-03-19T03:33:19.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.990+0000 7fcebde00640 1 -- 172.21.15.117:0/1195151463 learned_addr learned my addr 172.21.15.117:0/1195151463 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:19.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.990+0000 7fceb7e00640 1 --2- 172.21.15.117:0/1195151463 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcec0104c50 0x7fcec019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:19.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.990+0000 7fcebe800640 1 -- 172.21.15.117:0/1195151463 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcec0104c50 msgr2=0x7fcec019a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:19.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.990+0000 7fcebe800640 1 --2- 172.21.15.117:0/1195151463 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcec0104c50 0x7fcec019a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.990+0000 7fcebe800640 1 -- 172.21.15.117:0/1195151463 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcec0103a60 msgr2=0x7fcec0199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:19.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.990+0000 7fcebe800640 1 --2- 172.21.15.117:0/1195151463 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcec0103a60 0x7fcec0199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:19.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.990+0000 7fcebe800640 1 -- 172.21.15.117:0/1195151463 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec01a1990 con 0x7fcec0105610 2024-03-19T03:33:19.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.990+0000 7fcebde00640 1 --2- 172.21.15.117:0/1195151463 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcec0103a60 0x7fcec0199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:33:19.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.990+0000 7fceb7e00640 1 --2- 172.21.15.117:0/1195151463 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcec0104c50 0x7fcec019a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:33:19.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.991+0000 7fcebe800640 1 --2- 172.21.15.117:0/1195151463 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcec0105610 0x7fcec01a1320 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fceb800e940 tx=0x7fceb800ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:19.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.991+0000 7fceb6a00640 1 -- 172.21.15.117:0/1195151463 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb8004d20 con 0x7fcec0105610 2024-03-19T03:33:19.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.991+0000 7fceb6a00640 1 -- 172.21.15.117:0/1195151463 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fceb8005020 con 0x7fcec0105610 2024-03-19T03:33:19.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.991+0000 7fcec4c00640 1 -- 172.21.15.117:0/1195151463 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcec01a1c80 con 0x7fcec0105610 2024-03-19T03:33:19.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.991+0000 7fceb6a00640 1 -- 172.21.15.117:0/1195151463 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb80116b0 con 0x7fcec0105610 2024-03-19T03:33:19.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.991+0000 7fcec4c00640 1 -- 172.21.15.117:0/1195151463 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcec01a21d0 con 0x7fcec0105610 2024-03-19T03:33:19.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.992+0000 7fcec4c00640 1 -- 172.21.15.117:0/1195151463 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcec010ad20 con 0x7fcec0105610 2024-03-19T03:33:19.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.995+0000 7fceb6a00640 1 -- 172.21.15.117:0/1195151463 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fceb80054d0 con 0x7fcec0105610 2024-03-19T03:33:19.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.996+0000 7fceb6a00640 1 --2- 172.21.15.117:0/1195151463 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fcea40a2800 0x7fcea40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:19.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.997+0000 7fceb6a00640 1 -- 172.21.15.117:0/1195151463 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fceb80cf2c0 con 0x7fcec0105610 2024-03-19T03:33:19.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.997+0000 7fcebde00640 1 --2- 172.21.15.117:0/1195151463 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fcea40a2800 0x7fcea40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:19.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:19.997+0000 7fcebde00640 1 --2- 172.21.15.117:0/1195151463 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fcea40a2800 0x7fcea40a4cc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fceb0005ab0 tx=0x7fceb002cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:20.004 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.004+0000 7fceb6a00640 1 -- 172.21.15.117:0/1195151463 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fceb8016030 con 0x7fcec0105610 2024-03-19T03:33:20.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.246+0000 7fcec4c00640 1 -- 172.21.15.117:0/1195151463 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcec002d1f0 con 0x7fcec0105610 2024-03-19T03:33:20.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.247+0000 7fceb6a00640 1 -- 172.21.15.117:0/1195151463 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11953 (secure 0 0 0) 0x7fceb809bf70 con 0x7fcec0105610 2024-03-19T03:33:20.251 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:33:20.251 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/54638259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":54638259},{"type":"v1","addr":"172.21.15.117:6835","nonce":54638259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3608185039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3608185039},{"type":"v1","addr":"172.21.15.117:6837","nonce":3608185039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T03:30:35.081740+0000","modified":"2024-03-19T03:31:09.995133+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24599,"mds_1":24581,"mds_2":24569,"mds_3":24545,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":20,"addr":"172.21.15.119:6833/1291833214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1291833214},{"type":"v1","addr":"172.21.15.119:6833","nonce":1291833214}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6835/1579860804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1579860804},{"type":"v1","addr":"172.21.15.178:6835","nonce":1579860804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6835/1686938941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1686938941},{"type":"v1","addr":"172.21.15.119:6835","nonce":1686938941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6837/1770913538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":1770913538},{"type":"v1","addr":"172.21.15.178:6837","nonce":1770913538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6839/2655593383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2655593383},{"type":"v1","addr":"172.21.15.178:6839","nonce":2655593383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.119:6837/2054992738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2054992738},{"type":"v1","addr":"172.21.15.119:6837","nonce":2054992738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"g","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.117:6839/684335078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":684335078},{"type":"v1","addr":"172.21.15.117:6839","nonce":684335078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.117:6841/1329933892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6840","nonce":1329933892},{"type":"v1","addr":"172.21.15.117:6841","nonce":1329933892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/2383354833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2383354833},{"type":"v1","addr":"172.21.15.119:6839","nonce":2383354833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24581,24599,24569,24545,24467]},"id":1}]} 2024-03-19T03:33:20.252 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 28 2024-03-19T03:33:20.252 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.250+0000 7fcec4c00640 1 -- 172.21.15.117:0/1195151463 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fcea40a2800 msgr2=0x7fcea40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:20.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.250+0000 7fcec4c00640 1 --2- 172.21.15.117:0/1195151463 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fcea40a2800 0x7fcea40a4cc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fceb0005ab0 tx=0x7fceb002cf90 comp rx=0 tx=0).stop 2024-03-19T03:33:20.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.250+0000 7fcec4c00640 1 -- 172.21.15.117:0/1195151463 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcec0105610 msgr2=0x7fcec01a1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:20.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.250+0000 7fcec4c00640 1 --2- 172.21.15.117:0/1195151463 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcec0105610 0x7fcec01a1320 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fceb800e940 tx=0x7fceb800ee00 comp rx=0 tx=0).stop 2024-03-19T03:33:20.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.250+0000 7fcec4c00640 1 -- 172.21.15.117:0/1195151463 shutdown_connections 2024-03-19T03:33:20.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.250+0000 7fcec4c00640 1 --2- 172.21.15.117:0/1195151463 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fcea40a2800 0x7fcea40a4cc0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:20.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.250+0000 7fcec4c00640 1 --2- 172.21.15.117:0/1195151463 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcec0105610 0x7fcec01a1320 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:20.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.250+0000 7fcec4c00640 1 --2- 172.21.15.117:0/1195151463 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcec0104c50 0x7fcec019a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:20.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.250+0000 7fcec4c00640 1 --2- 172.21.15.117:0/1195151463 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcec0103a60 0x7fcec0199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:20.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.250+0000 7fcec4c00640 1 -- 172.21.15.117:0/1195151463 >> 172.21.15.117:0/1195151463 conn(0x7fcec00ff010 msgr2=0x7fcec0107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:20.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.250+0000 7fcec4c00640 1 -- 172.21.15.117:0/1195151463 shutdown_connections 2024-03-19T03:33:20.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.251+0000 7fcec4c00640 1 -- 172.21.15.117:0/1195151463 wait complete. 2024-03-19T03:33:20.270 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:33:20.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.488+0000 7f6b88000640 1 -- 172.21.15.117:0/1578650904 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b80100a30 msgr2=0x7f6b80100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:20.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.488+0000 7f6b88000640 1 --2- 172.21.15.117:0/1578650904 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b80100a30 0x7f6b80100e90 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c009900 tx=0x7f6b6c02bc30 comp rx=0 tx=0).stop 2024-03-19T03:33:20.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.489+0000 7f6b88000640 1 -- 172.21.15.117:0/1578650904 shutdown_connections 2024-03-19T03:33:20.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.489+0000 7f6b88000640 1 --2- 172.21.15.117:0/1578650904 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6b801013d0 0x7f6b80101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:20.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.489+0000 7f6b88000640 1 --2- 172.21.15.117:0/1578650904 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b80100a30 0x7f6b80100e90 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:20.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.489+0000 7f6b88000640 1 --2- 172.21.15.117:0/1578650904 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b801000d0 0x7f6b801004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:20.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.489+0000 7f6b88000640 1 -- 172.21.15.117:0/1578650904 >> 172.21.15.117:0/1578650904 conn(0x7f6b800fbcb0 msgr2=0x7f6b800fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:20.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.489+0000 7f6b88000640 1 -- 172.21.15.117:0/1578650904 shutdown_connections 2024-03-19T03:33:20.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.489+0000 7f6b88000640 1 -- 172.21.15.117:0/1578650904 wait complete. 2024-03-19T03:33:20.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.490+0000 7f6b88000640 1 Processor -- start 2024-03-19T03:33:20.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.490+0000 7f6b88000640 1 -- start start 2024-03-19T03:33:20.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.490+0000 7f6b88000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b801000d0 0x7f6b80199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:20.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.490+0000 7f6b88000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6b80100a30 0x7f6b8019a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:20.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.490+0000 7f6b88000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b801013d0 0x7f6b801a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:20.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.490+0000 7f6b88000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8010f700 con 0x7f6b801013d0 2024-03-19T03:33:20.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.490+0000 7f6b88000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8010f580 con 0x7f6b801000d0 2024-03-19T03:33:20.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.490+0000 7f6b88000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8010f880 con 0x7f6b80100a30 2024-03-19T03:33:20.491 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.491+0000 7f6b85600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b801000d0 0x7f6b80199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:20.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.491+0000 7f6b85600640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b801000d0 0x7f6b80199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52688/0 (socket says 172.21.15.117:52688) 2024-03-19T03:33:20.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.491+0000 7f6b85600640 1 -- 172.21.15.117:0/1419837190 learned_addr learned my addr 172.21.15.117:0/1419837190 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:20.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.491+0000 7f6b84c00640 1 --2- 172.21.15.117:0/1419837190 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6b80100a30 0x7f6b8019a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:20.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.491+0000 7f6b86000640 1 --2- 172.21.15.117:0/1419837190 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b801013d0 0x7f6b801a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:20.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.491+0000 7f6b85600640 1 -- 172.21.15.117:0/1419837190 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6b80100a30 msgr2=0x7f6b8019a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:20.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.491+0000 7f6b85600640 1 --2- 172.21.15.117:0/1419837190 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6b80100a30 0x7f6b8019a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:20.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.491+0000 7f6b85600640 1 -- 172.21.15.117:0/1419837190 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b801013d0 msgr2=0x7f6b801a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:20.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.491+0000 7f6b85600640 1 --2- 172.21.15.117:0/1419837190 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b801013d0 0x7f6b801a12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:20.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.491+0000 7f6b85600640 1 -- 172.21.15.117:0/1419837190 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b801a1950 con 0x7f6b801000d0 2024-03-19T03:33:20.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.491+0000 7f6b86000640 1 --2- 172.21.15.117:0/1419837190 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b801013d0 0x7f6b801a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:33:20.492 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.491+0000 7f6b85600640 1 --2- 172.21.15.117:0/1419837190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b801000d0 0x7f6b80199d40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6b7400e9f0 tx=0x7f6b7400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:20.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.492+0000 7f6b7e000640 1 -- 172.21.15.117:0/1419837190 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7400cd80 con 0x7f6b801000d0 2024-03-19T03:33:20.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.492+0000 7f6b7e000640 1 -- 172.21.15.117:0/1419837190 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b74010910 con 0x7f6b801000d0 2024-03-19T03:33:20.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.492+0000 7f6b88000640 1 -- 172.21.15.117:0/1419837190 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b801a1c40 con 0x7f6b801000d0 2024-03-19T03:33:20.493 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.492+0000 7f6b7e000640 1 -- 172.21.15.117:0/1419837190 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b74018990 con 0x7f6b801000d0 2024-03-19T03:33:20.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.492+0000 7f6b88000640 1 -- 172.21.15.117:0/1419837190 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b801a2190 con 0x7f6b801000d0 2024-03-19T03:33:20.494 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.493+0000 7f6b88000640 1 -- 172.21.15.117:0/1419837190 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b8010af70 con 0x7f6b801000d0 2024-03-19T03:33:20.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.494+0000 7f6b7e000640 1 -- 172.21.15.117:0/1419837190 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f6b74018b30 con 0x7f6b801000d0 2024-03-19T03:33:20.495 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.496+0000 7f6b7e000640 1 --2- 172.21.15.117:0/1419837190 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f6b680a2880 0x7f6b680a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:20.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.496+0000 7f6b7e000640 1 -- 172.21.15.117:0/1419837190 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f6b740cf220 con 0x7f6b801000d0 2024-03-19T03:33:20.496 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.496+0000 7f6b84c00640 1 --2- 172.21.15.117:0/1419837190 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f6b680a2880 0x7f6b680a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:20.497 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.497+0000 7f6b84c00640 1 --2- 172.21.15.117:0/1419837190 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f6b680a2880 0x7f6b680a4d40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c02c240 tx=0x7f6b6c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:20.503 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.504+0000 7f6b7e000640 1 -- 172.21.15.117:0/1419837190 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6b7409bed0 con 0x7f6b801000d0 2024-03-19T03:33:20.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:20 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3913760341' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:33:20.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:20 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1195151463' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:33:20.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:20 smithi117 ceph-mon[31838]: pgmap v404: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:33:20.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.752+0000 7f6b88000640 1 -- 172.21.15.117:0/1419837190 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6b80062450 con 0x7f6b801000d0 2024-03-19T03:33:20.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.755+0000 7f6b7e000640 1 -- 172.21.15.117:0/1419837190 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24989 (secure 0 0 0) 0x7f6b7409b7d0 con 0x7f6b801000d0 2024-03-19T03:33:20.755 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:33:20.755 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":93,"fsid":"a3a82456-e59f-11ee-95c9-87774f69a715","created":"2024-03-19T03:21:30.427136+0000","modified":"2024-03-19T03:32:40.453890+0000","last_up_change":"2024-03-19T03:29:32.967505+0000","last_in_change":"2024-03-19T03:29:12.848740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:25:18.824270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T03:30:33.137170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T03:30:34.598838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T03:30:38.230023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b32eb0ee-9d4b-4748-87ec-eda02c907415","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6803","nonce":3014934900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6805","nonce":3014934900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6809","nonce":3014934900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6807","nonce":3014934900}]},"public_addr":"172.21.15.117:6803/3014934900","cluster_addr":"172.21.15.117:6805/3014934900","heartbeat_back_addr":"172.21.15.117:6809/3014934900","heartbeat_front_addr":"172.21.15.117:6807/3014934900","state":["exists","up"]},{"osd":1,"uuid":"e07585be-c016-4466-b07c-abbc22a58de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6801","nonce":2425927341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6803","nonce":2425927341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6807","nonce":2425927341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6805","nonce":2425927341}]},"public_addr":"172.21.15.119:6801/2425927341","cluster_addr":"172.21.15.119:6803/2425927341","heartbeat_back_addr":"172.21.15.119:6807/2425927341","heartbeat_front_addr":"172.21.15.119:6805/2425927341","state":["exists","up"]},{"osd":2,"uuid":"ede9edda-3b9f-41ca-ad04-5dcb738ab757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6801","nonce":3733812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6803","nonce":3733812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6807","nonce":3733812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6805","nonce":3733812989}]},"public_addr":"172.21.15.178:6801/3733812989","cluster_addr":"172.21.15.178:6803/3733812989","heartbeat_back_addr":"172.21.15.178:6807/3733812989","heartbeat_front_addr":"172.21.15.178:6805/3733812989","state":["exists","up"]},{"osd":3,"uuid":"f81c41e1-3a69-40db-a90f-58151286b081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6811","nonce":2178310135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6813","nonce":2178310135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6817","nonce":2178310135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6815","nonce":2178310135}]},"public_addr":"172.21.15.117:6811/2178310135","cluster_addr":"172.21.15.117:6813/2178310135","heartbeat_back_addr":"172.21.15.117:6817/2178310135","heartbeat_front_addr":"172.21.15.117:6815/2178310135","state":["exists","up"]},{"osd":4,"uuid":"caf7ce86-caf0-4eaf-bbab-bd2d93e63c33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6809","nonce":3455398086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6811","nonce":3455398086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6815","nonce":3455398086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6813","nonce":3455398086}]},"public_addr":"172.21.15.119:6809/3455398086","cluster_addr":"172.21.15.119:6811/3455398086","heartbeat_back_addr":"172.21.15.119:6815/3455398086","heartbeat_front_addr":"172.21.15.119:6813/3455398086","state":["exists","up"]},{"osd":5,"uuid":"20ba0b8a-2e0f-4ea7-b435-430e44766200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6809","nonce":1259631912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6811","nonce":1259631912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6815","nonce":1259631912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6813","nonce":1259631912}]},"public_addr":"172.21.15.178:6809/1259631912","cluster_addr":"172.21.15.178:6811/1259631912","heartbeat_back_addr":"172.21.15.178:6815/1259631912","heartbeat_front_addr":"172.21.15.178:6813/1259631912","state":["exists","up"]},{"osd":6,"uuid":"75c2ddd7-f837-42e0-ac47-8cfbcf97a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6819","nonce":864146757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6821","nonce":864146757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6825","nonce":864146757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6823","nonce":864146757}]},"public_addr":"172.21.15.117:6819/864146757","cluster_addr":"172.21.15.117:6821/864146757","heartbeat_back_addr":"172.21.15.117:6825/864146757","heartbeat_front_addr":"172.21.15.117:6823/864146757","state":["exists","up"]},{"osd":7,"uuid":"929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6817","nonce":1273365545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6819","nonce":1273365545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6823","nonce":1273365545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6821","nonce":1273365545}]},"public_addr":"172.21.15.119:6817/1273365545","cluster_addr":"172.21.15.119:6819/1273365545","heartbeat_back_addr":"172.21.15.119:6823/1273365545","heartbeat_front_addr":"172.21.15.119:6821/1273365545","state":["exists","up"]},{"osd":8,"uuid":"fd371461-a538-4781-92e1-5eead4ca3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6817","nonce":1059587501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6819","nonce":1059587501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6823","nonce":1059587501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6821","nonce":1059587501}]},"public_addr":"172.21.15.178:6817/1059587501","cluster_addr":"172.21.15.178:6819/1059587501","heartbeat_back_addr":"172.21.15.178:6823/1059587501","heartbeat_front_addr":"172.21.15.178:6821/1059587501","state":["exists","up"]},{"osd":9,"uuid":"0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6827","nonce":3653875600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6829","nonce":3653875600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6833","nonce":3653875600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6831","nonce":3653875600}]},"public_addr":"172.21.15.117:6827/3653875600","cluster_addr":"172.21.15.117:6829/3653875600","heartbeat_back_addr":"172.21.15.117:6833/3653875600","heartbeat_front_addr":"172.21.15.117:6831/3653875600","state":["exists","up"]},{"osd":10,"uuid":"de2049da-d19e-4386-afbf-fd6b5fddbcba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6825","nonce":1050976194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6827","nonce":1050976194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6831","nonce":1050976194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6829","nonce":1050976194}]},"public_addr":"172.21.15.119:6825/1050976194","cluster_addr":"172.21.15.119:6827/1050976194","heartbeat_back_addr":"172.21.15.119:6831/1050976194","heartbeat_front_addr":"172.21.15.119:6829/1050976194","state":["exists","up"]},{"osd":11,"uuid":"3cccc0f9-d273-4b6b-9b8e-c926371472dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6825","nonce":2915080368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6827","nonce":2915080368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6831","nonce":2915080368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6829","nonce":2915080368}]},"public_addr":"172.21.15.178:6825/2915080368","cluster_addr":"172.21.15.178:6827/2915080368","heartbeat_back_addr":"172.21.15.178:6831/2915080368","heartbeat_front_addr":"172.21.15.178:6829/2915080368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:27.876274+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:53.094706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:16.598813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:48.160419+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:14.048925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:38.814304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:12.542328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:38.942747+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:04.488963+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:38.196053+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:05.254728+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:30.874915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7},{"from":5,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.117:0/185684447":"2024-03-20T03:22:38.134578+0000","172.21.15.117:0/361367101":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/2788067603":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3506200675":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6800/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3283471541":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3750920169":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/2780813298":"2024-03-20T03:22:03.905461+0000","172.21.15.117:6800/2788067603":"2024-03-20T03:22:38.134578+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:33:20.757 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.757+0000 7f6b88000640 1 -- 172.21.15.117:0/1419837190 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f6b680a2880 msgr2=0x7f6b680a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:20.757 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.757+0000 7f6b88000640 1 --2- 172.21.15.117:0/1419837190 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f6b680a2880 0x7f6b680a4d40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c02c240 tx=0x7f6b6c02df90 comp rx=0 tx=0).stop 2024-03-19T03:33:20.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.757+0000 7f6b88000640 1 -- 172.21.15.117:0/1419837190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b801000d0 msgr2=0x7f6b80199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:20.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.757+0000 7f6b88000640 1 --2- 172.21.15.117:0/1419837190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b801000d0 0x7f6b80199d40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6b7400e9f0 tx=0x7f6b7400eeb0 comp rx=0 tx=0).stop 2024-03-19T03:33:20.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.757+0000 7f6b88000640 1 -- 172.21.15.117:0/1419837190 shutdown_connections 2024-03-19T03:33:20.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.757+0000 7f6b88000640 1 --2- 172.21.15.117:0/1419837190 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b801013d0 0x7f6b801a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:20.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.757+0000 7f6b88000640 1 --2- 172.21.15.117:0/1419837190 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6b80100a30 0x7f6b8019a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:20.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.757+0000 7f6b88000640 1 --2- 172.21.15.117:0/1419837190 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f6b680a2880 0x7f6b680a4d40 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:20.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.758+0000 7f6b88000640 1 --2- 172.21.15.117:0/1419837190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b801000d0 0x7f6b80199d40 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:20.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.758+0000 7f6b88000640 1 -- 172.21.15.117:0/1419837190 >> 172.21.15.117:0/1419837190 conn(0x7f6b800fbcb0 msgr2=0x7f6b800fd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:20.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.758+0000 7f6b88000640 1 -- 172.21.15.117:0/1419837190 shutdown_connections 2024-03-19T03:33:20.758 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:20.758+0000 7f6b88000640 1 -- 172.21.15.117:0/1419837190 wait complete. 2024-03-19T03:33:20.782 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-19T03:30:35.081740+0000', 'modified': '2024-03-19T03:31:09.995133+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24599, 'mds_1': 24581, 'mds_2': 24569, 'mds_3': 24545, 'mds_4': 24467}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24442': {'gid': 24442, 'name': 'l', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.178:6833/3751720259', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6832', 'nonce': 3751720259}, {'type': 'v1', 'addr': '172.21.15.178:6833', 'nonce': 3751720259}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24467': {'gid': 24467, 'name': 'k', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 20, 'addr': '172.21.15.119:6833/1291833214', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6832', 'nonce': 1291833214}, {'type': 'v1', 'addr': '172.21.15.119:6833', 'nonce': 1291833214}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24508': {'gid': 24508, 'name': 'f', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.178:6835/1579860804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6834', 'nonce': 1579860804}, {'type': 'v1', 'addr': '172.21.15.178:6835', 'nonce': 1579860804}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24515': {'gid': 24515, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.119:6835/1686938941', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 1686938941}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 1686938941}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24520': {'gid': 24520, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.178:6837/1770913538', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6836', 'nonce': 1770913538}, {'type': 'v1', 'addr': '172.21.15.178:6837', 'nonce': 1770913538}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24541': {'gid': 24541, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.178:6839/2655593383', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6838', 'nonce': 2655593383}, {'type': 'v1', 'addr': '172.21.15.178:6839', 'nonce': 2655593383}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24545': {'gid': 24545, 'name': 'h', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.119:6837/2054992738', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6836', 'nonce': 2054992738}, {'type': 'v1', 'addr': '172.21.15.119:6837', 'nonce': 2054992738}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24569': {'gid': 24569, 'name': 'g', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.117:6839/684335078', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6838', 'nonce': 684335078}, {'type': 'v1', 'addr': '172.21.15.117:6839', 'nonce': 684335078}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24581': {'gid': 24581, 'name': 'a', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.117:6841/1329933892', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6840', 'nonce': 1329933892}, {'type': 'v1', 'addr': '172.21.15.117:6841', 'nonce': 1329933892}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24599': {'gid': 24599, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.119:6839/2383354833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6838', 'nonce': 2383354833}, {'type': 'v1', 'addr': '172.21.15.119:6839', 'nonce': 2383354833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24599, 'qdb_cluster': [24581, 24599, 24569, 24545, 24467]} 2024-03-19T03:33:20.782 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-19T03:33:20.782 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.k status 2024-03-19T03:33:20.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:20 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3913760341' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:33:20.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:20 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1195151463' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:33:20.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:20 smithi178 ceph-mon[67578]: pgmap v404: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:33:20.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:20 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3913760341' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:33:20.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:20 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1195151463' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:33:20.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:20 smithi119 ceph-mon[66861]: pgmap v404: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:33:21.008 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.008+0000 7f20c1c00640 1 -- 172.21.15.117:0/2094962871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20bc1031e0 msgr2=0x7f20bc109a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.008+0000 7f20c1c00640 1 --2- 172.21.15.117:0/2094962871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20bc1031e0 0x7f20bc109a70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f20b4009ca0 tx=0x7f20b402c100 comp rx=0 tx=0).stop 2024-03-19T03:33:21.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.008+0000 7f20c1c00640 1 -- 172.21.15.117:0/2094962871 shutdown_connections 2024-03-19T03:33:21.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.008+0000 7f20c1c00640 1 --2- 172.21.15.117:0/2094962871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20bc1031e0 0x7f20bc109a70 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.008+0000 7f20c1c00640 1 --2- 172.21.15.117:0/2094962871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20bc102820 0x7f20bc102ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.008+0000 7f20c1c00640 1 --2- 172.21.15.117:0/2094962871 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20bc101630 0x7f20bc101a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.008+0000 7f20c1c00640 1 -- 172.21.15.117:0/2094962871 >> 172.21.15.117:0/2094962871 conn(0x7f20bc0fcba0 msgr2=0x7f20bc0fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:21.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.008+0000 7f20c1c00640 1 -- 172.21.15.117:0/2094962871 shutdown_connections 2024-03-19T03:33:21.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.008+0000 7f20c1c00640 1 -- 172.21.15.117:0/2094962871 wait complete. 2024-03-19T03:33:21.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.009+0000 7f20c1c00640 1 Processor -- start 2024-03-19T03:33:21.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.010+0000 7f20c1c00640 1 -- start start 2024-03-19T03:33:21.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.010+0000 7f20c1c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20bc101630 0x7f20bc199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.011 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.010+0000 7f20c1c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20bc102820 0x7f20bc19a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.010+0000 7f20c1c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20bc1031e0 0x7f20bc1a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.010+0000 7f20c1c00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f20bc10d2b0 con 0x7f20bc101630 2024-03-19T03:33:21.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.010+0000 7f20c1c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f20bc10d130 con 0x7f20bc102820 2024-03-19T03:33:21.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.010+0000 7f20c1c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f20bc10d430 con 0x7f20bc1031e0 2024-03-19T03:33:21.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.010+0000 7f20b3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20bc102820 0x7f20bc19a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.010+0000 7f20b3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20bc102820 0x7f20bc19a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52694/0 (socket says 172.21.15.117:52694) 2024-03-19T03:33:21.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.010+0000 7f20b3e00640 1 -- 172.21.15.117:0/3473288920 learned_addr learned my addr 172.21.15.117:0/3473288920 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:21.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.010+0000 7f20c1200640 1 --2- 172.21.15.117:0/3473288920 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20bc1031e0 0x7f20bc1a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.010+0000 7f20bbe00640 1 --2- 172.21.15.117:0/3473288920 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20bc101630 0x7f20bc199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.011+0000 7f20b3e00640 1 -- 172.21.15.117:0/3473288920 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20bc1031e0 msgr2=0x7f20bc1a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.011+0000 7f20b3e00640 1 --2- 172.21.15.117:0/3473288920 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20bc1031e0 0x7f20bc1a1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.011+0000 7f20b3e00640 1 -- 172.21.15.117:0/3473288920 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20bc101630 msgr2=0x7f20bc199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.011+0000 7f20b3e00640 1 --2- 172.21.15.117:0/3473288920 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20bc101630 0x7f20bc199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.012 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.011+0000 7f20b3e00640 1 -- 172.21.15.117:0/3473288920 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20bc1a16f0 con 0x7f20bc102820 2024-03-19T03:33:21.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.011+0000 7f20b3e00640 1 --2- 172.21.15.117:0/3473288920 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20bc102820 0x7f20bc19a010 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f20a400ea60 tx=0x7f20a400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:21.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.011+0000 7f20b9600640 1 -- 172.21.15.117:0/3473288920 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a400cde0 con 0x7f20bc102820 2024-03-19T03:33:21.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.011+0000 7f20b9600640 1 -- 172.21.15.117:0/3473288920 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20a40035f0 con 0x7f20bc102820 2024-03-19T03:33:21.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.011+0000 7f20c1c00640 1 -- 172.21.15.117:0/3473288920 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20bc1a19e0 con 0x7f20bc102820 2024-03-19T03:33:21.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.011+0000 7f20b9600640 1 -- 172.21.15.117:0/3473288920 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a4010670 con 0x7f20bc102820 2024-03-19T03:33:21.013 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.011+0000 7f20c1c00640 1 -- 172.21.15.117:0/3473288920 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20bc1a1f30 con 0x7f20bc102820 2024-03-19T03:33:21.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.016+0000 7f20b9600640 1 -- 172.21.15.117:0/3473288920 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f20a40037a0 con 0x7f20bc102820 2024-03-19T03:33:21.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.016+0000 7f20c5f65740 1 Processor -- start 2024-03-19T03:33:21.016 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.017+0000 7f20c5f65740 1 -- start start 2024-03-19T03:33:21.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.017+0000 7f20c5f65740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a4d8f7f5c0 0x55a4d8f7f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.017+0000 7f20c5f65740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7ffb0 0x55a4d8f84ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.017+0000 7f20c5f65740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a4d8f85120 0x55a4d8f875b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.017+0000 7f20c5f65740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a4d8ee3d00 con 0x55a4d8f7f5c0 2024-03-19T03:33:21.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.017+0000 7f20c5f65740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55a4d8e8cb00 con 0x55a4d8f7ffb0 2024-03-19T03:33:21.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.017+0000 7f20c5f65740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x55a4d8edfda0 con 0x55a4d8f85120 2024-03-19T03:33:21.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.017+0000 7f20bbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7ffb0 0x55a4d8f84ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.017+0000 7f20c1200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a4d8f7f5c0 0x55a4d8f7f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.017+0000 7f20bbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7ffb0 0x55a4d8f84ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52698/0 (socket says 172.21.15.117:52698) 2024-03-19T03:33:21.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.017+0000 7f20bbe00640 1 -- 172.21.15.117:0/103753203 learned_addr learned my addr 172.21.15.117:0/103753203 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:21.018 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.018+0000 7f20b3e00640 1 --2- 172.21.15.117:0/103753203 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a4d8f85120 0x55a4d8f875b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.018+0000 7f20bbe00640 1 -- 172.21.15.117:0/103753203 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a4d8f85120 msgr2=0x55a4d8f875b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.018+0000 7f20bbe00640 1 --2- 172.21.15.117:0/103753203 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a4d8f85120 0x55a4d8f875b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.018+0000 7f20bbe00640 1 -- 172.21.15.117:0/103753203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a4d8f7f5c0 msgr2=0x55a4d8f7f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.018+0000 7f20bbe00640 1 --2- 172.21.15.117:0/103753203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a4d8f7f5c0 0x55a4d8f7f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.018+0000 7f20bbe00640 1 -- 172.21.15.117:0/103753203 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4d8bf82d0 con 0x55a4d8f7ffb0 2024-03-19T03:33:21.023 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.018+0000 7f20b3e00640 1 --2- 172.21.15.117:0/103753203 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a4d8f85120 0x55a4d8f875b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T03:33:21.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.018+0000 7f20bbe00640 1 --2- 172.21.15.117:0/103753203 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7ffb0 0x55a4d8f84ab0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f20ac009ca0 tx=0x7f20ac011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e06e201b3a6c838 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:21.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.018+0000 7f20b9600640 1 --2- 172.21.15.117:0/3473288920 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f20a00a2800 0x7f20a00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.018+0000 7f20b1600640 1 -- 172.21.15.117:0/103753203 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ac004220 con 0x55a4d8f7ffb0 2024-03-19T03:33:21.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.018+0000 7f20b9600640 1 -- 172.21.15.117:0/3473288920 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f20a40cf140 con 0x7f20bc102820 2024-03-19T03:33:21.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.018+0000 7f20b1600640 1 -- 172.21.15.117:0/103753203 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20ac0043c0 con 0x55a4d8f7ffb0 2024-03-19T03:33:21.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.018+0000 7f20b1600640 1 -- 172.21.15.117:0/103753203 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ac019900 con 0x55a4d8f7ffb0 2024-03-19T03:33:21.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.019+0000 7f20c1200640 1 --2- 172.21.15.117:0/3473288920 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f20a00a2800 0x7f20a00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.019+0000 7f20c5f65740 1 -- 172.21.15.117:0/103753203 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7ffb0 msgr2=0x55a4d8f84ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.019+0000 7f20c5f65740 1 --2- 172.21.15.117:0/103753203 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7ffb0 0x55a4d8f84ab0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f20ac009ca0 tx=0x7f20ac011810 comp rx=0 tx=0).stop 2024-03-19T03:33:21.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.019+0000 7f20c5f65740 1 -- 172.21.15.117:0/103753203 shutdown_connections 2024-03-19T03:33:21.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.019+0000 7f20c5f65740 1 --2- 172.21.15.117:0/103753203 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a4d8f85120 0x55a4d8f875b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.019+0000 7f20c5f65740 1 --2- 172.21.15.117:0/103753203 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7ffb0 0x55a4d8f84ab0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.019+0000 7f20c5f65740 1 --2- 172.21.15.117:0/103753203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a4d8f7f5c0 0x55a4d8f7f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.019+0000 7f20c5f65740 1 -- 172.21.15.117:0/103753203 >> 172.21.15.117:0/103753203 conn(0x55a4d8f7ab30 msgr2=0x55a4d8f7af30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:21.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.020+0000 7f20c1200640 1 --2- 172.21.15.117:0/3473288920 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f20a00a2800 0x7f20a00a4cc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f20b4005e30 tx=0x7f20b4032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:21.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.020+0000 7f20c5f65740 1 -- 172.21.15.117:0/103753203 shutdown_connections 2024-03-19T03:33:21.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.020+0000 7f20c5f65740 1 -- 172.21.15.117:0/103753203 wait complete. 2024-03-19T03:33:21.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.020+0000 7f20c5f65740 1 -- start start 2024-03-19T03:33:21.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.020+0000 7f20c5f65740 1 Processor -- start 2024-03-19T03:33:21.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.021+0000 7f20c5f65740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7f5c0 0x55a4d8f89ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.021+0000 7f20c5f65740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a4d8f7ffb0 0x55a4d8f89ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.021+0000 7f20c5f65740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a4d8f85120 0x55a4d8f91030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.021+0000 7f20c5f65740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a4d8f880d0 con 0x55a4d8f85120 2024-03-19T03:33:21.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.021+0000 7f20c5f65740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55a4d8e9fed0 con 0x55a4d8f7f5c0 2024-03-19T03:33:21.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.021+0000 7f20c5f65740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x55a4d8f88250 con 0x55a4d8f7ffb0 2024-03-19T03:33:21.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.021+0000 7f20c5f65740 10 client.0 _ll_register_callbacks cb 0x55a4d8e9dc10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:33:21.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.021+0000 7f20bbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7f5c0 0x55a4d8f89ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.021+0000 7f20bbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7f5c0 0x55a4d8f89ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52706/0 (socket says 172.21.15.117:52706) 2024-03-19T03:33:21.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.021+0000 7f20bbe00640 1 -- 172.21.15.117:0/3491757667 learned_addr learned my addr 172.21.15.117:0/3491757667 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:21.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.021+0000 7f20b3e00640 1 --2- 172.21.15.117:0/3491757667 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a4d8f7ffb0 0x55a4d8f89ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.021+0000 7f20c1200640 1 --2- 172.21.15.117:0/3491757667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a4d8f85120 0x55a4d8f91030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.021+0000 7f20bbe00640 1 -- 172.21.15.117:0/3491757667 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a4d8f7ffb0 msgr2=0x55a4d8f89ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.021+0000 7f20bbe00640 1 --2- 172.21.15.117:0/3491757667 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a4d8f7ffb0 0x55a4d8f89ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.021+0000 7f20bbe00640 1 -- 172.21.15.117:0/3491757667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a4d8f85120 msgr2=0x55a4d8f91030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.021+0000 7f20bbe00640 1 --2- 172.21.15.117:0/3491757667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a4d8f85120 0x55a4d8f91030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.021+0000 7f20bbe00640 1 -- 172.21.15.117:0/3491757667 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4d8cc26b0 con 0x55a4d8f7f5c0 2024-03-19T03:33:21.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.021+0000 7f20b3e00640 1 --2- 172.21.15.117:0/3491757667 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a4d8f7ffb0 0x55a4d8f89ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:33:21.026 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.021+0000 7f20c1200640 1 --2- 172.21.15.117:0/3491757667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a4d8f85120 0x55a4d8f91030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:33:21.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.022+0000 7f20bbe00640 1 --2- 172.21.15.117:0/3491757667 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7f5c0 0x55a4d8f89ab0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f20ac009c70 tx=0x7f20ac003e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d154f5e3fc66271e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:21.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.022+0000 7f20aa000640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-19T03:33:21.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.023+0000 7f20aa000640 1 -- 172.21.15.117:0/3491757667 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ac028070 con 0x55a4d8f7f5c0 2024-03-19T03:33:21.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.023+0000 7f20c5f65740 1 -- 172.21.15.117:0/3491757667 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20ac0043c0 con 0x55a4d8f7f5c0 2024-03-19T03:33:21.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.023+0000 7f20aa000640 1 -- 172.21.15.117:0/3491757667 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f20ac005390 con 0x55a4d8f7f5c0 2024-03-19T03:33:21.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.023+0000 7f20aa000640 1 -- 172.21.15.117:0/3491757667 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f20ac032050 con 0x55a4d8f7f5c0 2024-03-19T03:33:21.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.023+0000 7f20c5f65740 10 client.25376 fetch_fsmap learned FSMap version 28 2024-03-19T03:33:21.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.023+0000 7f20c5f65740 1 -- 172.21.15.117:0/3491757667 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a4d8f94360 con 0x55a4d8f7f5c0 2024-03-19T03:33:21.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.024+0000 7f20aa000640 1 -- 172.21.15.117:0/3491757667 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20ac0043c0 con 0x55a4d8f7f5c0 2024-03-19T03:33:21.029 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.024+0000 7f20aa000640 1 -- 172.21.15.117:0/3491757667 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f20ac004b00 con 0x55a4d8f7f5c0 2024-03-19T03:33:21.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.024+0000 7f20c5f65740 10 client.25376 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T03:33:21.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.024+0000 7f20c5f65740 10 client.25376 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-19T03:33:21.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.024+0000 7f20c5f65740 20 client.25376 populate_metadata read hostname 'smithi117' 2024-03-19T03:33:21.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.024+0000 7f20c5f65740 1 --2- 172.21.15.117:0/3491757667 >> [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] conn(0x55a4d8f96820 0x55a4d8f97c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.024+0000 7f20c5f65740 4 client.25376 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:33:21.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.024+0000 7f20c5f65740 1 -- 172.21.15.117:0/3491757667 --> [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a4d8ef1240 con 0x55a4d8f96820 2024-03-19T03:33:21.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.024+0000 7f20b3e00640 1 --2- 172.21.15.117:0/3491757667 >> [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] conn(0x55a4d8f96820 0x55a4d8f97c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.025+0000 7f20b3e00640 1 --2- 172.21.15.117:0/3491757667 >> [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] conn(0x55a4d8f96820 0x55a4d8f97c90 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=92a406305296fd7b server_cookie=af742e52ab4c5c82 in_seq=0 out_seq=0 2024-03-19T03:33:21.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.025+0000 7f20aa000640 10 client.25376 ms_handle_connect on v2:172.21.15.119:6832/1291833214 2024-03-19T03:33:21.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.026+0000 7f20aa000640 1 -- 172.21.15.117:0/3491757667 <== mds.4 v2:172.21.15.119:6832/1291833214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a4d8ef1240 con 0x55a4d8f96820 2024-03-19T03:33:21.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.026+0000 7f20aa000640 10 client.25376 handle_command_reply: tid=0 2024-03-19T03:33:21.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.026+0000 7f20c5f65740 1 client.25376 shutdown 2024-03-19T03:33:21.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.026+0000 7f20c5f65740 1 -- 172.21.15.117:0/3491757667 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7f5c0 msgr2=0x55a4d8f89ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.026+0000 7f20c5f65740 1 --2- 172.21.15.117:0/3491757667 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7f5c0 0x55a4d8f89ab0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f20ac009c70 tx=0x7f20ac003e30 comp rx=0 tx=0).stop 2024-03-19T03:33:21.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.026+0000 7f20c5f65740 1 -- 172.21.15.117:0/3491757667 shutdown_connections 2024-03-19T03:33:21.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.026+0000 7f20c5f65740 1 --2- 172.21.15.117:0/3491757667 >> [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] conn(0x55a4d8f96820 0x55a4d8f97c90 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.026+0000 7f20c5f65740 1 --2- 172.21.15.117:0/3491757667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a4d8f85120 0x55a4d8f91030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.026+0000 7f20c5f65740 1 --2- 172.21.15.117:0/3491757667 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a4d8f7ffb0 0x55a4d8f89ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.026+0000 7f20c5f65740 1 --2- 172.21.15.117:0/3491757667 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7f5c0 0x55a4d8f89ab0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.026+0000 7f20c5f65740 1 -- 172.21.15.117:0/3491757667 >> 172.21.15.117:0/3491757667 conn(0x55a4d8f7ab30 msgr2=0x55a4d8f79d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:21.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.026+0000 7f20aa000640 0 client.25376 ms_handle_reset on v2:172.21.15.119:6832/1291833214 2024-03-19T03:33:21.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.027+0000 7f20c5f65740 1 -- 172.21.15.117:0/3491757667 shutdown_connections 2024-03-19T03:33:21.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.027+0000 7f20c5f65740 1 -- 172.21.15.117:0/3491757667 wait complete. 2024-03-19T03:33:21.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.027+0000 7f20c5f65740 20 client.25376 trim_cache size 0 max 16384 2024-03-19T03:33:21.054 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.055+0000 7f20c5f65740 1 Processor -- start 2024-03-19T03:33:21.055 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.055+0000 7f20c5f65740 1 -- start start 2024-03-19T03:33:21.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.055+0000 7f20c5f65740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a4d8f7f5c0 0x55a4d8f87fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.055+0000 7f20c5f65740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7ffb0 0x55a4d8f90f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.055+0000 7f20c5f65740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a4d8f85120 0x55a4d8f87510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.056 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.055+0000 7f20c5f65740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a4d8f95fd0 con 0x55a4d8f85120 2024-03-19T03:33:21.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.055+0000 7f20c5f65740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55a4d8f966a0 con 0x55a4d8f7ffb0 2024-03-19T03:33:21.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.055+0000 7f20c5f65740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x55a4d8f96d70 con 0x55a4d8f7f5c0 2024-03-19T03:33:21.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.055+0000 7f20b3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7ffb0 0x55a4d8f90f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.055+0000 7f20b3e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7ffb0 0x55a4d8f90f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52712/0 (socket says 172.21.15.117:52712) 2024-03-19T03:33:21.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.055+0000 7f20b3e00640 1 -- 172.21.15.117:0/3626786346 learned_addr learned my addr 172.21.15.117:0/3626786346 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:21.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.056+0000 7f20c1200640 1 --2- 172.21.15.117:0/3626786346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a4d8f85120 0x55a4d8f87510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.056+0000 7f20bbe00640 1 --2- 172.21.15.117:0/3626786346 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a4d8f7f5c0 0x55a4d8f87fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.056+0000 7f20b3e00640 1 -- 172.21.15.117:0/3626786346 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a4d8f7f5c0 msgr2=0x55a4d8f87fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.056+0000 7f20b3e00640 1 --2- 172.21.15.117:0/3626786346 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a4d8f7f5c0 0x55a4d8f87fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.056+0000 7f20b3e00640 1 -- 172.21.15.117:0/3626786346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a4d8f85120 msgr2=0x55a4d8f87510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.056+0000 7f20b3e00640 1 --2- 172.21.15.117:0/3626786346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a4d8f85120 0x55a4d8f87510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.056+0000 7f20b3e00640 1 -- 172.21.15.117:0/3626786346 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4d8f80390 con 0x55a4d8f7ffb0 2024-03-19T03:33:21.057 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.056+0000 7f20c1200640 1 --2- 172.21.15.117:0/3626786346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a4d8f85120 0x55a4d8f87510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:33:21.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.056+0000 7f20b3e00640 1 --2- 172.21.15.117:0/3626786346 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7ffb0 0x55a4d8f90f50 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f20a400c760 tx=0x7f20a40d1650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b0694a64df9a002 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:21.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.056+0000 7f20b0c00640 1 -- 172.21.15.117:0/3626786346 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a409f840 con 0x55a4d8f7ffb0 2024-03-19T03:33:21.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.056+0000 7f20b0c00640 1 -- 172.21.15.117:0/3626786346 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20a40a3e90 con 0x55a4d8f7ffb0 2024-03-19T03:33:21.058 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.056+0000 7f20b0c00640 1 -- 172.21.15.117:0/3626786346 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a4003c20 con 0x55a4d8f7ffb0 2024-03-19T03:33:21.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.057+0000 7f20c5f65740 1 -- 172.21.15.117:0/3626786346 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7ffb0 msgr2=0x55a4d8f90f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.057+0000 7f20c5f65740 1 --2- 172.21.15.117:0/3626786346 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7ffb0 0x55a4d8f90f50 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f20a400c760 tx=0x7f20a40d1650 comp rx=0 tx=0).stop 2024-03-19T03:33:21.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.057+0000 7f20c5f65740 1 -- 172.21.15.117:0/3626786346 shutdown_connections 2024-03-19T03:33:21.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.057+0000 7f20c5f65740 1 --2- 172.21.15.117:0/3626786346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a4d8f85120 0x55a4d8f87510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.057+0000 7f20c5f65740 1 --2- 172.21.15.117:0/3626786346 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7ffb0 0x55a4d8f90f50 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.057+0000 7f20c5f65740 1 --2- 172.21.15.117:0/3626786346 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a4d8f7f5c0 0x55a4d8f87fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.057+0000 7f20c5f65740 1 -- 172.21.15.117:0/3626786346 >> 172.21.15.117:0/3626786346 conn(0x55a4d8f7ab30 msgr2=0x55a4d8f88650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:21.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.058+0000 7f20c5f65740 1 -- 172.21.15.117:0/3626786346 shutdown_connections 2024-03-19T03:33:21.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.058+0000 7f20c5f65740 1 -- 172.21.15.117:0/3626786346 wait complete. 2024-03-19T03:33:21.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.058+0000 7f20c5f65740 1 -- start start 2024-03-19T03:33:21.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.058+0000 7f20c5f65740 1 Processor -- start 2024-03-19T03:33:21.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.059+0000 7f20c5f65740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7f5c0 0x55a4d8f8c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.059+0000 7f20c5f65740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a4d8f7ffb0 0x55a4d8f8c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.059+0000 7f20c5f65740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a4d8f85120 0x55a4d8f8f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.059+0000 7f20c5f65740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a4d8f97b10 con 0x55a4d8f7ffb0 2024-03-19T03:33:21.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.059+0000 7f20c5f65740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55a4d8f97440 con 0x55a4d8f7f5c0 2024-03-19T03:33:21.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.059+0000 7f20c5f65740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x55a4d8f981e0 con 0x55a4d8f85120 2024-03-19T03:33:21.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.059+0000 7f20c5f65740 10 client.0 _ll_register_callbacks cb 0x7f20a4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:33:21.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.059+0000 7f20bbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7f5c0 0x55a4d8f8c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.059+0000 7f20bbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7f5c0 0x55a4d8f8c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52728/0 (socket says 172.21.15.117:52728) 2024-03-19T03:33:21.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.059+0000 7f20bbe00640 1 -- 172.21.15.117:0/456640929 learned_addr learned my addr 172.21.15.117:0/456640929 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:21.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.059+0000 7f20b3e00640 1 --2- 172.21.15.117:0/456640929 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a4d8f7ffb0 0x55a4d8f8c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.060+0000 7f20bbe00640 1 -- 172.21.15.117:0/456640929 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a4d8f85120 msgr2=0x55a4d8f8f6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T03:33:21.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.060+0000 7f20bbe00640 1 --2- 172.21.15.117:0/456640929 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a4d8f85120 0x55a4d8f8f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.060+0000 7f20bbe00640 1 -- 172.21.15.117:0/456640929 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a4d8f7ffb0 msgr2=0x55a4d8f8c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.060+0000 7f20bbe00640 1 --2- 172.21.15.117:0/456640929 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a4d8f7ffb0 0x55a4d8f8c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.060+0000 7f20bbe00640 1 -- 172.21.15.117:0/456640929 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4d8f8fc90 con 0x55a4d8f7f5c0 2024-03-19T03:33:21.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.060+0000 7f20bbe00640 1 --2- 172.21.15.117:0/456640929 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7f5c0 0x55a4d8f8c190 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f20ac009380 tx=0x7f20ac011e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1145393d6a84bbc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:21.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.060+0000 7f20aa000640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-19T03:33:21.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.060+0000 7f20aa000640 1 -- 172.21.15.117:0/456640929 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ac03b070 con 0x55a4d8f7f5c0 2024-03-19T03:33:21.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.060+0000 7f20aa000640 1 -- 172.21.15.117:0/456640929 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f20ac019d60 con 0x55a4d8f7f5c0 2024-03-19T03:33:21.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.060+0000 7f20c5f65740 1 -- 172.21.15.117:0/456640929 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20a40a3e90 con 0x55a4d8f7f5c0 2024-03-19T03:33:21.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.060+0000 7f20aa000640 1 -- 172.21.15.117:0/456640929 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f20ac047050 con 0x55a4d8f7f5c0 2024-03-19T03:33:21.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.060+0000 7f20c5f65740 10 client.25382 fetch_fsmap learned FSMap version 28 2024-03-19T03:33:21.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.060+0000 7f20c5f65740 1 -- 172.21.15.117:0/456640929 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a4d8f90f90 con 0x55a4d8f7f5c0 2024-03-19T03:33:21.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.061+0000 7f20aa000640 1 -- 172.21.15.117:0/456640929 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20a40a3e90 con 0x55a4d8f7f5c0 2024-03-19T03:33:21.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.061+0000 7f20aa000640 1 -- 172.21.15.117:0/456640929 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f20ac004310 con 0x55a4d8f7f5c0 2024-03-19T03:33:21.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.061+0000 7f20c5f65740 10 client.25382 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T03:33:21.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.061+0000 7f20c5f65740 10 client.25382 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-19T03:33:21.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.061+0000 7f20c5f65740 20 client.25382 populate_metadata read hostname 'smithi117' 2024-03-19T03:33:21.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.061+0000 7f20c5f65740 1 --2- 172.21.15.117:0/456640929 >> [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] conn(0x55a4d9000ca0 0x55a4d8ffd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.061+0000 7f20c5f65740 4 client.25382 mds_command: new command op to 24467 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T03:33:21.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.061+0000 7f20c5f65740 1 -- 172.21.15.117:0/456640929 --> [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] -- command(tid 0: {"prefix": "status"}) -- 0x55a4d8eeb590 con 0x55a4d9000ca0 2024-03-19T03:33:21.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.061+0000 7f20b3e00640 1 --2- 172.21.15.117:0/456640929 >> [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] conn(0x55a4d9000ca0 0x55a4d8ffd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.062+0000 7f20b3e00640 1 --2- 172.21.15.117:0/456640929 >> [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] conn(0x55a4d9000ca0 0x55a4d8ffd0e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4a23d9af46c2cc3b server_cookie=dc27bc08f8b867ea in_seq=0 out_seq=0 2024-03-19T03:33:21.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.062+0000 7f20aa000640 10 client.25382 ms_handle_connect on v2:172.21.15.119:6832/1291833214 2024-03-19T03:33:21.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.062+0000 7f20aa000640 1 -- 172.21.15.117:0/456640929 <== mds.4 v2:172.21.15.119:6832/1291833214 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55a4d8eeb590 con 0x55a4d9000ca0 2024-03-19T03:33:21.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.062+0000 7f20aa000640 10 client.25382 handle_command_reply: tid=0 2024-03-19T03:33:21.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.062+0000 7f20c5f65740 1 client.25382 shutdown 2024-03-19T03:33:21.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.062+0000 7f20c5f65740 1 -- 172.21.15.117:0/456640929 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7f5c0 msgr2=0x55a4d8f8c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.066 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.062+0000 7f20c5f65740 1 --2- 172.21.15.117:0/456640929 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7f5c0 0x55a4d8f8c190 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f20ac009380 tx=0x7f20ac011e40 comp rx=0 tx=0).stop 2024-03-19T03:33:21.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.062+0000 7f20c5f65740 1 -- 172.21.15.117:0/456640929 shutdown_connections 2024-03-19T03:33:21.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.062+0000 7f20c5f65740 1 --2- 172.21.15.117:0/456640929 >> [v2:172.21.15.119:6832/1291833214,v1:172.21.15.119:6833/1291833214] conn(0x55a4d9000ca0 0x55a4d8ffd0e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.062+0000 7f20c5f65740 1 --2- 172.21.15.117:0/456640929 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a4d8f85120 0x55a4d8f8f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.062+0000 7f20c5f65740 1 --2- 172.21.15.117:0/456640929 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a4d8f7ffb0 0x55a4d8f8c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.062+0000 7f20c5f65740 1 --2- 172.21.15.117:0/456640929 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a4d8f7f5c0 0x55a4d8f8c190 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.062+0000 7f20aa000640 0 client.25382 ms_handle_reset on v2:172.21.15.119:6832/1291833214 2024-03-19T03:33:21.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.062+0000 7f20c5f65740 1 -- 172.21.15.117:0/456640929 >> 172.21.15.117:0/456640929 conn(0x55a4d8f7ab30 msgr2=0x55a4d8f7e620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:21.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.062+0000 7f20c5f65740 1 -- 172.21.15.117:0/456640929 shutdown_connections 2024-03-19T03:33:21.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.062+0000 7f20c5f65740 1 -- 172.21.15.117:0/456640929 wait complete. 2024-03-19T03:33:21.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.063+0000 7f20c5f65740 20 client.25382 trim_cache size 0 max 16384 2024-03-19T03:33:21.067 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-19T03:33:21.068 INFO:teuthology.orchestra.run.smithi117.stdout: "cluster_fsid": "a3a82456-e59f-11ee-95c9-87774f69a715", 2024-03-19T03:33:21.068 INFO:teuthology.orchestra.run.smithi117.stdout: "whoami": 4, 2024-03-19T03:33:21.068 INFO:teuthology.orchestra.run.smithi117.stdout: "id": 24467, 2024-03-19T03:33:21.068 INFO:teuthology.orchestra.run.smithi117.stdout: "want_state": "up:active", 2024-03-19T03:33:21.068 INFO:teuthology.orchestra.run.smithi117.stdout: "state": "up:active", 2024-03-19T03:33:21.068 INFO:teuthology.orchestra.run.smithi117.stdout: "fs_name": "cephfs", 2024-03-19T03:33:21.068 INFO:teuthology.orchestra.run.smithi117.stdout: "rank_uptime": 132.07098083, 2024-03-19T03:33:21.068 INFO:teuthology.orchestra.run.smithi117.stdout: "mdsmap_epoch": 28, 2024-03-19T03:33:21.068 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch": 93, 2024-03-19T03:33:21.068 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch_barrier": 89, 2024-03-19T03:33:21.068 INFO:teuthology.orchestra.run.smithi117.stdout: "uptime": 206.32306655599999 2024-03-19T03:33:21.068 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-19T03:33:21.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.063+0000 7f20c1c00640 1 -- 172.21.15.117:0/3473288920 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f20a00a2800 msgr2=0x7f20a00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.063+0000 7f20c1c00640 1 --2- 172.21.15.117:0/3473288920 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f20a00a2800 0x7f20a00a4cc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f20b4005e30 tx=0x7f20b4032590 comp rx=0 tx=0).stop 2024-03-19T03:33:21.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.063+0000 7f20c1c00640 1 -- 172.21.15.117:0/3473288920 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20bc102820 msgr2=0x7f20bc19a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.063+0000 7f20c1c00640 1 --2- 172.21.15.117:0/3473288920 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20bc102820 0x7f20bc19a010 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f20a400ea60 tx=0x7f20a400ef20 comp rx=0 tx=0).stop 2024-03-19T03:33:21.069 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.063+0000 7f20c1c00640 1 -- 172.21.15.117:0/3473288920 shutdown_connections 2024-03-19T03:33:21.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.063+0000 7f20c1c00640 1 --2- 172.21.15.117:0/3473288920 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20bc1031e0 0x7f20bc1a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.063+0000 7f20c1c00640 1 --2- 172.21.15.117:0/3473288920 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f20a00a2800 0x7f20a00a4cc0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.063+0000 7f20c1c00640 1 --2- 172.21.15.117:0/3473288920 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20bc102820 0x7f20bc19a010 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.063+0000 7f20c1c00640 1 --2- 172.21.15.117:0/3473288920 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20bc101630 0x7f20bc199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.063+0000 7f20c1c00640 1 -- 172.21.15.117:0/3473288920 >> 172.21.15.117:0/3473288920 conn(0x7f20bc0fcba0 msgr2=0x7f20bc073a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:21.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.064+0000 7f20c1c00640 1 -- 172.21.15.117:0/3473288920 shutdown_connections 2024-03-19T03:33:21.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.064+0000 7f20c1c00640 1 -- 172.21.15.117:0/3473288920 wait complete. 2024-03-19T03:33:21.088 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-19T03:33:21.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.282+0000 7f881d400640 1 -- 172.21.15.117:0/2951361950 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8818102820 msgr2=0x7f8818102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.282+0000 7f881d400640 1 --2- 172.21.15.117:0/2951361950 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8818102820 0x7f8818102ca0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8800009ca0 tx=0x7f880002c100 comp rx=0 tx=0).stop 2024-03-19T03:33:21.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.282+0000 7f881d400640 1 -- 172.21.15.117:0/2951361950 shutdown_connections 2024-03-19T03:33:21.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.282+0000 7f881d400640 1 --2- 172.21.15.117:0/2951361950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88181031e0 0x7f8818109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.282+0000 7f881d400640 1 --2- 172.21.15.117:0/2951361950 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8818102820 0x7f8818102ca0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.282+0000 7f881d400640 1 --2- 172.21.15.117:0/2951361950 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8818101630 0x7f8818101a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.282+0000 7f881d400640 1 -- 172.21.15.117:0/2951361950 >> 172.21.15.117:0/2951361950 conn(0x7f88180fcb80 msgr2=0x7f88180fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:21.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.283+0000 7f881d400640 1 -- 172.21.15.117:0/2951361950 shutdown_connections 2024-03-19T03:33:21.284 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.283+0000 7f881d400640 1 -- 172.21.15.117:0/2951361950 wait complete. 2024-03-19T03:33:21.285 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.283+0000 7f881d400640 1 Processor -- start 2024-03-19T03:33:21.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.284+0000 7f881d400640 1 -- start start 2024-03-19T03:33:21.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.284+0000 7f881d400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8818101630 0x7f88180755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.284+0000 7f881d400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8818102820 0x7f8818073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.284+0000 7f881d400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88181031e0 0x7f8818074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.284+0000 7f881d400640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f881810d2c0 con 0x7f8818101630 2024-03-19T03:33:21.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.284+0000 7f881d400640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f881810d140 con 0x7f88181031e0 2024-03-19T03:33:21.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.284+0000 7f881d400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f881810d440 con 0x7f8818102820 2024-03-19T03:33:21.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.284+0000 7f881ca00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88181031e0 0x7f8818074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.284+0000 7f8817e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8818101630 0x7f88180755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.284+0000 7f881ca00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88181031e0 0x7f8818074150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52756/0 (socket says 172.21.15.117:52756) 2024-03-19T03:33:21.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.284+0000 7f881ca00640 1 -- 172.21.15.117:0/1883548416 learned_addr learned my addr 172.21.15.117:0/1883548416 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:21.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.285+0000 7f8817400640 1 --2- 172.21.15.117:0/1883548416 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8818102820 0x7f8818073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.285+0000 7f881ca00640 1 -- 172.21.15.117:0/1883548416 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8818102820 msgr2=0x7f8818073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.286 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.285+0000 7f881ca00640 1 --2- 172.21.15.117:0/1883548416 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8818102820 0x7f8818073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.285+0000 7f881ca00640 1 -- 172.21.15.117:0/1883548416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8818101630 msgr2=0x7f88180755d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.285+0000 7f881ca00640 1 --2- 172.21.15.117:0/1883548416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8818101630 0x7f88180755d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.285+0000 7f881ca00640 1 -- 172.21.15.117:0/1883548416 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88180749e0 con 0x7f88181031e0 2024-03-19T03:33:21.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.285+0000 7f8817400640 1 --2- 172.21.15.117:0/1883548416 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8818102820 0x7f8818073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:33:21.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.285+0000 7f881ca00640 1 --2- 172.21.15.117:0/1883548416 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88181031e0 0x7f8818074150 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f881000e970 tx=0x7f881000ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:21.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.285+0000 7f8814c00640 1 -- 172.21.15.117:0/1883548416 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f881000ccc0 con 0x7f88181031e0 2024-03-19T03:33:21.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.285+0000 7f8814c00640 1 -- 172.21.15.117:0/1883548416 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f881000ce60 con 0x7f88181031e0 2024-03-19T03:33:21.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.285+0000 7f8814c00640 1 -- 172.21.15.117:0/1883548416 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8810010490 con 0x7f88181031e0 2024-03-19T03:33:21.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.285+0000 7f881d400640 1 -- 172.21.15.117:0/1883548416 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88181a1a70 con 0x7f88181031e0 2024-03-19T03:33:21.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.286+0000 7f881d400640 1 -- 172.21.15.117:0/1883548416 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88181a1f40 con 0x7f88181031e0 2024-03-19T03:33:21.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.289+0000 7f8814c00640 1 -- 172.21.15.117:0/1883548416 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f8810005020 con 0x7f88181031e0 2024-03-19T03:33:21.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.290+0000 7f8821656740 1 Processor -- start 2024-03-19T03:33:21.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.290+0000 7f8821656740 1 -- start start 2024-03-19T03:33:21.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.290+0000 7f8821656740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0bd5c0 0x55bd8d0bd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.290+0000 7f8821656740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd8d0bdfb0 0x55bd8d0c2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.290+0000 7f8821656740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55bd8d0c3120 0x55bd8d0c55b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.290+0000 7f8821656740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bd8d021d00 con 0x55bd8d0bdfb0 2024-03-19T03:33:21.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.290+0000 7f8821656740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55bd8cfcab00 con 0x55bd8d0bd5c0 2024-03-19T03:33:21.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.290+0000 7f8821656740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x55bd8d01dda0 con 0x55bd8d0c3120 2024-03-19T03:33:21.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.291+0000 7f8817400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0bd5c0 0x55bd8d0bd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.291+0000 7f8817400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0bd5c0 0x55bd8d0bd9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52764/0 (socket says 172.21.15.117:52764) 2024-03-19T03:33:21.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.291+0000 7f8817400640 1 -- 172.21.15.117:0/3358647901 learned_addr learned my addr 172.21.15.117:0/3358647901 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:21.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.291+0000 7f8817e00640 1 --2- 172.21.15.117:0/3358647901 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd8d0bdfb0 0x55bd8d0c2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.291+0000 7f8817400640 1 --2- 172.21.15.117:0/3358647901 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55bd8d0c3120 0x55bd8d0c55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.291+0000 7f8817e00640 1 -- 172.21.15.117:0/3358647901 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55bd8d0c3120 msgr2=0x55bd8d0c55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.291+0000 7f8817e00640 1 --2- 172.21.15.117:0/3358647901 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55bd8d0c3120 0x55bd8d0c55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.291+0000 7f8817e00640 1 -- 172.21.15.117:0/3358647901 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0bd5c0 msgr2=0x55bd8d0bd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.291+0000 7f8817e00640 1 --2- 172.21.15.117:0/3358647901 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0bd5c0 0x55bd8d0bd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.291+0000 7f8817e00640 1 -- 172.21.15.117:0/3358647901 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd8cd362d0 con 0x55bd8d0bdfb0 2024-03-19T03:33:21.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.291+0000 7f8817400640 1 --2- 172.21.15.117:0/3358647901 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0bd5c0 0x55bd8d0bd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:33:21.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.291+0000 7f8814c00640 1 --2- 172.21.15.117:0/1883548416 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f87f80a2880 0x7f87f80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.292+0000 7f8814c00640 1 -- 172.21.15.117:0/1883548416 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f88100cf3c0 con 0x7f88181031e0 2024-03-19T03:33:21.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.292+0000 7f8817e00640 1 --2- 172.21.15.117:0/3358647901 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd8d0bdfb0 0x55bd8d0c2ab0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8808009bd0 tx=0x7f8808011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b527331eec806af8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:21.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.292+0000 7f8817400640 1 --2- 172.21.15.117:0/1883548416 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f87f80a2880 0x7f87f80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.292+0000 7f880d200640 1 -- 172.21.15.117:0/3358647901 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8808004230 con 0x55bd8d0bdfb0 2024-03-19T03:33:21.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.292+0000 7f880d200640 1 -- 172.21.15.117:0/3358647901 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8808009e40 con 0x55bd8d0bdfb0 2024-03-19T03:33:21.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.292+0000 7f880d200640 1 -- 172.21.15.117:0/3358647901 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88080197e0 con 0x55bd8d0bdfb0 2024-03-19T03:33:21.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.293+0000 7f8821656740 1 -- 172.21.15.117:0/3358647901 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd8d0bdfb0 msgr2=0x55bd8d0c2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.293+0000 7f8821656740 1 --2- 172.21.15.117:0/3358647901 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd8d0bdfb0 0x55bd8d0c2ab0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8808009bd0 tx=0x7f8808011810 comp rx=0 tx=0).stop 2024-03-19T03:33:21.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.293+0000 7f8817400640 1 --2- 172.21.15.117:0/1883548416 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f87f80a2880 0x7f87f80a4d40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x55bd8d0be810 tx=0x7f880002e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:21.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.293+0000 7f8821656740 1 -- 172.21.15.117:0/3358647901 shutdown_connections 2024-03-19T03:33:21.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.293+0000 7f8821656740 1 --2- 172.21.15.117:0/3358647901 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55bd8d0c3120 0x55bd8d0c55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.293+0000 7f8821656740 1 --2- 172.21.15.117:0/3358647901 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd8d0bdfb0 0x55bd8d0c2ab0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.297 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.293+0000 7f8821656740 1 --2- 172.21.15.117:0/3358647901 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0bd5c0 0x55bd8d0bd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.293+0000 7f8821656740 1 -- 172.21.15.117:0/3358647901 >> 172.21.15.117:0/3358647901 conn(0x55bd8d0b8b30 msgr2=0x55bd8d0b8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:21.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.293+0000 7f8821656740 1 -- 172.21.15.117:0/3358647901 shutdown_connections 2024-03-19T03:33:21.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.293+0000 7f8821656740 1 -- 172.21.15.117:0/3358647901 wait complete. 2024-03-19T03:33:21.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.293+0000 7f8821656740 1 -- start start 2024-03-19T03:33:21.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.294+0000 7f8821656740 1 Processor -- start 2024-03-19T03:33:21.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.294+0000 7f8821656740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0bd5c0 0x55bd8d0c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.294+0000 7f8821656740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55bd8d0bdfb0 0x55bd8d0c8020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.294+0000 7f8821656740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd8d0c3120 0x55bd8d0cf060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.294+0000 7f8821656740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bd8d0c60d0 con 0x55bd8d0c3120 2024-03-19T03:33:21.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.294+0000 7f8821656740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55bd8cfdded0 con 0x55bd8d0bd5c0 2024-03-19T03:33:21.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.294+0000 7f8821656740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x55bd8d0c6250 con 0x55bd8d0bdfb0 2024-03-19T03:33:21.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.294+0000 7f8821656740 10 client.0 _ll_register_callbacks cb 0x55bd8cfdbc10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:33:21.298 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.294+0000 7f8817e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0bd5c0 0x55bd8d0c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.294+0000 7f8817e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0bd5c0 0x55bd8d0c7ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52774/0 (socket says 172.21.15.117:52774) 2024-03-19T03:33:21.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.294+0000 7f8817e00640 1 -- 172.21.15.117:0/4001978412 learned_addr learned my addr 172.21.15.117:0/4001978412 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:21.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.294+0000 7f881ca00640 1 --2- 172.21.15.117:0/4001978412 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd8d0c3120 0x55bd8d0cf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.295+0000 7f8817400640 1 --2- 172.21.15.117:0/4001978412 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55bd8d0bdfb0 0x55bd8d0c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.295+0000 7f8817e00640 1 -- 172.21.15.117:0/4001978412 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55bd8d0bdfb0 msgr2=0x55bd8d0c8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.295+0000 7f8817e00640 1 --2- 172.21.15.117:0/4001978412 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55bd8d0bdfb0 0x55bd8d0c8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.295+0000 7f8817e00640 1 -- 172.21.15.117:0/4001978412 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd8d0c3120 msgr2=0x55bd8d0cf060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.295+0000 7f8817e00640 1 --2- 172.21.15.117:0/4001978412 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd8d0c3120 0x55bd8d0cf060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.295+0000 7f8817e00640 1 -- 172.21.15.117:0/4001978412 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd8ce006b0 con 0x55bd8d0bd5c0 2024-03-19T03:33:21.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.295+0000 7f881ca00640 1 --2- 172.21.15.117:0/4001978412 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd8d0c3120 0x55bd8d0cf060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:33:21.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.295+0000 7f8817e00640 1 --2- 172.21.15.117:0/4001978412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0bd5c0 0x55bd8d0c7ae0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8808011d10 tx=0x7f8808003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8fa745561f0d045 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:21.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.295+0000 7f8805600640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-19T03:33:21.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.295+0000 7f8805600640 1 -- 172.21.15.117:0/4001978412 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8808028070 con 0x55bd8d0bd5c0 2024-03-19T03:33:21.299 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.295+0000 7f8805600640 1 -- 172.21.15.117:0/4001978412 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8808005040 con 0x55bd8d0bd5c0 2024-03-19T03:33:21.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.295+0000 7f8821656740 1 -- 172.21.15.117:0/4001978412 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bd8d02f240 con 0x55bd8d0bd5c0 2024-03-19T03:33:21.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.296+0000 7f8805600640 1 -- 172.21.15.117:0/4001978412 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8808034050 con 0x55bd8d0bd5c0 2024-03-19T03:33:21.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.296+0000 7f8821656740 10 client.25394 fetch_fsmap learned FSMap version 28 2024-03-19T03:33:21.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.296+0000 7f8821656740 1 -- 172.21.15.117:0/4001978412 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bd8d0d23d0 con 0x55bd8d0bd5c0 2024-03-19T03:33:21.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.296+0000 7f8805600640 1 -- 172.21.15.117:0/4001978412 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55bd8d02f240 con 0x55bd8d0bd5c0 2024-03-19T03:33:21.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.296+0000 7f8805600640 1 -- 172.21.15.117:0/4001978412 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8808004b10 con 0x55bd8d0bd5c0 2024-03-19T03:33:21.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.296+0000 7f8821656740 10 client.25394 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T03:33:21.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.296+0000 7f8821656740 10 client.25394 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-19T03:33:21.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.296+0000 7f8821656740 20 client.25394 populate_metadata read hostname 'smithi117' 2024-03-19T03:33:21.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.296+0000 7f8821656740 1 --2- 172.21.15.117:0/4001978412 >> [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] conn(0x55bd8d0d4860 0x55bd8d0d5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.296+0000 7f8821656740 4 client.25394 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:33:21.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.296+0000 7f8821656740 1 -- 172.21.15.117:0/4001978412 --> [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd8d029590 con 0x55bd8d0d4860 2024-03-19T03:33:21.300 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.297+0000 7f8817400640 1 --2- 172.21.15.117:0/4001978412 >> [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] conn(0x55bd8d0d4860 0x55bd8d0d5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.297+0000 7f8817400640 1 --2- 172.21.15.117:0/4001978412 >> [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] conn(0x55bd8d0d4860 0x55bd8d0d5cd0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9d25a39aaf9c088b server_cookie=c4ac7e531e845738 in_seq=0 out_seq=0 2024-03-19T03:33:21.301 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.297+0000 7f8805600640 10 client.25394 ms_handle_connect on v2:172.21.15.119:6836/2054992738 2024-03-19T03:33:21.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.298+0000 7f8805600640 1 -- 172.21.15.117:0/4001978412 <== mds.3 v2:172.21.15.119:6836/2054992738 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bd8d029590 con 0x55bd8d0d4860 2024-03-19T03:33:21.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.298+0000 7f8805600640 10 client.25394 handle_command_reply: tid=0 2024-03-19T03:33:21.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.298+0000 7f8821656740 1 client.25394 shutdown 2024-03-19T03:33:21.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.298+0000 7f8821656740 1 -- 172.21.15.117:0/4001978412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0bd5c0 msgr2=0x55bd8d0c7ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.298+0000 7f8821656740 1 --2- 172.21.15.117:0/4001978412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0bd5c0 0x55bd8d0c7ae0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8808011d10 tx=0x7f8808003820 comp rx=0 tx=0).stop 2024-03-19T03:33:21.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.298+0000 7f8821656740 1 -- 172.21.15.117:0/4001978412 shutdown_connections 2024-03-19T03:33:21.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.298+0000 7f8821656740 1 --2- 172.21.15.117:0/4001978412 >> [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] conn(0x55bd8d0d4860 0x55bd8d0d5cd0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.298+0000 7f8821656740 1 --2- 172.21.15.117:0/4001978412 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd8d0c3120 0x55bd8d0cf060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.298+0000 7f8821656740 1 --2- 172.21.15.117:0/4001978412 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55bd8d0bdfb0 0x55bd8d0c8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.298+0000 7f8805600640 0 client.25394 ms_handle_reset on v2:172.21.15.119:6836/2054992738 2024-03-19T03:33:21.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.298+0000 7f8821656740 1 --2- 172.21.15.117:0/4001978412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0bd5c0 0x55bd8d0c7ae0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.302 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.298+0000 7f8821656740 1 -- 172.21.15.117:0/4001978412 >> 172.21.15.117:0/4001978412 conn(0x55bd8d0b8b30 msgr2=0x55bd8d0b7d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:21.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.298+0000 7f8821656740 1 -- 172.21.15.117:0/4001978412 shutdown_connections 2024-03-19T03:33:21.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.298+0000 7f8821656740 1 -- 172.21.15.117:0/4001978412 wait complete. 2024-03-19T03:33:21.303 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.298+0000 7f8821656740 20 client.25394 trim_cache size 0 max 16384 2024-03-19T03:33:21.317 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.318+0000 7f8821656740 1 Processor -- start 2024-03-19T03:33:21.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.318+0000 7f8821656740 1 -- start start 2024-03-19T03:33:21.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.318+0000 7f8821656740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd8d0bd5c0 0x55bd8d0d37e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.318+0000 7f8821656740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55bd8d0bdfb0 0x55bd8d0cef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.318+0000 7f8821656740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0c3120 0x55bd8d0b8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.318+0000 7f8821656740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bd8d0d4010 con 0x55bd8d0bd5c0 2024-03-19T03:33:21.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.318+0000 7f8821656740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55bd8d0d46e0 con 0x55bd8d0c3120 2024-03-19T03:33:21.318 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.318+0000 7f8821656740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x55bd8d0d4db0 con 0x55bd8d0bdfb0 2024-03-19T03:33:21.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.318+0000 7f881ca00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0c3120 0x55bd8d0b8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.318+0000 7f881ca00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0c3120 0x55bd8d0b8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52778/0 (socket says 172.21.15.117:52778) 2024-03-19T03:33:21.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.318+0000 7f881ca00640 1 -- 172.21.15.117:0/2421481469 learned_addr learned my addr 172.21.15.117:0/2421481469 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:21.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.318+0000 7f8817e00640 1 --2- 172.21.15.117:0/2421481469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd8d0bd5c0 0x55bd8d0d37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.318+0000 7f8817400640 1 --2- 172.21.15.117:0/2421481469 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55bd8d0bdfb0 0x55bd8d0cef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.320 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.319+0000 7f881ca00640 1 -- 172.21.15.117:0/2421481469 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55bd8d0bdfb0 msgr2=0x55bd8d0cef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.319+0000 7f881ca00640 1 --2- 172.21.15.117:0/2421481469 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55bd8d0bdfb0 0x55bd8d0cef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.319+0000 7f881ca00640 1 -- 172.21.15.117:0/2421481469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd8d0bd5c0 msgr2=0x55bd8d0d37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.319+0000 7f881ca00640 1 --2- 172.21.15.117:0/2421481469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd8d0bd5c0 0x55bd8d0d37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.319+0000 7f881ca00640 1 -- 172.21.15.117:0/2421481469 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd8d0be390 con 0x55bd8d0c3120 2024-03-19T03:33:21.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.319+0000 7f8817e00640 1 --2- 172.21.15.117:0/2421481469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd8d0bd5c0 0x55bd8d0d37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:33:21.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.319+0000 7f8817400640 1 --2- 172.21.15.117:0/2421481469 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55bd8d0bdfb0 0x55bd8d0cef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:33:21.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.319+0000 7f881ca00640 1 --2- 172.21.15.117:0/2421481469 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0c3120 0x55bd8d0b8f10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f881000e940 tx=0x7f881009c010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c914bb63bbeb59f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:21.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.319+0000 7f8807e00640 1 -- 172.21.15.117:0/2421481469 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88100a40b0 con 0x55bd8d0c3120 2024-03-19T03:33:21.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.319+0000 7f8807e00640 1 -- 172.21.15.117:0/2421481469 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88100cf160 con 0x55bd8d0c3120 2024-03-19T03:33:21.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.319+0000 7f8807e00640 1 -- 172.21.15.117:0/2421481469 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8810005310 con 0x55bd8d0c3120 2024-03-19T03:33:21.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.320+0000 7f8821656740 1 -- 172.21.15.117:0/2421481469 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0c3120 msgr2=0x55bd8d0b8f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.320+0000 7f8821656740 1 --2- 172.21.15.117:0/2421481469 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0c3120 0x55bd8d0b8f10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f881000e940 tx=0x7f881009c010 comp rx=0 tx=0).stop 2024-03-19T03:33:21.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.320+0000 7f8821656740 1 -- 172.21.15.117:0/2421481469 shutdown_connections 2024-03-19T03:33:21.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.320+0000 7f8821656740 1 --2- 172.21.15.117:0/2421481469 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0c3120 0x55bd8d0b8f10 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.321 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.320+0000 7f8821656740 1 --2- 172.21.15.117:0/2421481469 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55bd8d0bdfb0 0x55bd8d0cef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.320+0000 7f8821656740 1 --2- 172.21.15.117:0/2421481469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd8d0bd5c0 0x55bd8d0d37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.320+0000 7f8821656740 1 -- 172.21.15.117:0/2421481469 >> 172.21.15.117:0/2421481469 conn(0x55bd8d0b8b30 msgr2=0x55bd8d0b3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:21.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.320+0000 7f8821656740 1 -- 172.21.15.117:0/2421481469 shutdown_connections 2024-03-19T03:33:21.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.320+0000 7f8821656740 1 -- 172.21.15.117:0/2421481469 wait complete. 2024-03-19T03:33:21.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.320+0000 7f8821656740 1 -- start start 2024-03-19T03:33:21.322 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.321+0000 7f8821656740 1 Processor -- start 2024-03-19T03:33:21.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.321+0000 7f8821656740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd8d0bd5c0 0x55bd8d0ca850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.321+0000 7f8821656740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55bd8d0bdfb0 0x55bd8d0cad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.321+0000 7f8821656740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0c3120 0x55bd8d0cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.321+0000 7f8821656740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bd8d0d5b50 con 0x55bd8d0bd5c0 2024-03-19T03:33:21.325 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.321+0000 7f8821656740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x55bd8d0d5480 con 0x55bd8d0c3120 2024-03-19T03:33:21.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.321+0000 7f8821656740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x55bd8d0d6220 con 0x55bd8d0bdfb0 2024-03-19T03:33:21.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.321+0000 7f8821656740 10 client.0 _ll_register_callbacks cb 0x7f8800037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:33:21.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.321+0000 7f881ca00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0c3120 0x55bd8d0cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.321+0000 7f8817e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd8d0bd5c0 0x55bd8d0ca850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.321+0000 7f881ca00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0c3120 0x55bd8d0cddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52782/0 (socket says 172.21.15.117:52782) 2024-03-19T03:33:21.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.321+0000 7f881ca00640 1 -- 172.21.15.117:0/2708811872 learned_addr learned my addr 172.21.15.117:0/2708811872 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:21.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.321+0000 7f8817400640 1 --2- 172.21.15.117:0/2708811872 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55bd8d0bdfb0 0x55bd8d0cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.322+0000 7f881ca00640 1 -- 172.21.15.117:0/2708811872 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55bd8d0bdfb0 msgr2=0x55bd8d0cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.322+0000 7f881ca00640 1 --2- 172.21.15.117:0/2708811872 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55bd8d0bdfb0 0x55bd8d0cad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.322+0000 7f881ca00640 1 -- 172.21.15.117:0/2708811872 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd8d0bd5c0 msgr2=0x55bd8d0ca850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.322+0000 7f881ca00640 1 --2- 172.21.15.117:0/2708811872 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd8d0bd5c0 0x55bd8d0ca850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.322+0000 7f881ca00640 1 -- 172.21.15.117:0/2708811872 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd8d0ce3a0 con 0x55bd8d0c3120 2024-03-19T03:33:21.326 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.322+0000 7f881ca00640 1 --2- 172.21.15.117:0/2708811872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0c3120 0x55bd8d0cddb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f881000c6a0 tx=0x7f881009fb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e009287df261a4b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:21.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.322+0000 7f8805600640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-19T03:33:21.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.322+0000 7f8805600640 1 -- 172.21.15.117:0/2708811872 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88100999b0 con 0x55bd8d0c3120 2024-03-19T03:33:21.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.322+0000 7f8805600640 1 -- 172.21.15.117:0/2708811872 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8810005310 con 0x55bd8d0c3120 2024-03-19T03:33:21.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.322+0000 7f8821656740 1 -- 172.21.15.117:0/2708811872 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88100cf160 con 0x55bd8d0c3120 2024-03-19T03:33:21.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.322+0000 7f8805600640 1 -- 172.21.15.117:0/2708811872 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8810082050 con 0x55bd8d0c3120 2024-03-19T03:33:21.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.322+0000 7f8821656740 10 client.25400 fetch_fsmap learned FSMap version 28 2024-03-19T03:33:21.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.322+0000 7f8821656740 1 -- 172.21.15.117:0/2708811872 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bd8d13a770 con 0x55bd8d0c3120 2024-03-19T03:33:21.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.323+0000 7f8805600640 1 -- 172.21.15.117:0/2708811872 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88100cf160 con 0x55bd8d0c3120 2024-03-19T03:33:21.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.323+0000 7f8805600640 1 -- 172.21.15.117:0/2708811872 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f881009ad60 con 0x55bd8d0c3120 2024-03-19T03:33:21.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.323+0000 7f8821656740 10 client.25400 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T03:33:21.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.323+0000 7f8821656740 10 client.25400 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-19T03:33:21.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.323+0000 7f8821656740 20 client.25400 populate_metadata read hostname 'smithi117' 2024-03-19T03:33:21.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.323+0000 7f8821656740 1 --2- 172.21.15.117:0/2708811872 >> [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] conn(0x55bd8d13ee80 0x55bd8d13db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.327 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.323+0000 7f8821656740 4 client.25400 mds_command: new command op to 24545 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T03:33:21.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.323+0000 7f8821656740 1 -- 172.21.15.117:0/2708811872 --> [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] -- command(tid 0: {"prefix": "status"}) -- 0x55bd8cfa78c0 con 0x55bd8d13ee80 2024-03-19T03:33:21.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.323+0000 7f8817e00640 1 --2- 172.21.15.117:0/2708811872 >> [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] conn(0x55bd8d13ee80 0x55bd8d13db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.324+0000 7f8817e00640 1 --2- 172.21.15.117:0/2708811872 >> [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] conn(0x55bd8d13ee80 0x55bd8d13db90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e5d7a7764b0b8801 server_cookie=c09dcf86eed32671 in_seq=0 out_seq=0 2024-03-19T03:33:21.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.324+0000 7f8805600640 10 client.25400 ms_handle_connect on v2:172.21.15.119:6836/2054992738 2024-03-19T03:33:21.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.324+0000 7f8805600640 1 -- 172.21.15.117:0/2708811872 <== mds.3 v2:172.21.15.119:6836/2054992738 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55bd8cfa78c0 con 0x55bd8d13ee80 2024-03-19T03:33:21.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.324+0000 7f8805600640 10 client.25400 handle_command_reply: tid=0 2024-03-19T03:33:21.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.324+0000 7f8821656740 1 client.25400 shutdown 2024-03-19T03:33:21.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.324+0000 7f8821656740 1 -- 172.21.15.117:0/2708811872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0c3120 msgr2=0x55bd8d0cddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.324+0000 7f8821656740 1 --2- 172.21.15.117:0/2708811872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0c3120 0x55bd8d0cddb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f881000c6a0 tx=0x7f881009fb20 comp rx=0 tx=0).stop 2024-03-19T03:33:21.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.324+0000 7f8821656740 1 -- 172.21.15.117:0/2708811872 shutdown_connections 2024-03-19T03:33:21.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.324+0000 7f8821656740 1 --2- 172.21.15.117:0/2708811872 >> [v2:172.21.15.119:6836/2054992738,v1:172.21.15.119:6837/2054992738] conn(0x55bd8d13ee80 0x55bd8d13db90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.324+0000 7f8821656740 1 --2- 172.21.15.117:0/2708811872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bd8d0c3120 0x55bd8d0cddb0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.328 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.324+0000 7f8821656740 1 --2- 172.21.15.117:0/2708811872 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55bd8d0bdfb0 0x55bd8d0cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.324+0000 7f8805600640 0 client.25400 ms_handle_reset on v2:172.21.15.119:6836/2054992738 2024-03-19T03:33:21.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.324+0000 7f8821656740 1 --2- 172.21.15.117:0/2708811872 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bd8d0bd5c0 0x55bd8d0ca850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.324+0000 7f8821656740 1 -- 172.21.15.117:0/2708811872 >> 172.21.15.117:0/2708811872 conn(0x55bd8d0b8b30 msgr2=0x55bd8d0d37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:21.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.324+0000 7f8821656740 1 -- 172.21.15.117:0/2708811872 shutdown_connections 2024-03-19T03:33:21.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.324+0000 7f8821656740 1 -- 172.21.15.117:0/2708811872 wait complete. 2024-03-19T03:33:21.329 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.324+0000 7f8821656740 20 client.25400 trim_cache size 0 max 16384 2024-03-19T03:33:21.329 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-19T03:33:21.329 INFO:teuthology.orchestra.run.smithi117.stdout: "cluster_fsid": "a3a82456-e59f-11ee-95c9-87774f69a715", 2024-03-19T03:33:21.329 INFO:teuthology.orchestra.run.smithi117.stdout: "whoami": 3, 2024-03-19T03:33:21.330 INFO:teuthology.orchestra.run.smithi117.stdout: "id": 24545, 2024-03-19T03:33:21.330 INFO:teuthology.orchestra.run.smithi117.stdout: "want_state": "up:active", 2024-03-19T03:33:21.330 INFO:teuthology.orchestra.run.smithi117.stdout: "state": "up:active", 2024-03-19T03:33:21.330 INFO:teuthology.orchestra.run.smithi117.stdout: "fs_name": "cephfs", 2024-03-19T03:33:21.330 INFO:teuthology.orchestra.run.smithi117.stdout: "rank_uptime": 132.84222071100001, 2024-03-19T03:33:21.330 INFO:teuthology.orchestra.run.smithi117.stdout: "mdsmap_epoch": 28, 2024-03-19T03:33:21.330 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch": 93, 2024-03-19T03:33:21.330 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch_barrier": 89, 2024-03-19T03:33:21.330 INFO:teuthology.orchestra.run.smithi117.stdout: "uptime": 187.08962809299999 2024-03-19T03:33:21.330 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-19T03:33:21.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.325+0000 7f881d400640 1 -- 172.21.15.117:0/1883548416 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f87f80a2880 msgr2=0x7f87f80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.325+0000 7f881d400640 1 --2- 172.21.15.117:0/1883548416 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f87f80a2880 0x7f87f80a4d40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x55bd8d0be810 tx=0x7f880002e040 comp rx=0 tx=0).stop 2024-03-19T03:33:21.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.325+0000 7f881d400640 1 -- 172.21.15.117:0/1883548416 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88181031e0 msgr2=0x7f8818074150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.325+0000 7f881d400640 1 --2- 172.21.15.117:0/1883548416 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88181031e0 0x7f8818074150 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f881000e970 tx=0x7f881000ed70 comp rx=0 tx=0).stop 2024-03-19T03:33:21.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.325+0000 7f881d400640 1 -- 172.21.15.117:0/1883548416 shutdown_connections 2024-03-19T03:33:21.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.325+0000 7f881d400640 1 --2- 172.21.15.117:0/1883548416 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f87f80a2880 0x7f87f80a4d40 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.325+0000 7f881d400640 1 --2- 172.21.15.117:0/1883548416 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88181031e0 0x7f8818074150 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.325+0000 7f881d400640 1 --2- 172.21.15.117:0/1883548416 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8818102820 0x7f8818073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.325+0000 7f881d400640 1 --2- 172.21.15.117:0/1883548416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8818101630 0x7f88180755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.331 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.325+0000 7f881d400640 1 -- 172.21.15.117:0/1883548416 >> 172.21.15.117:0/1883548416 conn(0x7f88180fcb80 msgr2=0x7f8818105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:21.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.325+0000 7f881d400640 1 -- 172.21.15.117:0/1883548416 shutdown_connections 2024-03-19T03:33:21.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.325+0000 7f881d400640 1 -- 172.21.15.117:0/1883548416 wait complete. 2024-03-19T03:33:21.345 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-03-19T03:33:21.553 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.553+0000 7fa0c1200640 1 -- 172.21.15.117:0/2404949114 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0bc074280 msgr2=0x7fa0bc074700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.553+0000 7fa0c1200640 1 --2- 172.21.15.117:0/2404949114 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0bc074280 0x7fa0bc074700 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8009930 tx=0x7fa0a802bc20 comp rx=0 tx=0).stop 2024-03-19T03:33:21.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.553+0000 7fa0c1200640 1 -- 172.21.15.117:0/2404949114 shutdown_connections 2024-03-19T03:33:21.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.553+0000 7fa0c1200640 1 --2- 172.21.15.117:0/2404949114 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa0bc10dc00 0x7fa0bc10fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.553+0000 7fa0c1200640 1 --2- 172.21.15.117:0/2404949114 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0bc074280 0x7fa0bc074700 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.553+0000 7fa0c1200640 1 --2- 172.21.15.117:0/2404949114 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa0bc075860 0x7fa0bc073cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.553+0000 7fa0c1200640 1 -- 172.21.15.117:0/2404949114 >> 172.21.15.117:0/2404949114 conn(0x7fa0bc0fcb80 msgr2=0x7fa0bc0fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:21.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.554+0000 7fa0c1200640 1 -- 172.21.15.117:0/2404949114 shutdown_connections 2024-03-19T03:33:21.554 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.554+0000 7fa0c1200640 1 -- 172.21.15.117:0/2404949114 wait complete. 2024-03-19T03:33:21.556 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.555+0000 7fa0c1200640 1 Processor -- start 2024-03-19T03:33:21.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.555+0000 7fa0c1200640 1 -- start start 2024-03-19T03:33:21.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.555+0000 7fa0c1200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0bc074280 0x7fa0bc19dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.555+0000 7fa0c1200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa0bc075860 0x7fa0bc19e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.555+0000 7fa0c1200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa0bc10dc00 0x7fa0bc1a5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.555+0000 7fa0c1200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa0bc1138f0 con 0x7fa0bc10dc00 2024-03-19T03:33:21.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.555+0000 7fa0c1200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa0bc113770 con 0x7fa0bc074280 2024-03-19T03:33:21.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.555+0000 7fa0c1200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fa0bc113a70 con 0x7fa0bc075860 2024-03-19T03:33:21.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.555+0000 7fa0bb400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0bc074280 0x7fa0bc19dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.555+0000 7fa0bb400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0bc074280 0x7fa0bc19dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52798/0 (socket says 172.21.15.117:52798) 2024-03-19T03:33:21.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.555+0000 7fa0bb400640 1 -- 172.21.15.117:0/3940570546 learned_addr learned my addr 172.21.15.117:0/3940570546 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:21.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.555+0000 7fa0bbe00640 1 --2- 172.21.15.117:0/3940570546 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa0bc10dc00 0x7fa0bc1a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.556+0000 7fa0baa00640 1 --2- 172.21.15.117:0/3940570546 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa0bc075860 0x7fa0bc19e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.557 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.556+0000 7fa0bb400640 1 -- 172.21.15.117:0/3940570546 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa0bc075860 msgr2=0x7fa0bc19e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.556+0000 7fa0bb400640 1 --2- 172.21.15.117:0/3940570546 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa0bc075860 0x7fa0bc19e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.556+0000 7fa0bb400640 1 -- 172.21.15.117:0/3940570546 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa0bc10dc00 msgr2=0x7fa0bc1a5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.556+0000 7fa0bb400640 1 --2- 172.21.15.117:0/3940570546 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa0bc10dc00 0x7fa0bc1a5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.556+0000 7fa0bb400640 1 -- 172.21.15.117:0/3940570546 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0bc1a5b90 con 0x7fa0bc074280 2024-03-19T03:33:21.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.556+0000 7fa0baa00640 1 --2- 172.21.15.117:0/3940570546 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa0bc075860 0x7fa0bc19e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:33:21.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.556+0000 7fa0bbe00640 1 --2- 172.21.15.117:0/3940570546 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa0bc10dc00 0x7fa0bc1a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:33:21.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.556+0000 7fa0bb400640 1 --2- 172.21.15.117:0/3940570546 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0bc074280 0x7fa0bc19dfc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac00e9f0 tx=0x7fa0ac00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:21.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.556+0000 7fa0b3e00640 1 -- 172.21.15.117:0/3940570546 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac00cd80 con 0x7fa0bc074280 2024-03-19T03:33:21.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.556+0000 7fa0b3e00640 1 -- 172.21.15.117:0/3940570546 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0ac005500 con 0x7fa0bc074280 2024-03-19T03:33:21.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.556+0000 7fa0c1200640 1 -- 172.21.15.117:0/3940570546 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0bc1a5e80 con 0x7fa0bc074280 2024-03-19T03:33:21.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.557+0000 7fa0b3e00640 1 -- 172.21.15.117:0/3940570546 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac010690 con 0x7fa0bc074280 2024-03-19T03:33:21.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.557+0000 7fa0c1200640 1 -- 172.21.15.117:0/3940570546 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0bc1a6290 con 0x7fa0bc074280 2024-03-19T03:33:21.560 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.560+0000 7fa0c5481740 1 Processor -- start 2024-03-19T03:33:21.560 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.560+0000 7fa0c5481740 1 -- start start 2024-03-19T03:33:21.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.560+0000 7fa0c5481740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc1c2b5c0 0x561bc1c2b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.560+0000 7fa0c5481740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bc1c2bfb0 0x561bc1c30ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.560+0000 7fa0c5481740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c31120 0x561bc1c335b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.560+0000 7fa0c5481740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561bc1b8fd00 con 0x561bc1c2b5c0 2024-03-19T03:33:21.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.560+0000 7fa0c5481740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561bc1b38b00 con 0x561bc1c31120 2024-03-19T03:33:21.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.560+0000 7fa0c5481740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x561bc1b8bda0 con 0x561bc1c2bfb0 2024-03-19T03:33:21.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.560+0000 7fa0bbe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc1c2b5c0 0x561bc1c2b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.561+0000 7fa0baa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c31120 0x561bc1c335b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.561+0000 7fa0baa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c31120 0x561bc1c335b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52804/0 (socket says 172.21.15.117:52804) 2024-03-19T03:33:21.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.561+0000 7fa0baa00640 1 -- 172.21.15.117:0/1882817631 learned_addr learned my addr 172.21.15.117:0/1882817631 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:21.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.561+0000 7fa0bbe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc1c2b5c0 0x561bc1c2b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42360/0 (socket says 172.21.15.117:42360) 2024-03-19T03:33:21.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.561+0000 7fa0baa00640 1 -- 172.21.15.117:0/1882817631 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bc1c2bfb0 msgr2=0x561bc1c30ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T03:33:21.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.561+0000 7fa0baa00640 1 --2- 172.21.15.117:0/1882817631 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bc1c2bfb0 0x561bc1c30ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.561+0000 7fa0baa00640 1 -- 172.21.15.117:0/1882817631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc1c2b5c0 msgr2=0x561bc1c2b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.561+0000 7fa0baa00640 1 --2- 172.21.15.117:0/1882817631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc1c2b5c0 0x561bc1c2b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.561+0000 7fa0baa00640 1 -- 172.21.15.117:0/1882817631 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bc18a42d0 con 0x561bc1c31120 2024-03-19T03:33:21.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.561+0000 7fa0b3e00640 1 -- 172.21.15.117:0/3940570546 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fa0ac0046c0 con 0x7fa0bc074280 2024-03-19T03:33:21.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.561+0000 7fa0baa00640 1 --2- 172.21.15.117:0/1882817631 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c31120 0x561bc1c335b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa0a802c8d0 tx=0x7fa0a80095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65ed5210e799db10 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:21.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.561+0000 7fa0b0c00640 1 -- 172.21.15.117:0/1882817631 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8044020 con 0x561bc1c31120 2024-03-19T03:33:21.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.562+0000 7fa0b0c00640 1 -- 172.21.15.117:0/1882817631 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0a802cc30 con 0x561bc1c31120 2024-03-19T03:33:21.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.562+0000 7fa0b0c00640 1 -- 172.21.15.117:0/1882817631 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8004920 con 0x561bc1c31120 2024-03-19T03:33:21.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.562+0000 7fa0c5481740 1 -- 172.21.15.117:0/1882817631 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c31120 msgr2=0x561bc1c335b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.562+0000 7fa0c5481740 1 --2- 172.21.15.117:0/1882817631 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c31120 0x561bc1c335b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa0a802c8d0 tx=0x7fa0a80095e0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.563+0000 7fa0c5481740 1 -- 172.21.15.117:0/1882817631 shutdown_connections 2024-03-19T03:33:21.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.563+0000 7fa0c5481740 1 --2- 172.21.15.117:0/1882817631 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c31120 0x561bc1c335b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.563+0000 7fa0c5481740 1 --2- 172.21.15.117:0/1882817631 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bc1c2bfb0 0x561bc1c30ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.563+0000 7fa0c5481740 1 --2- 172.21.15.117:0/1882817631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc1c2b5c0 0x561bc1c2b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.563+0000 7fa0c5481740 1 -- 172.21.15.117:0/1882817631 >> 172.21.15.117:0/1882817631 conn(0x561bc1c26b30 msgr2=0x561bc1c26f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:21.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.563+0000 7fa0c5481740 1 -- 172.21.15.117:0/1882817631 shutdown_connections 2024-03-19T03:33:21.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.563+0000 7fa0c5481740 1 -- 172.21.15.117:0/1882817631 wait complete. 2024-03-19T03:33:21.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.563+0000 7fa0b3e00640 1 --2- 172.21.15.117:0/3940570546 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa0a00a2710 0x7fa0a00a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.563+0000 7fa0c5481740 1 -- start start 2024-03-19T03:33:21.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.563+0000 7fa0b3e00640 1 -- 172.21.15.117:0/3940570546 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fa0ac0cf3a0 con 0x7fa0bc074280 2024-03-19T03:33:21.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.564+0000 7fa0bbe00640 1 --2- 172.21.15.117:0/3940570546 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa0a00a2710 0x7fa0a00a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.564+0000 7fa0c5481740 1 Processor -- start 2024-03-19T03:33:21.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.564+0000 7fa0c5481740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bc1c2b5c0 0x561bc1c35ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.564+0000 7fa0c5481740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c2bfb0 0x561bc1c36000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.564+0000 7fa0c5481740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc1c31120 0x561bc1c3d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.564+0000 7fa0c5481740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561bc1c34070 con 0x561bc1c31120 2024-03-19T03:33:21.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.564+0000 7fa0c5481740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561bc1b4bed0 con 0x561bc1c2bfb0 2024-03-19T03:33:21.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.564+0000 7fa0c5481740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x561bc1c341f0 con 0x561bc1c2b5c0 2024-03-19T03:33:21.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.564+0000 7fa0c5481740 10 client.0 _ll_register_callbacks cb 0x561bc1b49c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:33:21.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.564+0000 7fa0bbe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc1c31120 0x561bc1c3d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.564+0000 7fa0bb400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bc1c2b5c0 0x561bc1c35ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.565+0000 7fa0baa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c2bfb0 0x561bc1c36000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.565+0000 7fa0bbe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc1c31120 0x561bc1c3d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42366/0 (socket says 172.21.15.117:42366) 2024-03-19T03:33:21.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.565+0000 7fa0bbe00640 1 -- 172.21.15.117:0/473505792 learned_addr learned my addr 172.21.15.117:0/473505792 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:21.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.565+0000 7fa0baa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c2bfb0 0x561bc1c36000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52812/0 (socket says 172.21.15.117:52812) 2024-03-19T03:33:21.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.565+0000 7fa0bb400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bc1c2b5c0 0x561bc1c35ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:45008/0 (socket says 172.21.15.117:45008) 2024-03-19T03:33:21.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.565+0000 7fa0bbe00640 1 --2- 172.21.15.117:0/3940570546 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa0a00a2710 0x7fa0a00a4bd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x561bc1c2c810 tx=0x7fa0b40098a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:21.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.565+0000 7fa0baa00640 1 -- 172.21.15.117:0/473505792 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bc1c2b5c0 msgr2=0x561bc1c35ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.565+0000 7fa0baa00640 1 --2- 172.21.15.117:0/473505792 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bc1c2b5c0 0x561bc1c35ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.565+0000 7fa0baa00640 1 -- 172.21.15.117:0/473505792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc1c31120 msgr2=0x561bc1c3d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.565+0000 7fa0baa00640 1 --2- 172.21.15.117:0/473505792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc1c31120 0x561bc1c3d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.565+0000 7fa0baa00640 1 -- 172.21.15.117:0/473505792 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bc196e6b0 con 0x561bc1c2bfb0 2024-03-19T03:33:21.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.565+0000 7fa0baa00640 1 --2- 172.21.15.117:0/473505792 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c2bfb0 0x561bc1c36000 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa0a802c7a0 tx=0x7fa0a80038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=900fac1f299dfd46 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:21.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.565+0000 7fa0a5600640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-19T03:33:21.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.565+0000 7fa0a5600640 1 -- 172.21.15.117:0/473505792 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a804d070 con 0x561bc1c2bfb0 2024-03-19T03:33:21.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.566+0000 7fa0a5600640 1 -- 172.21.15.117:0/473505792 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fa0a8003ca0 con 0x561bc1c2bfb0 2024-03-19T03:33:21.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.566+0000 7fa0c5481740 1 -- 172.21.15.117:0/473505792 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0a802cc30 con 0x561bc1c2bfb0 2024-03-19T03:33:21.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.566+0000 7fa0a5600640 1 -- 172.21.15.117:0/473505792 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa0a8058050 con 0x561bc1c2bfb0 2024-03-19T03:33:21.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.566+0000 7fa0c5481740 10 client.25412 fetch_fsmap learned FSMap version 28 2024-03-19T03:33:21.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.566+0000 7fa0c5481740 1 -- 172.21.15.117:0/473505792 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561bc1c403b0 con 0x561bc1c2bfb0 2024-03-19T03:33:21.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.566+0000 7fa0a5600640 1 -- 172.21.15.117:0/473505792 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0a802cc30 con 0x561bc1c2bfb0 2024-03-19T03:33:21.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.566+0000 7fa0a5600640 1 -- 172.21.15.117:0/473505792 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa0a8034440 con 0x561bc1c2bfb0 2024-03-19T03:33:21.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.566+0000 7fa0c5481740 10 client.25412 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T03:33:21.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.566+0000 7fa0c5481740 10 client.25412 resolve_mds: resolved name 'g' to daemon mds.g 2024-03-19T03:33:21.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.566+0000 7fa0c5481740 20 client.25412 populate_metadata read hostname 'smithi117' 2024-03-19T03:33:21.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.566+0000 7fa0c5481740 1 --2- 172.21.15.117:0/473505792 >> [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] conn(0x561bc1c42500 0x561bc1c43970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.567+0000 7fa0c5481740 4 client.25412 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:33:21.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.567+0000 7fa0c5481740 1 -- 172.21.15.117:0/473505792 --> [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561bc1b9d240 con 0x561bc1c42500 2024-03-19T03:33:21.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.567+0000 7fa0bb400640 1 --2- 172.21.15.117:0/473505792 >> [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] conn(0x561bc1c42500 0x561bc1c43970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.567+0000 7fa0a5600640 10 client.25412 ms_handle_connect on v2:172.21.15.117:6838/684335078 2024-03-19T03:33:21.570 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.567+0000 7fa0bb400640 1 --2- 172.21.15.117:0/473505792 >> [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] conn(0x561bc1c42500 0x561bc1c43970 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7ed98c4f7db107d7 server_cookie=1d4fca4ccd9640fa in_seq=0 out_seq=0 2024-03-19T03:33:21.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.569+0000 7fa0a5600640 1 -- 172.21.15.117:0/473505792 <== mds.2 v2:172.21.15.117:6838/684335078 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561bc1b9d240 con 0x561bc1c42500 2024-03-19T03:33:21.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.569+0000 7fa0a5600640 10 client.25412 handle_command_reply: tid=0 2024-03-19T03:33:21.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.569+0000 7fa0c5481740 1 client.25412 shutdown 2024-03-19T03:33:21.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.569+0000 7fa0c5481740 1 -- 172.21.15.117:0/473505792 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c2bfb0 msgr2=0x561bc1c36000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.569+0000 7fa0c5481740 1 --2- 172.21.15.117:0/473505792 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c2bfb0 0x561bc1c36000 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa0a802c7a0 tx=0x7fa0a80038a0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.569+0000 7fa0c5481740 1 -- 172.21.15.117:0/473505792 shutdown_connections 2024-03-19T03:33:21.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.569+0000 7fa0c5481740 1 --2- 172.21.15.117:0/473505792 >> [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] conn(0x561bc1c42500 0x561bc1c43970 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.569+0000 7fa0c5481740 1 --2- 172.21.15.117:0/473505792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc1c31120 0x561bc1c3d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.569+0000 7fa0c5481740 1 --2- 172.21.15.117:0/473505792 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c2bfb0 0x561bc1c36000 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.569+0000 7fa0c5481740 1 --2- 172.21.15.117:0/473505792 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bc1c2b5c0 0x561bc1c35ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.569+0000 7fa0c5481740 1 -- 172.21.15.117:0/473505792 >> 172.21.15.117:0/473505792 conn(0x561bc1c26b30 msgr2=0x561bc1c25d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:21.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.569+0000 7fa0a5600640 0 client.25412 ms_handle_reset on v2:172.21.15.117:6838/684335078 2024-03-19T03:33:21.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.569+0000 7fa0c5481740 1 -- 172.21.15.117:0/473505792 shutdown_connections 2024-03-19T03:33:21.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.570+0000 7fa0c5481740 1 -- 172.21.15.117:0/473505792 wait complete. 2024-03-19T03:33:21.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.570+0000 7fa0c5481740 20 client.25412 trim_cache size 0 max 16384 2024-03-19T03:33:21.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.589+0000 7fa0c5481740 1 Processor -- start 2024-03-19T03:33:21.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.589+0000 7fa0c5481740 1 -- start start 2024-03-19T03:33:21.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.590+0000 7fa0c5481740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc1c2b5c0 0x561bc1c33510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.589 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.590+0000 7fa0c5481740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c2bfb0 0x561bc1c3d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.590+0000 7fa0c5481740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bc1c31120 0x561bc1c26f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.590+0000 7fa0c5481740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561bc1c41cb0 con 0x561bc1c2b5c0 2024-03-19T03:33:21.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.590+0000 7fa0c5481740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561bc1c42380 con 0x561bc1c2bfb0 2024-03-19T03:33:21.590 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.590+0000 7fa0c5481740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x561bc1c42a50 con 0x561bc1c31120 2024-03-19T03:33:21.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.590+0000 7fa0baa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c2bfb0 0x561bc1c3d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.590+0000 7fa0baa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c2bfb0 0x561bc1c3d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52818/0 (socket says 172.21.15.117:52818) 2024-03-19T03:33:21.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.590+0000 7fa0baa00640 1 -- 172.21.15.117:0/4180935650 learned_addr learned my addr 172.21.15.117:0/4180935650 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:21.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.590+0000 7fa0bb400640 1 --2- 172.21.15.117:0/4180935650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc1c2b5c0 0x561bc1c33510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.590+0000 7fa0bbe00640 1 --2- 172.21.15.117:0/4180935650 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bc1c31120 0x561bc1c26f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.590+0000 7fa0baa00640 1 -- 172.21.15.117:0/4180935650 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bc1c31120 msgr2=0x561bc1c26f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.590+0000 7fa0baa00640 1 --2- 172.21.15.117:0/4180935650 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bc1c31120 0x561bc1c26f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.590+0000 7fa0baa00640 1 -- 172.21.15.117:0/4180935650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc1c2b5c0 msgr2=0x561bc1c33510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.590+0000 7fa0baa00640 1 --2- 172.21.15.117:0/4180935650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc1c2b5c0 0x561bc1c33510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.592 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.590+0000 7fa0baa00640 1 -- 172.21.15.117:0/4180935650 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bc1c34130 con 0x561bc1c2bfb0 2024-03-19T03:33:21.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.591+0000 7fa0baa00640 1 --2- 172.21.15.117:0/4180935650 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c2bfb0 0x561bc1c3d000 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8009800 tx=0x7fa0a8005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=132e00931597b8d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:21.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.591+0000 7fa0a7e00640 1 -- 172.21.15.117:0/4180935650 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a804d040 con 0x561bc1c2bfb0 2024-03-19T03:33:21.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.591+0000 7fa0a7e00640 1 -- 172.21.15.117:0/4180935650 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0a8005510 con 0x561bc1c2bfb0 2024-03-19T03:33:21.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.591+0000 7fa0a7e00640 1 -- 172.21.15.117:0/4180935650 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8044040 con 0x561bc1c2bfb0 2024-03-19T03:33:21.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.592+0000 7fa0c5481740 1 -- 172.21.15.117:0/4180935650 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c2bfb0 msgr2=0x561bc1c3d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.592+0000 7fa0c5481740 1 --2- 172.21.15.117:0/4180935650 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c2bfb0 0x561bc1c3d000 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8009800 tx=0x7fa0a8005a20 comp rx=0 tx=0).stop 2024-03-19T03:33:21.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.592+0000 7fa0c5481740 1 -- 172.21.15.117:0/4180935650 shutdown_connections 2024-03-19T03:33:21.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.592+0000 7fa0c5481740 1 --2- 172.21.15.117:0/4180935650 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bc1c31120 0x561bc1c26f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.592+0000 7fa0c5481740 1 --2- 172.21.15.117:0/4180935650 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c2bfb0 0x561bc1c3d000 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.593 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.592+0000 7fa0c5481740 1 --2- 172.21.15.117:0/4180935650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc1c2b5c0 0x561bc1c33510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.592+0000 7fa0c5481740 1 -- 172.21.15.117:0/4180935650 >> 172.21.15.117:0/4180935650 conn(0x561bc1c26b30 msgr2=0x561bc1c2a620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:21.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.592+0000 7fa0c5481740 1 -- 172.21.15.117:0/4180935650 shutdown_connections 2024-03-19T03:33:21.594 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.592+0000 7fa0c5481740 1 -- 172.21.15.117:0/4180935650 wait complete. 2024-03-19T03:33:21.598 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.592+0000 7fa0c5481740 1 -- start start 2024-03-19T03:33:21.598 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.593+0000 7fa0c5481740 1 Processor -- start 2024-03-19T03:33:21.598 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.593+0000 7fa0c5481740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bc1c2b5c0 0x561bc1c391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.593+0000 7fa0c5481740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc1c2bfb0 0x561bc1c396f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.593+0000 7fa0c5481740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c31120 0x561bc1c3c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.593+0000 7fa0c5481740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561bc1c437f0 con 0x561bc1c2bfb0 2024-03-19T03:33:21.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.593+0000 7fa0c5481740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561bc1c43120 con 0x561bc1c31120 2024-03-19T03:33:21.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.593+0000 7fa0c5481740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x561bc1c43ec0 con 0x561bc1c2b5c0 2024-03-19T03:33:21.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.593+0000 7fa0c5481740 10 client.0 _ll_register_callbacks cb 0x7fa0b4009740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:33:21.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.593+0000 7fa0bbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c31120 0x561bc1c3c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.593+0000 7fa0baa00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc1c2bfb0 0x561bc1c396f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.594+0000 7fa0bbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c31120 0x561bc1c3c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52834/0 (socket says 172.21.15.117:52834) 2024-03-19T03:33:21.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.594+0000 7fa0bbe00640 1 -- 172.21.15.117:0/1873742593 learned_addr learned my addr 172.21.15.117:0/1873742593 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:21.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.594+0000 7fa0bb400640 1 --2- 172.21.15.117:0/1873742593 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bc1c2b5c0 0x561bc1c391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.594+0000 7fa0bbe00640 1 -- 172.21.15.117:0/1873742593 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bc1c2b5c0 msgr2=0x561bc1c391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.594+0000 7fa0bbe00640 1 --2- 172.21.15.117:0/1873742593 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bc1c2b5c0 0x561bc1c391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.599 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.594+0000 7fa0bbe00640 1 -- 172.21.15.117:0/1873742593 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc1c2bfb0 msgr2=0x561bc1c396f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.594+0000 7fa0bbe00640 1 --2- 172.21.15.117:0/1873742593 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc1c2bfb0 0x561bc1c396f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.594+0000 7fa0bbe00640 1 -- 172.21.15.117:0/1873742593 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bc1c3c850 con 0x561bc1c31120 2024-03-19T03:33:21.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.594+0000 7fa0bbe00640 1 --2- 172.21.15.117:0/1873742593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c31120 0x561bc1c3c210 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa0b4010a20 tx=0x7fa0b4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d069681c2ee7431 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:21.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.594+0000 7fa0a5600640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-19T03:33:21.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.595+0000 7fa0a5600640 1 -- 172.21.15.117:0/1873742593 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b401e8e0 con 0x561bc1c31120 2024-03-19T03:33:21.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.595+0000 7fa0a5600640 1 -- 172.21.15.117:0/1873742593 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fa0b401ea80 con 0x561bc1c31120 2024-03-19T03:33:21.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.595+0000 7fa0c5481740 1 -- 172.21.15.117:0/1873742593 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0a8005510 con 0x561bc1c31120 2024-03-19T03:33:21.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.595+0000 7fa0a5600640 1 -- 172.21.15.117:0/1873742593 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa0b4036050 con 0x561bc1c31120 2024-03-19T03:33:21.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.595+0000 7fa0c5481740 10 client.25418 fetch_fsmap learned FSMap version 28 2024-03-19T03:33:21.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.595+0000 7fa0c5481740 1 -- 172.21.15.117:0/1873742593 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561bc1ca88b0 con 0x561bc1c31120 2024-03-19T03:33:21.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.595+0000 7fa0a5600640 1 -- 172.21.15.117:0/1873742593 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0a8005510 con 0x561bc1c31120 2024-03-19T03:33:21.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.595+0000 7fa0a5600640 1 -- 172.21.15.117:0/1873742593 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa0b401d400 con 0x561bc1c31120 2024-03-19T03:33:21.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.595+0000 7fa0c5481740 10 client.25418 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T03:33:21.600 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.595+0000 7fa0c5481740 10 client.25418 resolve_mds: resolved name 'g' to daemon mds.g 2024-03-19T03:33:21.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.595+0000 7fa0c5481740 20 client.25418 populate_metadata read hostname 'smithi117' 2024-03-19T03:33:21.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.595+0000 7fa0c5481740 1 --2- 172.21.15.117:0/1873742593 >> [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] conn(0x561bc1cad570 0x561bc1cac280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.595+0000 7fa0c5481740 4 client.25418 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T03:33:21.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.595+0000 7fa0c5481740 1 -- 172.21.15.117:0/1873742593 --> [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] -- command(tid 0: {"prefix": "status"}) -- 0x561bc1b97590 con 0x561bc1cad570 2024-03-19T03:33:21.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.596+0000 7fa0bb400640 1 --2- 172.21.15.117:0/1873742593 >> [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] conn(0x561bc1cad570 0x561bc1cac280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.596+0000 7fa0bb400640 1 --2- 172.21.15.117:0/1873742593 >> [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] conn(0x561bc1cad570 0x561bc1cac280 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7925d5c1e153d497 server_cookie=b81af50e83340bba in_seq=0 out_seq=0 2024-03-19T03:33:21.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.596+0000 7fa0a5600640 10 client.25418 ms_handle_connect on v2:172.21.15.117:6838/684335078 2024-03-19T03:33:21.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.597+0000 7fa0a5600640 1 -- 172.21.15.117:0/1873742593 <== mds.2 v2:172.21.15.117:6838/684335078 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x561bc1b97590 con 0x561bc1cad570 2024-03-19T03:33:21.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.597+0000 7fa0a5600640 10 client.25418 handle_command_reply: tid=0 2024-03-19T03:33:21.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.597+0000 7fa0c5481740 1 client.25418 shutdown 2024-03-19T03:33:21.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.597+0000 7fa0c5481740 1 -- 172.21.15.117:0/1873742593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c31120 msgr2=0x561bc1c3c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.597+0000 7fa0c5481740 1 --2- 172.21.15.117:0/1873742593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c31120 0x561bc1c3c210 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa0b4010a20 tx=0x7fa0b4004a40 comp rx=0 tx=0).stop 2024-03-19T03:33:21.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.597+0000 7fa0c5481740 1 -- 172.21.15.117:0/1873742593 shutdown_connections 2024-03-19T03:33:21.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.597+0000 7fa0c5481740 1 --2- 172.21.15.117:0/1873742593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bc1c31120 0x561bc1c3c210 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.597+0000 7fa0c5481740 1 --2- 172.21.15.117:0/1873742593 >> [v2:172.21.15.117:6838/684335078,v1:172.21.15.117:6839/684335078] conn(0x561bc1cad570 0x561bc1cac280 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.597+0000 7fa0c5481740 1 --2- 172.21.15.117:0/1873742593 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc1c2bfb0 0x561bc1c396f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.597+0000 7fa0c5481740 1 --2- 172.21.15.117:0/1873742593 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bc1c2b5c0 0x561bc1c391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.597+0000 7fa0c5481740 1 -- 172.21.15.117:0/1873742593 >> 172.21.15.117:0/1873742593 conn(0x561bc1c26b30 msgr2=0x561bc1c2a620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:21.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.597+0000 7fa0a5600640 0 client.25418 ms_handle_reset on v2:172.21.15.117:6838/684335078 2024-03-19T03:33:21.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.597+0000 7fa0c5481740 1 -- 172.21.15.117:0/1873742593 shutdown_connections 2024-03-19T03:33:21.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.597+0000 7fa0c5481740 1 -- 172.21.15.117:0/1873742593 wait complete. 2024-03-19T03:33:21.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.597+0000 7fa0c5481740 20 client.25418 trim_cache size 0 max 16384 2024-03-19T03:33:21.603 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-19T03:33:21.603 INFO:teuthology.orchestra.run.smithi117.stdout: "cluster_fsid": "a3a82456-e59f-11ee-95c9-87774f69a715", 2024-03-19T03:33:21.603 INFO:teuthology.orchestra.run.smithi117.stdout: "whoami": 2, 2024-03-19T03:33:21.603 INFO:teuthology.orchestra.run.smithi117.stdout: "id": 24569, 2024-03-19T03:33:21.603 INFO:teuthology.orchestra.run.smithi117.stdout: "want_state": "up:active", 2024-03-19T03:33:21.603 INFO:teuthology.orchestra.run.smithi117.stdout: "state": "up:active", 2024-03-19T03:33:21.603 INFO:teuthology.orchestra.run.smithi117.stdout: "fs_name": "cephfs", 2024-03-19T03:33:21.603 INFO:teuthology.orchestra.run.smithi117.stdout: "rank_uptime": 134.13099885599999, 2024-03-19T03:33:21.603 INFO:teuthology.orchestra.run.smithi117.stdout: "mdsmap_epoch": 28, 2024-03-19T03:33:21.603 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch": 93, 2024-03-19T03:33:21.604 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch_barrier": 89, 2024-03-19T03:33:21.604 INFO:teuthology.orchestra.run.smithi117.stdout: "uptime": 180.37912017900001 2024-03-19T03:33:21.604 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-19T03:33:21.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.598+0000 7fa0c1200640 1 -- 172.21.15.117:0/3940570546 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa0a00a2710 msgr2=0x7fa0a00a4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.598+0000 7fa0c1200640 1 --2- 172.21.15.117:0/3940570546 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa0a00a2710 0x7fa0a00a4bd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x561bc1c2c810 tx=0x7fa0b40098a0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.598+0000 7fa0c1200640 1 -- 172.21.15.117:0/3940570546 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0bc074280 msgr2=0x7fa0bc19dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.598+0000 7fa0c1200640 1 --2- 172.21.15.117:0/3940570546 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0bc074280 0x7fa0bc19dfc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac00e9f0 tx=0x7fa0ac00edf0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.598+0000 7fa0c1200640 1 -- 172.21.15.117:0/3940570546 shutdown_connections 2024-03-19T03:33:21.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.598+0000 7fa0c1200640 1 --2- 172.21.15.117:0/3940570546 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa0bc10dc00 0x7fa0bc1a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.598+0000 7fa0c1200640 1 --2- 172.21.15.117:0/3940570546 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa0bc075860 0x7fa0bc19e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.598+0000 7fa0c1200640 1 --2- 172.21.15.117:0/3940570546 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa0a00a2710 0x7fa0a00a4bd0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.598+0000 7fa0c1200640 1 --2- 172.21.15.117:0/3940570546 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0bc074280 0x7fa0bc19dfc0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.598+0000 7fa0c1200640 1 -- 172.21.15.117:0/3940570546 >> 172.21.15.117:0/3940570546 conn(0x7fa0bc0fcb80 msgr2=0x7fa0bc100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:21.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.598+0000 7fa0c1200640 1 -- 172.21.15.117:0/3940570546 shutdown_connections 2024-03-19T03:33:21.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.598+0000 7fa0c1200640 1 -- 172.21.15.117:0/3940570546 wait complete. 2024-03-19T03:33:21.613 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:21 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1419837190' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:33:21.616 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-19T03:33:21.747 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:21 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1419837190' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:33:21.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.807+0000 7f7602c00640 1 -- 172.21.15.117:0/937005686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75fc107560 msgr2=0x7f75fc109950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.807+0000 7f7602c00640 1 --2- 172.21.15.117:0/937005686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75fc107560 0x7f75fc109950 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f75f4009ca0 tx=0x7f75f402c0f0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.808+0000 7f7602c00640 1 -- 172.21.15.117:0/937005686 shutdown_connections 2024-03-19T03:33:21.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.808+0000 7f7602c00640 1 --2- 172.21.15.117:0/937005686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75fc107560 0x7f75fc109950 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.808+0000 7f7602c00640 1 --2- 172.21.15.117:0/937005686 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75fc104c30 0x7f75fc107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.808+0000 7f7602c00640 1 --2- 172.21.15.117:0/937005686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75fc102300 0x7f75fc1046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.808+0000 7f7602c00640 1 -- 172.21.15.117:0/937005686 >> 172.21.15.117:0/937005686 conn(0x7f75fc0fbc90 msgr2=0x7f75fc0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:21.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.808+0000 7f7602c00640 1 -- 172.21.15.117:0/937005686 shutdown_connections 2024-03-19T03:33:21.808 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.808+0000 7f7602c00640 1 -- 172.21.15.117:0/937005686 wait complete. 2024-03-19T03:33:21.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.809+0000 7f7602c00640 1 Processor -- start 2024-03-19T03:33:21.810 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.809+0000 7f7602c00640 1 -- start start 2024-03-19T03:33:21.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.809+0000 7f7602c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75fc102300 0x7f75fc197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.809+0000 7f7602c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75fc104c30 0x7f75fc198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.809+0000 7f7602c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75fc107560 0x7f75fc19f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.809+0000 7f7602c00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f75fc10d4c0 con 0x7f75fc102300 2024-03-19T03:33:21.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.809+0000 7f7602c00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f75fc10d340 con 0x7f75fc107560 2024-03-19T03:33:21.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.809+0000 7f7602c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f75fc10d640 con 0x7f75fc104c30 2024-03-19T03:33:21.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.809+0000 7f75fbe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75fc102300 0x7f75fc197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.809+0000 7f75fbe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75fc102300 0x7f75fc197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42406/0 (socket says 172.21.15.117:42406) 2024-03-19T03:33:21.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.809+0000 7f75fbe00640 1 -- 172.21.15.117:0/2815293436 learned_addr learned my addr 172.21.15.117:0/2815293436 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:21.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.809+0000 7f7600c00640 1 --2- 172.21.15.117:0/2815293436 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75fc107560 0x7f75fc19f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.809+0000 7f75fb400640 1 --2- 172.21.15.117:0/2815293436 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75fc104c30 0x7f75fc198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.810+0000 7f75fbe00640 1 -- 172.21.15.117:0/2815293436 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75fc104c30 msgr2=0x7f75fc198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.810+0000 7f75fbe00640 1 --2- 172.21.15.117:0/2815293436 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75fc104c30 0x7f75fc198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.811 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.810+0000 7f75fbe00640 1 -- 172.21.15.117:0/2815293436 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75fc107560 msgr2=0x7f75fc19f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.810+0000 7f75fbe00640 1 --2- 172.21.15.117:0/2815293436 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75fc107560 0x7f75fc19f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.810+0000 7f75fbe00640 1 -- 172.21.15.117:0/2815293436 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75fc19f700 con 0x7f75fc102300 2024-03-19T03:33:21.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.810+0000 7f7600c00640 1 --2- 172.21.15.117:0/2815293436 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75fc107560 0x7f75fc19f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:33:21.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.810+0000 7f75fbe00640 1 --2- 172.21.15.117:0/2815293436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75fc102300 0x7f75fc197ae0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00e9f0 tx=0x7f75ec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:21.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.810+0000 7f75f8c00640 1 -- 172.21.15.117:0/2815293436 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ec00cd80 con 0x7f75fc102300 2024-03-19T03:33:21.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.810+0000 7f75f8c00640 1 -- 172.21.15.117:0/2815293436 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75ec010910 con 0x7f75fc102300 2024-03-19T03:33:21.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.810+0000 7f75f8c00640 1 -- 172.21.15.117:0/2815293436 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ec018990 con 0x7f75fc102300 2024-03-19T03:33:21.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.810+0000 7f7602c00640 1 -- 172.21.15.117:0/2815293436 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75fc19f9f0 con 0x7f75fc102300 2024-03-19T03:33:21.812 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.811+0000 7f7602c00640 1 -- 172.21.15.117:0/2815293436 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75fc19fef0 con 0x7f75fc102300 2024-03-19T03:33:21.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.814+0000 7f76057f7740 1 Processor -- start 2024-03-19T03:33:21.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.814+0000 7f76057f7740 1 -- start start 2024-03-19T03:33:21.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.814+0000 7f76057f7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7bb0370 0x561bf7bb0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.814+0000 7f76057f7740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bf7bb0d60 0x561bf7bb3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.816 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.814+0000 7f76057f7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bf7bb3ec0 0x561bf7bc0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.814+0000 7f76057f7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561bf7b14d00 con 0x561bf7bb3ec0 2024-03-19T03:33:21.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.814+0000 7f76057f7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561bf7abdb00 con 0x561bf7bb0370 2024-03-19T03:33:21.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.814+0000 7f76057f7740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x561bf7b10da0 con 0x561bf7bb0d60 2024-03-19T03:33:21.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.814+0000 7f7600c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7bb0370 0x561bf7bb0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.815+0000 7f7600c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7bb0370 0x561bf7bb0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52858/0 (socket says 172.21.15.117:52858) 2024-03-19T03:33:21.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.815+0000 7f7600c00640 1 -- 172.21.15.117:0/4161893940 learned_addr learned my addr 172.21.15.117:0/4161893940 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:21.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.815+0000 7f75fb400640 1 --2- 172.21.15.117:0/4161893940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bf7bb3ec0 0x561bf7bc0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.815+0000 7f7600c00640 1 -- 172.21.15.117:0/4161893940 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bf7bb0d60 msgr2=0x561bf7bb3850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T03:33:21.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.815+0000 7f7600c00640 1 --2- 172.21.15.117:0/4161893940 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bf7bb0d60 0x561bf7bb3850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.815+0000 7f7600c00640 1 -- 172.21.15.117:0/4161893940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bf7bb3ec0 msgr2=0x561bf7bc0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.815+0000 7f7600c00640 1 --2- 172.21.15.117:0/4161893940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bf7bb3ec0 0x561bf7bc0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.815+0000 7f7600c00640 1 -- 172.21.15.117:0/4161893940 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bf78292d0 con 0x561bf7bb0370 2024-03-19T03:33:21.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.815+0000 7f75f8c00640 1 -- 172.21.15.117:0/2815293436 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f75ec010ac0 con 0x7f75fc102300 2024-03-19T03:33:21.817 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.815+0000 7f7600c00640 1 --2- 172.21.15.117:0/4161893940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7bb0370 0x561bf7bb0790 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f75f402c700 tx=0x7f75f40323d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f03e37a348788854 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:21.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.816+0000 7f75f1200640 1 -- 172.21.15.117:0/4161893940 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f4043070 con 0x561bf7bb0370 2024-03-19T03:33:21.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.816+0000 7f75f1200640 1 -- 172.21.15.117:0/4161893940 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75f402cca0 con 0x561bf7bb0370 2024-03-19T03:33:21.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.816+0000 7f75f1200640 1 -- 172.21.15.117:0/4161893940 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f4039040 con 0x561bf7bb0370 2024-03-19T03:33:21.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.816+0000 7f76057f7740 1 -- 172.21.15.117:0/4161893940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7bb0370 msgr2=0x561bf7bb0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.816+0000 7f76057f7740 1 --2- 172.21.15.117:0/4161893940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7bb0370 0x561bf7bb0790 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f75f402c700 tx=0x7f75f40323d0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.816+0000 7f76057f7740 1 -- 172.21.15.117:0/4161893940 shutdown_connections 2024-03-19T03:33:21.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.816+0000 7f76057f7740 1 --2- 172.21.15.117:0/4161893940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bf7bb3ec0 0x561bf7bc0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.816+0000 7f76057f7740 1 --2- 172.21.15.117:0/4161893940 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bf7bb0d60 0x561bf7bb3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.816+0000 7f76057f7740 1 --2- 172.21.15.117:0/4161893940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7bb0370 0x561bf7bb0790 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.816+0000 7f76057f7740 1 -- 172.21.15.117:0/4161893940 >> 172.21.15.117:0/4161893940 conn(0x561bf7bab960 msgr2=0x561bf7babd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:21.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.817+0000 7f76057f7740 1 -- 172.21.15.117:0/4161893940 shutdown_connections 2024-03-19T03:33:21.818 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.817+0000 7f76057f7740 1 -- 172.21.15.117:0/4161893940 wait complete. 2024-03-19T03:33:21.821 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.817+0000 7f76057f7740 1 -- start start 2024-03-19T03:33:21.821 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.817+0000 7f76057f7740 1 Processor -- start 2024-03-19T03:33:21.821 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.818+0000 7f76057f7740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bf7bb0370 0x561bf7bc3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.822 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.818+0000 7f76057f7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7bb0d60 0x561bf7bc3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.822 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.818+0000 7f76057f7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bf7bb3ec0 0x561bf7bca690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.822 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.818+0000 7f76057f7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561bf7bc16d0 con 0x561bf7bb3ec0 2024-03-19T03:33:21.822 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.818+0000 7f75f8c00640 1 --2- 172.21.15.117:0/2815293436 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f75e00a2790 0x7f75e00a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.822 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.818+0000 7f76057f7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561bf7ad0ed0 con 0x561bf7bb0d60 2024-03-19T03:33:21.822 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.818+0000 7f76057f7740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x561bf7bc1850 con 0x561bf7bb0370 2024-03-19T03:33:21.822 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.818+0000 7f76057f7740 10 client.0 _ll_register_callbacks cb 0x561bf7acec10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:33:21.822 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.818+0000 7f75fbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7bb0d60 0x561bf7bc3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.822 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.818+0000 7f75fbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7bb0d60 0x561bf7bc3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52872/0 (socket says 172.21.15.117:52872) 2024-03-19T03:33:21.822 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.818+0000 7f75fbe00640 1 -- 172.21.15.117:0/3119939406 learned_addr learned my addr 172.21.15.117:0/3119939406 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:21.822 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.818+0000 7f75fb400640 1 --2- 172.21.15.117:0/3119939406 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bf7bb3ec0 0x561bf7bca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.822 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.818+0000 7f75f8c00640 1 -- 172.21.15.117:0/2815293436 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f75ec0cf1c0 con 0x7f75fc102300 2024-03-19T03:33:21.822 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.818+0000 7f7600c00640 1 --2- 172.21.15.117:0/3119939406 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bf7bb0370 0x561bf7bc3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.819+0000 7f7600c00640 1 --2- 172.21.15.117:0/2815293436 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f75e00a2790 0x7f75e00a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.819+0000 7f75fb400640 1 -- 172.21.15.117:0/3119939406 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bf7bb0370 msgr2=0x561bf7bc3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.819+0000 7f75fb400640 1 --2- 172.21.15.117:0/3119939406 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bf7bb0370 0x561bf7bc3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.819+0000 7f75fb400640 1 -- 172.21.15.117:0/3119939406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7bb0d60 msgr2=0x561bf7bc3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.819+0000 7f75fb400640 1 --2- 172.21.15.117:0/3119939406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7bb0d60 0x561bf7bc3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.819+0000 7f75fb400640 1 -- 172.21.15.117:0/3119939406 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bf78f36b0 con 0x561bf7bb3ec0 2024-03-19T03:33:21.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.819+0000 7f75fb400640 1 --2- 172.21.15.117:0/3119939406 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bf7bb3ec0 0x561bf7bca690 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f75fc198ff0 tx=0x7f75e800bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de618df0b3d8cf43 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:21.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.819+0000 7f75e5600640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-19T03:33:21.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.819+0000 7f75e5600640 1 -- 172.21.15.117:0/3119939406 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e8003e50 con 0x561bf7bb3ec0 2024-03-19T03:33:21.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.819+0000 7f75e5600640 1 -- 172.21.15.117:0/3119939406 <== mon.0 v2:172.21.15.117:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f75e8005020 con 0x561bf7bb3ec0 2024-03-19T03:33:21.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.819+0000 7f76057f7740 1 -- 172.21.15.117:0/3119939406 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75f402cca0 con 0x561bf7bb3ec0 2024-03-19T03:33:21.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.820+0000 7f75e5600640 1 -- 172.21.15.117:0/3119939406 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f75e8005220 con 0x561bf7bb3ec0 2024-03-19T03:33:21.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.820+0000 7f7600c00640 1 --2- 172.21.15.117:0/2815293436 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f75e00a2790 0x7f75e00a4c50 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f75f4009ca0 tx=0x7f75f40095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:21.823 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.820+0000 7f76057f7740 10 client.15291 fetch_fsmap learned FSMap version 28 2024-03-19T03:33:21.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.820+0000 7f76057f7740 1 -- 172.21.15.117:0/3119939406 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561bf7bcdad0 con 0x561bf7bb3ec0 2024-03-19T03:33:21.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.820+0000 7f75e5600640 1 -- 172.21.15.117:0/3119939406 <== mon.0 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75f402cca0 con 0x561bf7bb3ec0 2024-03-19T03:33:21.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.821+0000 7f75e5600640 1 -- 172.21.15.117:0/3119939406 <== mon.0 v2:172.21.15.117:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f75e801b500 con 0x561bf7bb3ec0 2024-03-19T03:33:21.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.821+0000 7f76057f7740 10 client.15291 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T03:33:21.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.821+0000 7f76057f7740 10 client.15291 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-19T03:33:21.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.821+0000 7f76057f7740 20 client.15291 populate_metadata read hostname 'smithi117' 2024-03-19T03:33:21.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.821+0000 7f76057f7740 1 --2- 172.21.15.117:0/3119939406 >> [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] conn(0x561bf7bd0080 0x561bf7bd14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.821+0000 7f76057f7740 4 client.15291 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:33:21.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.821+0000 7f76057f7740 1 -- 172.21.15.117:0/3119939406 --> [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561bf7b22240 con 0x561bf7bd0080 2024-03-19T03:33:21.824 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.821+0000 7f75fbe00640 1 --2- 172.21.15.117:0/3119939406 >> [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] conn(0x561bf7bd0080 0x561bf7bd14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.825 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.822+0000 7f75fbe00640 1 --2- 172.21.15.117:0/3119939406 >> [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] conn(0x561bf7bd0080 0x561bf7bd14f0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5643537ae32f71a3 server_cookie=98993dc486e9d05f in_seq=0 out_seq=0 2024-03-19T03:33:21.825 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.822+0000 7f75e5600640 10 client.15291 ms_handle_connect on v2:172.21.15.117:6840/1329933892 2024-03-19T03:33:21.825 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.823+0000 7f75e5600640 1 -- 172.21.15.117:0/3119939406 <== mds.1 v2:172.21.15.117:6840/1329933892 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561bf7b22240 con 0x561bf7bd0080 2024-03-19T03:33:21.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.823+0000 7f75e5600640 10 client.15291 handle_command_reply: tid=0 2024-03-19T03:33:21.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.823+0000 7f76057f7740 1 client.15291 shutdown 2024-03-19T03:33:21.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.823+0000 7f76057f7740 1 -- 172.21.15.117:0/3119939406 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bf7bb3ec0 msgr2=0x561bf7bca690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.823+0000 7f76057f7740 1 --2- 172.21.15.117:0/3119939406 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bf7bb3ec0 0x561bf7bca690 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f75fc198ff0 tx=0x7f75e800bdf0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.823+0000 7f76057f7740 1 -- 172.21.15.117:0/3119939406 shutdown_connections 2024-03-19T03:33:21.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.823+0000 7f76057f7740 1 --2- 172.21.15.117:0/3119939406 >> [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] conn(0x561bf7bd0080 0x561bf7bd14f0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.823+0000 7f76057f7740 1 --2- 172.21.15.117:0/3119939406 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bf7bb3ec0 0x561bf7bca690 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.823+0000 7f76057f7740 1 --2- 172.21.15.117:0/3119939406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7bb0d60 0x561bf7bc3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.823+0000 7f76057f7740 1 --2- 172.21.15.117:0/3119939406 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bf7bb0370 0x561bf7bc3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.823+0000 7f76057f7740 1 -- 172.21.15.117:0/3119939406 >> 172.21.15.117:0/3119939406 conn(0x561bf7bab960 msgr2=0x561bf7baaaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:21.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.823+0000 7f75e5600640 0 client.15291 ms_handle_reset on v2:172.21.15.117:6840/1329933892 2024-03-19T03:33:21.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.823+0000 7f76057f7740 1 -- 172.21.15.117:0/3119939406 shutdown_connections 2024-03-19T03:33:21.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.823+0000 7f76057f7740 1 -- 172.21.15.117:0/3119939406 wait complete. 2024-03-19T03:33:21.826 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.824+0000 7f76057f7740 20 client.15291 trim_cache size 0 max 16384 2024-03-19T03:33:21.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.843+0000 7f76057f7740 1 Processor -- start 2024-03-19T03:33:21.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.843+0000 7f76057f7740 1 -- start start 2024-03-19T03:33:21.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.843+0000 7f76057f7740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bf7bb0370 0x561bf7babd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.843+0000 7f76057f7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bf7bb0d60 0x561bf7bca650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.843+0000 7f76057f7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7baa210 0x561bf7baa650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.843+0000 7f76057f7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561bf7bcf830 con 0x561bf7bb0d60 2024-03-19T03:33:21.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.843+0000 7f76057f7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561bf7bcff00 con 0x561bf7baa210 2024-03-19T03:33:21.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.843+0000 7f76057f7740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x561bf7bd05d0 con 0x561bf7bb0370 2024-03-19T03:33:21.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.844+0000 7f75fb400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bf7bb0d60 0x561bf7bca650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.844+0000 7f7600c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7baa210 0x561bf7baa650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.844+0000 7f7600c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7baa210 0x561bf7baa650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52886/0 (socket says 172.21.15.117:52886) 2024-03-19T03:33:21.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.844+0000 7f7600c00640 1 -- 172.21.15.117:0/1370973961 learned_addr learned my addr 172.21.15.117:0/1370973961 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:21.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.844+0000 7f75fb400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bf7bb0d60 0x561bf7bca650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42440/0 (socket says 172.21.15.117:42440) 2024-03-19T03:33:21.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.844+0000 7f75fbe00640 1 --2- 172.21.15.117:0/1370973961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bf7bb0370 0x561bf7babd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.844+0000 7f7600c00640 1 -- 172.21.15.117:0/1370973961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bf7bb0370 msgr2=0x561bf7babd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.844+0000 7f7600c00640 1 --2- 172.21.15.117:0/1370973961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bf7bb0370 0x561bf7babd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.844+0000 7f7600c00640 1 -- 172.21.15.117:0/1370973961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bf7bb0d60 msgr2=0x561bf7bca650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.844+0000 7f7600c00640 1 --2- 172.21.15.117:0/1370973961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bf7bb0d60 0x561bf7bca650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.844+0000 7f7600c00640 1 -- 172.21.15.117:0/1370973961 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bf7baffb0 con 0x561bf7baa210 2024-03-19T03:33:21.845 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.844+0000 7f75fbe00640 1 --2- 172.21.15.117:0/1370973961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bf7bb0370 0x561bf7babd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:33:21.846 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.844+0000 7f7600c00640 1 --2- 172.21.15.117:0/1370973961 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7baa210 0x561bf7baa650 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f75f402c700 tx=0x7f75f404c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40f4da2312684058 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:21.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.845+0000 7f75e7e00640 1 -- 172.21.15.117:0/1370973961 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f405c070 con 0x561bf7baa210 2024-03-19T03:33:21.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.845+0000 7f75e7e00640 1 -- 172.21.15.117:0/1370973961 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75f4003970 con 0x561bf7baa210 2024-03-19T03:33:21.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.845+0000 7f75e7e00640 1 -- 172.21.15.117:0/1370973961 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f4039070 con 0x561bf7baa210 2024-03-19T03:33:21.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.846+0000 7f76057f7740 1 -- 172.21.15.117:0/1370973961 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7baa210 msgr2=0x561bf7baa650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.846+0000 7f76057f7740 1 --2- 172.21.15.117:0/1370973961 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7baa210 0x561bf7baa650 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f75f402c700 tx=0x7f75f404c000 comp rx=0 tx=0).stop 2024-03-19T03:33:21.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.846+0000 7f76057f7740 1 -- 172.21.15.117:0/1370973961 shutdown_connections 2024-03-19T03:33:21.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.846+0000 7f76057f7740 1 --2- 172.21.15.117:0/1370973961 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7baa210 0x561bf7baa650 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.846+0000 7f76057f7740 1 --2- 172.21.15.117:0/1370973961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bf7bb0d60 0x561bf7bca650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.846+0000 7f76057f7740 1 --2- 172.21.15.117:0/1370973961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bf7bb0370 0x561bf7babd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.846+0000 7f76057f7740 1 -- 172.21.15.117:0/1370973961 >> 172.21.15.117:0/1370973961 conn(0x561bf7bab960 msgr2=0x561bf7bc1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:21.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.846+0000 7f76057f7740 1 -- 172.21.15.117:0/1370973961 shutdown_connections 2024-03-19T03:33:21.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.846+0000 7f76057f7740 1 -- 172.21.15.117:0/1370973961 wait complete. 2024-03-19T03:33:21.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.846+0000 7f76057f7740 1 -- start start 2024-03-19T03:33:21.847 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.847+0000 7f76057f7740 1 Processor -- start 2024-03-19T03:33:21.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.847+0000 7f76057f7740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bf7baa210 0x561bf7bca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.847+0000 7f76057f7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bf7bb0370 0x561bf7bca610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.847+0000 7f76057f7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7bb0d60 0x561bf7bc6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.847+0000 7f76057f7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561bf7bd1370 con 0x561bf7bb0370 2024-03-19T03:33:21.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.847+0000 7f76057f7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x561bf7bd0ca0 con 0x561bf7bb0d60 2024-03-19T03:33:21.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.847+0000 7f76057f7740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x561bf7bd1a40 con 0x561bf7baa210 2024-03-19T03:33:21.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.848+0000 7f76057f7740 10 client.0 _ll_register_callbacks cb 0x7f75e80042b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:33:21.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.848+0000 7f7600c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7bb0d60 0x561bf7bc6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.848+0000 7f75fb400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bf7bb0370 0x561bf7bca610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.848+0000 7f7600c00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7bb0d60 0x561bf7bc6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52896/0 (socket says 172.21.15.117:52896) 2024-03-19T03:33:21.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.848+0000 7f7600c00640 1 -- 172.21.15.117:0/2171525321 learned_addr learned my addr 172.21.15.117:0/2171525321 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:21.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.848+0000 7f75fbe00640 1 --2- 172.21.15.117:0/2171525321 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bf7baa210 0x561bf7bca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.848+0000 7f7600c00640 1 -- 172.21.15.117:0/2171525321 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bf7baa210 msgr2=0x561bf7bca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.848+0000 7f7600c00640 1 --2- 172.21.15.117:0/2171525321 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bf7baa210 0x561bf7bca0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.848+0000 7f7600c00640 1 -- 172.21.15.117:0/2171525321 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bf7bb0370 msgr2=0x561bf7bca610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:21.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.848+0000 7f7600c00640 1 --2- 172.21.15.117:0/2171525321 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bf7bb0370 0x561bf7bca610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.848+0000 7f7600c00640 1 -- 172.21.15.117:0/2171525321 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bf7bc7540 con 0x561bf7bb0d60 2024-03-19T03:33:21.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.848+0000 7f7600c00640 1 --2- 172.21.15.117:0/2171525321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7bb0d60 0x561bf7bc6d40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f75f402c630 tx=0x7f75f4032b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcd7b670f8bae88 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:21.852 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.848+0000 7f75e5600640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-19T03:33:21.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.849+0000 7f75e5600640 1 -- 172.21.15.117:0/2171525321 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f405c030 con 0x561bf7bb0d60 2024-03-19T03:33:21.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.849+0000 7f75e5600640 1 -- 172.21.15.117:0/2171525321 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f75f4034a90 con 0x561bf7bb0d60 2024-03-19T03:33:21.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.849+0000 7f76057f7740 1 -- 172.21.15.117:0/2171525321 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75f4003970 con 0x561bf7bb0d60 2024-03-19T03:33:21.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.849+0000 7f75e5600640 1 -- 172.21.15.117:0/2171525321 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f75f4039040 con 0x561bf7bb0d60 2024-03-19T03:33:21.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.849+0000 7f76057f7740 10 client.25436 fetch_fsmap learned FSMap version 28 2024-03-19T03:33:21.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.849+0000 7f76057f7740 1 -- 172.21.15.117:0/2171525321 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561bf7c36be0 con 0x561bf7bb0d60 2024-03-19T03:33:21.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.849+0000 7f75e5600640 1 -- 172.21.15.117:0/2171525321 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75f4003970 con 0x561bf7bb0d60 2024-03-19T03:33:21.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.849+0000 7f75e5600640 1 -- 172.21.15.117:0/2171525321 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f75f4048570 con 0x561bf7bb0d60 2024-03-19T03:33:21.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.849+0000 7f76057f7740 10 client.25436 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T03:33:21.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.850+0000 7f76057f7740 10 client.25436 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-19T03:33:21.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.850+0000 7f76057f7740 20 client.25436 populate_metadata read hostname 'smithi117' 2024-03-19T03:33:21.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.850+0000 7f76057f7740 1 --2- 172.21.15.117:0/2171525321 >> [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] conn(0x561bf7c3b200 0x561bf7c39f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:21.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.850+0000 7f76057f7740 4 client.25436 mds_command: new command op to 24581 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T03:33:21.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.850+0000 7f76057f7740 1 -- 172.21.15.117:0/2171525321 --> [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] -- command(tid 0: {"prefix": "status"}) -- 0x561bf7b1c590 con 0x561bf7c3b200 2024-03-19T03:33:21.853 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.850+0000 7f75fbe00640 1 --2- 172.21.15.117:0/2171525321 >> [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] conn(0x561bf7c3b200 0x561bf7c39f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:21.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.850+0000 7f75fbe00640 1 --2- 172.21.15.117:0/2171525321 >> [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] conn(0x561bf7c3b200 0x561bf7c39f10 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2d830483b32f4a7e server_cookie=6cada602c626fbf1 in_seq=0 out_seq=0 2024-03-19T03:33:21.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.850+0000 7f75e5600640 10 client.25436 ms_handle_connect on v2:172.21.15.117:6840/1329933892 2024-03-19T03:33:21.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.851+0000 7f75e5600640 1 -- 172.21.15.117:0/2171525321 <== mds.1 v2:172.21.15.117:6840/1329933892 1 ==== command_reply(tid 0: 0 ) ==== 8+0+320 (crc 0 0 0) 0x561bf7b1c590 con 0x561bf7c3b200 2024-03-19T03:33:21.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.851+0000 7f75e5600640 10 client.25436 handle_command_reply: tid=0 2024-03-19T03:33:21.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.851+0000 7f76057f7740 1 client.25436 shutdown 2024-03-19T03:33:21.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.851+0000 7f76057f7740 1 -- 172.21.15.117:0/2171525321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7bb0d60 msgr2=0x561bf7bc6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.851+0000 7f76057f7740 1 --2- 172.21.15.117:0/2171525321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7bb0d60 0x561bf7bc6d40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f75f402c630 tx=0x7f75f4032b00 comp rx=0 tx=0).stop 2024-03-19T03:33:21.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.851+0000 7f76057f7740 1 -- 172.21.15.117:0/2171525321 shutdown_connections 2024-03-19T03:33:21.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.851+0000 7f76057f7740 1 --2- 172.21.15.117:0/2171525321 >> [v2:172.21.15.117:6840/1329933892,v1:172.21.15.117:6841/1329933892] conn(0x561bf7c3b200 0x561bf7c39f10 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.851+0000 7f76057f7740 1 --2- 172.21.15.117:0/2171525321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561bf7bb0d60 0x561bf7bc6d40 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.851+0000 7f76057f7740 1 --2- 172.21.15.117:0/2171525321 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bf7bb0370 0x561bf7bca610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.851+0000 7f76057f7740 1 --2- 172.21.15.117:0/2171525321 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561bf7baa210 0x561bf7bca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.851+0000 7f75e5600640 0 client.25436 ms_handle_reset on v2:172.21.15.117:6840/1329933892 2024-03-19T03:33:21.854 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.851+0000 7f76057f7740 1 -- 172.21.15.117:0/2171525321 >> 172.21.15.117:0/2171525321 conn(0x561bf7bab960 msgr2=0x561bf7baa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:21.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.851+0000 7f76057f7740 1 -- 172.21.15.117:0/2171525321 shutdown_connections 2024-03-19T03:33:21.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.851+0000 7f76057f7740 1 -- 172.21.15.117:0/2171525321 wait complete. 2024-03-19T03:33:21.855 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.851+0000 7f76057f7740 20 client.25436 trim_cache size 0 max 16384 2024-03-19T03:33:21.855 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-19T03:33:21.855 INFO:teuthology.orchestra.run.smithi117.stdout: "cluster_fsid": "a3a82456-e59f-11ee-95c9-87774f69a715", 2024-03-19T03:33:21.855 INFO:teuthology.orchestra.run.smithi117.stdout: "whoami": 1, 2024-03-19T03:33:21.855 INFO:teuthology.orchestra.run.smithi117.stdout: "id": 24581, 2024-03-19T03:33:21.855 INFO:teuthology.orchestra.run.smithi117.stdout: "want_state": "up:active", 2024-03-19T03:33:21.855 INFO:teuthology.orchestra.run.smithi117.stdout: "state": "up:active", 2024-03-19T03:33:21.855 INFO:teuthology.orchestra.run.smithi117.stdout: "fs_name": "cephfs", 2024-03-19T03:33:21.855 INFO:teuthology.orchestra.run.smithi117.stdout: "rank_uptime": 135.3972431, 2024-03-19T03:33:21.856 INFO:teuthology.orchestra.run.smithi117.stdout: "mdsmap_epoch": 28, 2024-03-19T03:33:21.856 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch": 93, 2024-03-19T03:33:21.856 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch_barrier": 89, 2024-03-19T03:33:21.856 INFO:teuthology.orchestra.run.smithi117.stdout: "uptime": 176.467297672 2024-03-19T03:33:21.856 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-19T03:33:21.856 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.852+0000 7f7602c00640 1 -- 172.21.15.117:0/2815293436 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f75e00a2790 msgr2=0x7f75e00a4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.856 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.852+0000 7f7602c00640 1 --2- 172.21.15.117:0/2815293436 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f75e00a2790 0x7f75e00a4c50 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f75f4009ca0 tx=0x7f75f40095c0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.856 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.852+0000 7f7602c00640 1 -- 172.21.15.117:0/2815293436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75fc102300 msgr2=0x7f75fc197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:21.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.852+0000 7f7602c00640 1 --2- 172.21.15.117:0/2815293436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75fc102300 0x7f75fc197ae0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00e9f0 tx=0x7f75ec00eeb0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.852+0000 7f7602c00640 1 -- 172.21.15.117:0/2815293436 shutdown_connections 2024-03-19T03:33:21.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.852+0000 7f7602c00640 1 --2- 172.21.15.117:0/2815293436 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f75e00a2790 0x7f75e00a4c50 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.852+0000 7f7602c00640 1 --2- 172.21.15.117:0/2815293436 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75fc107560 0x7f75fc19f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.852+0000 7f7602c00640 1 --2- 172.21.15.117:0/2815293436 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75fc104c30 0x7f75fc198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.852+0000 7f7602c00640 1 --2- 172.21.15.117:0/2815293436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75fc102300 0x7f75fc197ae0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:21.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.852+0000 7f7602c00640 1 -- 172.21.15.117:0/2815293436 >> 172.21.15.117:0/2815293436 conn(0x7f75fc0fbc90 msgr2=0x7f75fc102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:21.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.852+0000 7f7602c00640 1 -- 172.21.15.117:0/2815293436 shutdown_connections 2024-03-19T03:33:21.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:21.853+0000 7f7602c00640 1 -- 172.21.15.117:0/2815293436 wait complete. 2024-03-19T03:33:21.869 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-19T03:33:21.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:21 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1419837190' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:33:22.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.058+0000 7f7d65800640 1 -- 172.21.15.117:0/4143138872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d60101940 msgr2=0x7f7d60101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:22.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.058+0000 7f7d65800640 1 --2- 172.21.15.117:0/4143138872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d60101940 0x7f7d60101dc0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c009900 tx=0x7f7d4c02bc40 comp rx=0 tx=0).stop 2024-03-19T03:33:22.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.059+0000 7f7d65800640 1 -- 172.21.15.117:0/4143138872 shutdown_connections 2024-03-19T03:33:22.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.059+0000 7f7d65800640 1 --2- 172.21.15.117:0/4143138872 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7d60102300 0x7f7d60108b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.059+0000 7f7d65800640 1 --2- 172.21.15.117:0/4143138872 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d60101940 0x7f7d60101dc0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.059 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.059+0000 7f7d65800640 1 --2- 172.21.15.117:0/4143138872 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d60100750 0x7f7d60100b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.059+0000 7f7d65800640 1 -- 172.21.15.117:0/4143138872 >> 172.21.15.117:0/4143138872 conn(0x7f7d600fbd00 msgr2=0x7f7d600fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:22.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.059+0000 7f7d65800640 1 -- 172.21.15.117:0/4143138872 shutdown_connections 2024-03-19T03:33:22.060 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.059+0000 7f7d65800640 1 -- 172.21.15.117:0/4143138872 wait complete. 2024-03-19T03:33:22.061 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.060+0000 7f7d65800640 1 Processor -- start 2024-03-19T03:33:22.061 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.060+0000 7f7d65800640 1 -- start start 2024-03-19T03:33:22.061 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.061+0000 7f7d65800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7d60100750 0x7f7d60197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:22.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.061+0000 7f7d65800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d60101940 0x7f7d60198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:22.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.061+0000 7f7d65800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d60102300 0x7f7d6019f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:22.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.061+0000 7f7d65800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6010d550 con 0x7f7d60101940 2024-03-19T03:33:22.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.061+0000 7f7d65800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6010d3d0 con 0x7f7d60102300 2024-03-19T03:33:22.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.061+0000 7f7d65800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6010d6d0 con 0x7f7d60100750 2024-03-19T03:33:22.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.061+0000 7f7d5f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d60102300 0x7f7d6019f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:22.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.061+0000 7f7d5f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d60102300 0x7f7d6019f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52904/0 (socket says 172.21.15.117:52904) 2024-03-19T03:33:22.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.061+0000 7f7d5ea00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7d60100750 0x7f7d60197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:22.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.061+0000 7f7d5f400640 1 -- 172.21.15.117:0/2836912495 learned_addr learned my addr 172.21.15.117:0/2836912495 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:22.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.061+0000 7f7d5e000640 1 --2- 172.21.15.117:0/2836912495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d60101940 0x7f7d60198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:22.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.062+0000 7f7d5f400640 1 -- 172.21.15.117:0/2836912495 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7d60100750 msgr2=0x7f7d60197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:22.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.062+0000 7f7d5f400640 1 --2- 172.21.15.117:0/2836912495 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7d60100750 0x7f7d60197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.062 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.062+0000 7f7d5f400640 1 -- 172.21.15.117:0/2836912495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d60101940 msgr2=0x7f7d60198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:22.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.062+0000 7f7d5f400640 1 --2- 172.21.15.117:0/2836912495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d60101940 0x7f7d60198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.062+0000 7f7d5f400640 1 -- 172.21.15.117:0/2836912495 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d6019f770 con 0x7f7d60102300 2024-03-19T03:33:22.063 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.062+0000 7f7d5f400640 1 --2- 172.21.15.117:0/2836912495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d60102300 0x7f7d6019f100 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7d58009fd0 tx=0x7f7d5800b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:22.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.062+0000 7f7d57400640 1 -- 172.21.15.117:0/2836912495 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d58017070 con 0x7f7d60102300 2024-03-19T03:33:22.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.062+0000 7f7d57400640 1 -- 172.21.15.117:0/2836912495 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d58003e50 con 0x7f7d60102300 2024-03-19T03:33:22.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.062+0000 7f7d65800640 1 -- 172.21.15.117:0/2836912495 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d6019fa60 con 0x7f7d60102300 2024-03-19T03:33:22.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.062+0000 7f7d57400640 1 -- 172.21.15.117:0/2836912495 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d580106c0 con 0x7f7d60102300 2024-03-19T03:33:22.064 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.062+0000 7f7d65800640 1 -- 172.21.15.117:0/2836912495 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d6019ff80 con 0x7f7d60102300 2024-03-19T03:33:22.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.065+0000 7f7d685af740 1 Processor -- start 2024-03-19T03:33:22.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.065+0000 7f7d57400640 1 -- 172.21.15.117:0/2836912495 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f7d58003970 con 0x7f7d60102300 2024-03-19T03:33:22.065 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.066+0000 7f7d685af740 1 -- start start 2024-03-19T03:33:22.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.066+0000 7f7d685af740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563a6e4a33f0 0x563a6e4a3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:22.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.066+0000 7f7d685af740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563a6e4a3de0 0x563a6e4a88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:22.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.066+0000 7f7d685af740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a8f50 0x563a6e4ab3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:22.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.066+0000 7f7d685af740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563a6e407d00 con 0x563a6e4a3de0 2024-03-19T03:33:22.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.066+0000 7f7d685af740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x563a6e3b0b00 con 0x563a6e4a8f50 2024-03-19T03:33:22.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.066+0000 7f7d685af740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x563a6e403da0 con 0x563a6e4a33f0 2024-03-19T03:33:22.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.066+0000 7f7d5ea00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563a6e4a3de0 0x563a6e4a88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:22.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.066+0000 7f7d5ea00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563a6e4a3de0 0x563a6e4a88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:42488/0 (socket says 172.21.15.117:42488) 2024-03-19T03:33:22.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.066+0000 7f7d5ea00640 1 -- 172.21.15.117:0/616310881 learned_addr learned my addr 172.21.15.117:0/616310881 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:22.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.066+0000 7f7d5e000640 1 --2- 172.21.15.117:0/616310881 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a8f50 0x563a6e4ab3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:22.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.067+0000 7f7d5e000640 1 --2- 172.21.15.117:0/616310881 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563a6e4a33f0 0x563a6e4a3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:22.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.067+0000 7f7d5ea00640 1 -- 172.21.15.117:0/616310881 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563a6e4a33f0 msgr2=0x563a6e4a3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:22.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.067+0000 7f7d5ea00640 1 --2- 172.21.15.117:0/616310881 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563a6e4a33f0 0x563a6e4a3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.067+0000 7f7d5ea00640 1 -- 172.21.15.117:0/616310881 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a8f50 msgr2=0x563a6e4ab3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:22.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.067+0000 7f7d5ea00640 1 --2- 172.21.15.117:0/616310881 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a8f50 0x563a6e4ab3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.067+0000 7f7d5ea00640 1 -- 172.21.15.117:0/616310881 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a6e11c2d0 con 0x563a6e4a3de0 2024-03-19T03:33:22.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.067+0000 7f7d5e000640 1 --2- 172.21.15.117:0/616310881 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563a6e4a33f0 0x563a6e4a3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:33:22.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.067+0000 7f7d5ea00640 1 --2- 172.21.15.117:0/616310881 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563a6e4a3de0 0x563a6e4a88e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7d5000bcb0 tx=0x7f7d500107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19c4e94336495419 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:22.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.067+0000 7f7d57400640 1 --2- 172.21.15.117:0/2836912495 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7d440a2880 0x7f7d440a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:22.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.067+0000 7f7d4be00640 1 -- 172.21.15.117:0/616310881 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d50004020 con 0x563a6e4a3de0 2024-03-19T03:33:22.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.067+0000 7f7d4be00640 1 -- 172.21.15.117:0/616310881 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d50005500 con 0x563a6e4a3de0 2024-03-19T03:33:22.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.067+0000 7f7d57400640 1 -- 172.21.15.117:0/2836912495 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7d580ceb30 con 0x7f7d60102300 2024-03-19T03:33:22.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.068+0000 7f7d4be00640 1 -- 172.21.15.117:0/616310881 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d50004df0 con 0x563a6e4a3de0 2024-03-19T03:33:22.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.068+0000 7f7d5e000640 1 --2- 172.21.15.117:0/2836912495 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7d440a2880 0x7f7d440a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:22.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.068+0000 7f7d685af740 1 -- 172.21.15.117:0/616310881 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563a6e4a3de0 msgr2=0x563a6e4a88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:22.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.068+0000 7f7d5e000640 1 --2- 172.21.15.117:0/2836912495 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7d440a2880 0x7f7d440a4d40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c02efd0 tx=0x7f7d4c02cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:22.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.068+0000 7f7d685af740 1 --2- 172.21.15.117:0/616310881 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563a6e4a3de0 0x563a6e4a88e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7d5000bcb0 tx=0x7f7d500107a0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.068+0000 7f7d685af740 1 -- 172.21.15.117:0/616310881 shutdown_connections 2024-03-19T03:33:22.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.068+0000 7f7d685af740 1 --2- 172.21.15.117:0/616310881 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a8f50 0x563a6e4ab3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.068+0000 7f7d685af740 1 --2- 172.21.15.117:0/616310881 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563a6e4a3de0 0x563a6e4a88e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.068+0000 7f7d685af740 1 --2- 172.21.15.117:0/616310881 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563a6e4a33f0 0x563a6e4a3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.071 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.068+0000 7f7d685af740 1 -- 172.21.15.117:0/616310881 >> 172.21.15.117:0/616310881 conn(0x563a6e49e960 msgr2=0x563a6e49ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:22.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.069+0000 7f7d685af740 1 -- 172.21.15.117:0/616310881 shutdown_connections 2024-03-19T03:33:22.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.069+0000 7f7d685af740 1 -- 172.21.15.117:0/616310881 wait complete. 2024-03-19T03:33:22.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.069+0000 7f7d685af740 1 -- start start 2024-03-19T03:33:22.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.070+0000 7f7d685af740 1 Processor -- start 2024-03-19T03:33:22.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.070+0000 7f7d685af740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563a6e4a33f0 0x563a6e4ad900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:22.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.070+0000 7f7d685af740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a3de0 0x563a6e4ade40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:22.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.070+0000 7f7d685af740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563a6e4a8f50 0x563a6e4b4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:22.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.070+0000 7f7d685af740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563a6e4abf00 con 0x563a6e4a8f50 2024-03-19T03:33:22.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.070+0000 7f7d685af740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x563a6e3c3ed0 con 0x563a6e4a3de0 2024-03-19T03:33:22.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.070+0000 7f7d685af740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x563a6e4ac080 con 0x563a6e4a33f0 2024-03-19T03:33:22.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.070+0000 7f7d685af740 10 client.0 _ll_register_callbacks cb 0x563a6e3c1c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:33:22.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.071+0000 7f7d5e000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a3de0 0x563a6e4ade40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:22.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.071+0000 7f7d5f400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563a6e4a8f50 0x563a6e4b4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:22.072 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.071+0000 7f7d5e000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a3de0 0x563a6e4ade40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52924/0 (socket says 172.21.15.117:52924) 2024-03-19T03:33:22.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.071+0000 7f7d5e000640 1 -- 172.21.15.117:0/21178232 learned_addr learned my addr 172.21.15.117:0/21178232 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:22.073 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.071+0000 7f7d5ea00640 1 --2- 172.21.15.117:0/21178232 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563a6e4a33f0 0x563a6e4ad900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:22.075 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.071+0000 7f7d5e000640 1 -- 172.21.15.117:0/21178232 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563a6e4a33f0 msgr2=0x563a6e4ad900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:22.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.071+0000 7f7d5e000640 1 --2- 172.21.15.117:0/21178232 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563a6e4a33f0 0x563a6e4ad900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.071+0000 7f7d5e000640 1 -- 172.21.15.117:0/21178232 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563a6e4a8f50 msgr2=0x563a6e4b4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:22.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.071+0000 7f7d5e000640 1 --2- 172.21.15.117:0/21178232 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563a6e4a8f50 0x563a6e4b4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.071+0000 7f7d5e000640 1 -- 172.21.15.117:0/21178232 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a6e1e66b0 con 0x563a6e4a3de0 2024-03-19T03:33:22.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.071+0000 7f7d5e000640 1 --2- 172.21.15.117:0/21178232 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a3de0 0x563a6e4ade40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c006010 tx=0x7f7d4c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23d2b93eafb52f9c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:22.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.071+0000 7f7d48c00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-19T03:33:22.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.072+0000 7f7d48c00640 1 -- 172.21.15.117:0/21178232 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c052070 con 0x563a6e4a3de0 2024-03-19T03:33:22.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.072+0000 7f7d48c00640 1 -- 172.21.15.117:0/21178232 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7d4c0050f0 con 0x563a6e4a3de0 2024-03-19T03:33:22.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.072+0000 7f7d685af740 1 -- 172.21.15.117:0/21178232 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d50005500 con 0x563a6e4a3de0 2024-03-19T03:33:22.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.072+0000 7f7d48c00640 1 -- 172.21.15.117:0/21178232 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7d4c05e050 con 0x563a6e4a3de0 2024-03-19T03:33:22.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.072+0000 7f7d685af740 10 client.25448 fetch_fsmap learned FSMap version 28 2024-03-19T03:33:22.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.072+0000 7f7d685af740 1 -- 172.21.15.117:0/21178232 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563a6e4b8260 con 0x563a6e4a3de0 2024-03-19T03:33:22.076 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.072+0000 7f7d48c00640 1 -- 172.21.15.117:0/21178232 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d50005500 con 0x563a6e4a3de0 2024-03-19T03:33:22.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.072+0000 7f7d48c00640 1 -- 172.21.15.117:0/21178232 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7d4c04b050 con 0x563a6e4a3de0 2024-03-19T03:33:22.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.073+0000 7f7d685af740 10 client.25448 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T03:33:22.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.073+0000 7f7d685af740 10 client.25448 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-19T03:33:22.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.073+0000 7f7d685af740 20 client.25448 populate_metadata read hostname 'smithi117' 2024-03-19T03:33:22.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.073+0000 7f7d685af740 1 --2- 172.21.15.117:0/21178232 >> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] conn(0x563a6e4ba890 0x563a6e4bbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:22.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.073+0000 7f7d685af740 4 client.25448 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T03:33:22.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.073+0000 7f7d685af740 1 -- 172.21.15.117:0/21178232 --> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a6e415240 con 0x563a6e4ba890 2024-03-19T03:33:22.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.073+0000 7f7d5ea00640 1 --2- 172.21.15.117:0/21178232 >> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] conn(0x563a6e4ba890 0x563a6e4bbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:22.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.073+0000 7f7d5ea00640 1 --2- 172.21.15.117:0/21178232 >> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] conn(0x563a6e4ba890 0x563a6e4bbd00 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7b813e34d65f3e7 server_cookie=296f1fffd77fac27 in_seq=0 out_seq=0 2024-03-19T03:33:22.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.073+0000 7f7d48c00640 10 client.25448 ms_handle_connect on v2:172.21.15.119:6838/2383354833 2024-03-19T03:33:22.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.074+0000 7f7d48c00640 1 -- 172.21.15.117:0/21178232 <== mds.0 v2:172.21.15.119:6838/2383354833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563a6e415240 con 0x563a6e4ba890 2024-03-19T03:33:22.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.074+0000 7f7d48c00640 10 client.25448 handle_command_reply: tid=0 2024-03-19T03:33:22.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.074+0000 7f7d685af740 1 client.25448 shutdown 2024-03-19T03:33:22.077 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.074+0000 7f7d685af740 1 -- 172.21.15.117:0/21178232 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a3de0 msgr2=0x563a6e4ade40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:22.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.074+0000 7f7d685af740 1 --2- 172.21.15.117:0/21178232 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a3de0 0x563a6e4ade40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c006010 tx=0x7f7d4c009260 comp rx=0 tx=0).stop 2024-03-19T03:33:22.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.075+0000 7f7d685af740 1 -- 172.21.15.117:0/21178232 shutdown_connections 2024-03-19T03:33:22.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.075+0000 7f7d685af740 1 --2- 172.21.15.117:0/21178232 >> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] conn(0x563a6e4ba890 0x563a6e4bbd00 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.075+0000 7f7d685af740 1 --2- 172.21.15.117:0/21178232 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563a6e4a8f50 0x563a6e4b4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.075+0000 7f7d685af740 1 --2- 172.21.15.117:0/21178232 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a3de0 0x563a6e4ade40 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.075+0000 7f7d685af740 1 --2- 172.21.15.117:0/21178232 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563a6e4a33f0 0x563a6e4ad900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.075+0000 7f7d48c00640 0 client.25448 ms_handle_reset on v2:172.21.15.119:6838/2383354833 2024-03-19T03:33:22.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.075+0000 7f7d685af740 1 -- 172.21.15.117:0/21178232 >> 172.21.15.117:0/21178232 conn(0x563a6e49e960 msgr2=0x563a6e49daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:22.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.075+0000 7f7d685af740 1 -- 172.21.15.117:0/21178232 shutdown_connections 2024-03-19T03:33:22.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.075+0000 7f7d685af740 1 -- 172.21.15.117:0/21178232 wait complete. 2024-03-19T03:33:22.078 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.075+0000 7f7d685af740 20 client.25448 trim_cache size 0 max 16384 2024-03-19T03:33:22.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.112+0000 7f7d685af740 1 Processor -- start 2024-03-19T03:33:22.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.112+0000 7f7d685af740 1 -- start start 2024-03-19T03:33:22.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.112+0000 7f7d685af740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a33f0 0x563a6e49d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:22.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.112+0000 7f7d685af740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563a6e4a3de0 0x563a6e4b4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:22.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.112+0000 7f7d685af740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563a6e4a8f50 0x563a6e4acbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:22.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.112+0000 7f7d685af740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563a6e4ba040 con 0x563a6e4a8f50 2024-03-19T03:33:22.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.112+0000 7f7d685af740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x563a6e4ba710 con 0x563a6e4a33f0 2024-03-19T03:33:22.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.112+0000 7f7d685af740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x563a6e4bade0 con 0x563a6e4a3de0 2024-03-19T03:33:22.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.113+0000 7f7d5ea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a33f0 0x563a6e49d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:22.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.113+0000 7f7d5ea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a33f0 0x563a6e49d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52936/0 (socket says 172.21.15.117:52936) 2024-03-19T03:33:22.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.113+0000 7f7d5ea00640 1 -- 172.21.15.117:0/125146420 learned_addr learned my addr 172.21.15.117:0/125146420 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:22.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.113+0000 7f7d5e000640 1 --2- 172.21.15.117:0/125146420 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563a6e4a3de0 0x563a6e4b4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:22.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.113+0000 7f7d5f400640 1 --2- 172.21.15.117:0/125146420 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563a6e4a8f50 0x563a6e4acbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:22.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.113+0000 7f7d5ea00640 1 -- 172.21.15.117:0/125146420 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563a6e4a3de0 msgr2=0x563a6e4b4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:22.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.113+0000 7f7d5ea00640 1 --2- 172.21.15.117:0/125146420 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563a6e4a3de0 0x563a6e4b4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.113+0000 7f7d5ea00640 1 -- 172.21.15.117:0/125146420 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563a6e4a8f50 msgr2=0x563a6e4acbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:22.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.113+0000 7f7d5ea00640 1 --2- 172.21.15.117:0/125146420 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563a6e4a8f50 0x563a6e4acbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.113+0000 7f7d5ea00640 1 -- 172.21.15.117:0/125146420 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a6e4aea80 con 0x563a6e4a33f0 2024-03-19T03:33:22.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.113+0000 7f7d5e000640 1 --2- 172.21.15.117:0/125146420 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563a6e4a3de0 0x563a6e4b4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:33:22.114 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.113+0000 7f7d5ea00640 1 --2- 172.21.15.117:0/125146420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a33f0 0x563a6e49d920 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f7d5000ba40 tx=0x7f7d5000b390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a69cad905637389 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:22.118 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.113+0000 7f7d4b400640 1 -- 172.21.15.117:0/125146420 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d50005220 con 0x563a6e4a33f0 2024-03-19T03:33:22.118 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.113+0000 7f7d4b400640 1 -- 172.21.15.117:0/125146420 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d5000be60 con 0x563a6e4a33f0 2024-03-19T03:33:22.118 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.113+0000 7f7d4b400640 1 -- 172.21.15.117:0/125146420 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d50003850 con 0x563a6e4a33f0 2024-03-19T03:33:22.118 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.114+0000 7f7d685af740 1 -- 172.21.15.117:0/125146420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a33f0 msgr2=0x563a6e49d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:22.118 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.114+0000 7f7d685af740 1 --2- 172.21.15.117:0/125146420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a33f0 0x563a6e49d920 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f7d5000ba40 tx=0x7f7d5000b390 comp rx=0 tx=0).stop 2024-03-19T03:33:22.118 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.114+0000 7f7d685af740 1 -- 172.21.15.117:0/125146420 shutdown_connections 2024-03-19T03:33:22.118 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.114+0000 7f7d685af740 1 --2- 172.21.15.117:0/125146420 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563a6e4a8f50 0x563a6e4acbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.118 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.114+0000 7f7d685af740 1 --2- 172.21.15.117:0/125146420 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563a6e4a3de0 0x563a6e4b4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.118 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.114+0000 7f7d685af740 1 --2- 172.21.15.117:0/125146420 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a33f0 0x563a6e49d920 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.118 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.114+0000 7f7d685af740 1 -- 172.21.15.117:0/125146420 >> 172.21.15.117:0/125146420 conn(0x563a6e49e960 msgr2=0x563a6e49d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:22.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.114+0000 7f7d685af740 1 -- 172.21.15.117:0/125146420 shutdown_connections 2024-03-19T03:33:22.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.114+0000 7f7d685af740 1 -- 172.21.15.117:0/125146420 wait complete. 2024-03-19T03:33:22.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.115+0000 7f7d685af740 1 -- start start 2024-03-19T03:33:22.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.115+0000 7f7d685af740 1 Processor -- start 2024-03-19T03:33:22.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.115+0000 7f7d685af740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563a6e4a33f0 0x563a6e4b0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:22.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.115+0000 7f7d685af740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563a6e4a3de0 0x563a6e4b1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:22.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.116+0000 7f7d685af740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a8f50 0x563a6e4b4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:22.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.116+0000 7f7d685af740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563a6e4bbb80 con 0x563a6e4a3de0 2024-03-19T03:33:22.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.116+0000 7f7d685af740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x563a6e4bb4b0 con 0x563a6e4a8f50 2024-03-19T03:33:22.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.116+0000 7f7d685af740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x563a6e4bc250 con 0x563a6e4a33f0 2024-03-19T03:33:22.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.116+0000 7f7d685af740 10 client.0 _ll_register_callbacks cb 0x7f7d4c005bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T03:33:22.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.116+0000 7f7d5f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a8f50 0x563a6e4b4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:22.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.116+0000 7f7d5f400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a8f50 0x563a6e4b4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:52950/0 (socket says 172.21.15.117:52950) 2024-03-19T03:33:22.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.116+0000 7f7d5f400640 1 -- 172.21.15.117:0/488828139 learned_addr learned my addr 172.21.15.117:0/488828139 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:22.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.116+0000 7f7d5e000640 1 --2- 172.21.15.117:0/488828139 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563a6e4a3de0 0x563a6e4b1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:22.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.116+0000 7f7d5ea00640 1 --2- 172.21.15.117:0/488828139 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563a6e4a33f0 0x563a6e4b0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:22.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.116+0000 7f7d5f400640 1 -- 172.21.15.117:0/488828139 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563a6e4a33f0 msgr2=0x563a6e4b0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:22.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.116+0000 7f7d5f400640 1 --2- 172.21.15.117:0/488828139 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563a6e4a33f0 0x563a6e4b0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.116+0000 7f7d5f400640 1 -- 172.21.15.117:0/488828139 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563a6e4a3de0 msgr2=0x563a6e4b1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T03:33:22.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.116+0000 7f7d5f400640 1 --2- 172.21.15.117:0/488828139 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563a6e4a3de0 0x563a6e4b1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.116+0000 7f7d5f400640 1 -- 172.21.15.117:0/488828139 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a6e4b4b40 con 0x563a6e4a8f50 2024-03-19T03:33:22.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.116+0000 7f7d5ea00640 1 --2- 172.21.15.117:0/488828139 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563a6e4a33f0 0x563a6e4b0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:33:22.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.116+0000 7f7d5f400640 1 --2- 172.21.15.117:0/488828139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a8f50 0x563a6e4b4550 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7d5809b0c0 tx=0x7f7d5809aa90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10d1fa59cfcebeec server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:22.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.116+0000 7f7d48c00640 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-03-19T03:33:22.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.117+0000 7f7d48c00640 1 -- 172.21.15.117:0/488828139 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d58020040 con 0x563a6e4a8f50 2024-03-19T03:33:22.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.117+0000 7f7d48c00640 1 -- 172.21.15.117:0/488828139 <== mon.2 v2:172.21.15.119:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f7d580977a0 con 0x563a6e4a8f50 2024-03-19T03:33:22.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.117+0000 7f7d685af740 1 -- 172.21.15.117:0/488828139 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d5000be60 con 0x563a6e4a8f50 2024-03-19T03:33:22.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.117+0000 7f7d48c00640 1 -- 172.21.15.117:0/488828139 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7d580a3820 con 0x563a6e4a8f50 2024-03-19T03:33:22.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.117+0000 7f7d685af740 10 client.25454 fetch_fsmap learned FSMap version 28 2024-03-19T03:33:22.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.117+0000 7f7d685af740 1 -- 172.21.15.117:0/488828139 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563a6e521080 con 0x563a6e4a8f50 2024-03-19T03:33:22.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.117+0000 7f7d48c00640 1 -- 172.21.15.117:0/488828139 <== mon.2 v2:172.21.15.119:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d5000be60 con 0x563a6e4a8f50 2024-03-19T03:33:22.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.117+0000 7f7d48c00640 1 -- 172.21.15.117:0/488828139 <== mon.2 v2:172.21.15.119:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7d5809a580 con 0x563a6e4a8f50 2024-03-19T03:33:22.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.117+0000 7f7d685af740 10 client.25454 fetch_fsmap finished waiting for FSMap version 28 2024-03-19T03:33:22.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.117+0000 7f7d685af740 10 client.25454 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-19T03:33:22.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.117+0000 7f7d685af740 20 client.25454 populate_metadata read hostname 'smithi117' 2024-03-19T03:33:22.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.117+0000 7f7d685af740 1 --2- 172.21.15.117:0/488828139 >> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] conn(0x563a6e525cc0 0x563a6e522100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:22.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.117+0000 7f7d685af740 4 client.25454 mds_command: new command op to 24599 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T03:33:22.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.117+0000 7f7d685af740 1 -- 172.21.15.117:0/488828139 --> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] -- command(tid 0: {"prefix": "status"}) -- 0x563a6e40f590 con 0x563a6e525cc0 2024-03-19T03:33:22.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.118+0000 7f7d5ea00640 1 --2- 172.21.15.117:0/488828139 >> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] conn(0x563a6e525cc0 0x563a6e522100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:22.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.118+0000 7f7d5ea00640 1 --2- 172.21.15.117:0/488828139 >> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] conn(0x563a6e525cc0 0x563a6e522100 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=475feec038e8934f server_cookie=527a3cafe464f613 in_seq=0 out_seq=0 2024-03-19T03:33:22.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.118+0000 7f7d48c00640 10 client.25454 ms_handle_connect on v2:172.21.15.119:6838/2383354833 2024-03-19T03:33:22.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.118+0000 7f7d48c00640 1 -- 172.21.15.117:0/488828139 <== mds.0 v2:172.21.15.119:6838/2383354833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x563a6e40f590 con 0x563a6e525cc0 2024-03-19T03:33:22.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.118+0000 7f7d48c00640 10 client.25454 handle_command_reply: tid=0 2024-03-19T03:33:22.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.118+0000 7f7d685af740 1 client.25454 shutdown 2024-03-19T03:33:22.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.119+0000 7f7d685af740 1 -- 172.21.15.117:0/488828139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a8f50 msgr2=0x563a6e4b4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:22.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.119+0000 7f7d685af740 1 --2- 172.21.15.117:0/488828139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a8f50 0x563a6e4b4550 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7d5809b0c0 tx=0x7f7d5809aa90 comp rx=0 tx=0).stop 2024-03-19T03:33:22.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.119+0000 7f7d685af740 1 -- 172.21.15.117:0/488828139 shutdown_connections 2024-03-19T03:33:22.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.119+0000 7f7d685af740 1 --2- 172.21.15.117:0/488828139 >> [v2:172.21.15.119:6838/2383354833,v1:172.21.15.119:6839/2383354833] conn(0x563a6e525cc0 0x563a6e522100 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.119+0000 7f7d685af740 1 --2- 172.21.15.117:0/488828139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x563a6e4a8f50 0x563a6e4b4550 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.119+0000 7f7d685af740 1 --2- 172.21.15.117:0/488828139 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563a6e4a3de0 0x563a6e4b1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.119+0000 7f7d48c00640 0 client.25454 ms_handle_reset on v2:172.21.15.119:6838/2383354833 2024-03-19T03:33:22.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.119+0000 7f7d685af740 1 --2- 172.21.15.117:0/488828139 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563a6e4a33f0 0x563a6e4b0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.119+0000 7f7d685af740 1 -- 172.21.15.117:0/488828139 >> 172.21.15.117:0/488828139 conn(0x563a6e49e960 msgr2=0x563a6e49daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:22.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.119+0000 7f7d685af740 1 -- 172.21.15.117:0/488828139 shutdown_connections 2024-03-19T03:33:22.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.119+0000 7f7d685af740 1 -- 172.21.15.117:0/488828139 wait complete. 2024-03-19T03:33:22.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.119+0000 7f7d685af740 20 client.25454 trim_cache size 0 max 16384 2024-03-19T03:33:22.124 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-19T03:33:22.124 INFO:teuthology.orchestra.run.smithi117.stdout: "cluster_fsid": "a3a82456-e59f-11ee-95c9-87774f69a715", 2024-03-19T03:33:22.124 INFO:teuthology.orchestra.run.smithi117.stdout: "whoami": 0, 2024-03-19T03:33:22.124 INFO:teuthology.orchestra.run.smithi117.stdout: "id": 24599, 2024-03-19T03:33:22.124 INFO:teuthology.orchestra.run.smithi117.stdout: "want_state": "up:active", 2024-03-19T03:33:22.124 INFO:teuthology.orchestra.run.smithi117.stdout: "state": "up:active", 2024-03-19T03:33:22.124 INFO:teuthology.orchestra.run.smithi117.stdout: "fs_name": "cephfs", 2024-03-19T03:33:22.124 INFO:teuthology.orchestra.run.smithi117.stdout: "rank_uptime": 166.450882245, 2024-03-19T03:33:22.124 INFO:teuthology.orchestra.run.smithi117.stdout: "mdsmap_epoch": 28, 2024-03-19T03:33:22.124 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch": 93, 2024-03-19T03:33:22.125 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_epoch_barrier": 89, 2024-03-19T03:33:22.125 INFO:teuthology.orchestra.run.smithi117.stdout: "uptime": 173.79468171299999 2024-03-19T03:33:22.125 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-19T03:33:22.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.120+0000 7f7d65800640 1 -- 172.21.15.117:0/2836912495 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7d440a2880 msgr2=0x7f7d440a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:22.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.120+0000 7f7d65800640 1 --2- 172.21.15.117:0/2836912495 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7d440a2880 0x7f7d440a4d40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c02efd0 tx=0x7f7d4c02cf90 comp rx=0 tx=0).stop 2024-03-19T03:33:22.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.120+0000 7f7d65800640 1 -- 172.21.15.117:0/2836912495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d60102300 msgr2=0x7f7d6019f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:22.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.120+0000 7f7d65800640 1 --2- 172.21.15.117:0/2836912495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d60102300 0x7f7d6019f100 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7d58009fd0 tx=0x7f7d5800b260 comp rx=0 tx=0).stop 2024-03-19T03:33:22.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.120+0000 7f7d65800640 1 -- 172.21.15.117:0/2836912495 shutdown_connections 2024-03-19T03:33:22.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.120+0000 7f7d65800640 1 --2- 172.21.15.117:0/2836912495 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f7d440a2880 0x7f7d440a4d40 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.120+0000 7f7d65800640 1 --2- 172.21.15.117:0/2836912495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d60102300 0x7f7d6019f100 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.120+0000 7f7d65800640 1 --2- 172.21.15.117:0/2836912495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d60101940 0x7f7d60198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.120+0000 7f7d65800640 1 --2- 172.21.15.117:0/2836912495 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7d60100750 0x7f7d60197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:22.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.120+0000 7f7d65800640 1 -- 172.21.15.117:0/2836912495 >> 172.21.15.117:0/2836912495 conn(0x7f7d600fbd00 msgr2=0x7f7d60104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:22.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.120+0000 7f7d65800640 1 -- 172.21.15.117:0/2836912495 shutdown_connections 2024-03-19T03:33:22.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:22.120+0000 7f7d65800640 1 -- 172.21.15.117:0/2836912495 wait complete. 2024-03-19T03:33:22.137 INFO:teuthology.run_tasks:Running task kclient... 2024-03-19T03:33:22.151 INFO:tasks.kclient:Mounting kernel clients... 2024-03-19T03:33:22.151 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-19T03:33:22.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:33:22.151 DEBUG:teuthology.orchestra.run.smithi117:> ip netns list 2024-03-19T03:33:22.197 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-19T03:33:22.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:33:22.198 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-19T03:33:22.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:33:22.324 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip link delete ceph-brx 2024-03-19T03:33:22.450 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-03-19T03:33:22.450 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-19T03:33:22.450 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-19T03:33:22.451 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-19T03:33:22.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:22 smithi117 ceph-mon[31838]: pgmap v405: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T03:33:22.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:22 smithi178 ceph-mon[67578]: pgmap v405: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T03:33:22.901 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:33:22.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:22 smithi119 ceph-mon[66861]: pgmap v405: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T03:33:25.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.242+0000 7fc280200700 1 -- 172.21.15.117:0/1949004044 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc278107630 msgr2=0x7fc27810f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:25.243 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.242+0000 7fc280200700 1 --2- 172.21.15.117:0/1949004044 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc278107630 0x7fc27810f200 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc27000b0a0 tx=0x7fc27000b3b0 comp rx=0 tx=0).stop 2024-03-19T03:33:25.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.242+0000 7fc280200700 1 -- 172.21.15.117:0/1949004044 shutdown_connections 2024-03-19T03:33:25.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.242+0000 7fc280200700 1 --2- 172.21.15.117:0/1949004044 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc278107630 0x7fc27810f200 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:25.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.242+0000 7fc280200700 1 --2- 172.21.15.117:0/1949004044 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc278102550 0x7fc278102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:25.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.242+0000 7fc280200700 1 --2- 172.21.15.117:0/1949004044 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc278102f00 0x7fc278106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:25.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.242+0000 7fc280200700 1 -- 172.21.15.117:0/1949004044 >> 172.21.15.117:0/1949004044 conn(0x7fc2780fddc0 msgr2=0x7fc2781001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:25.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.243+0000 7fc280200700 1 -- 172.21.15.117:0/1949004044 shutdown_connections 2024-03-19T03:33:25.244 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.243+0000 7fc280200700 1 -- 172.21.15.117:0/1949004044 wait complete. 2024-03-19T03:33:25.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.244+0000 7fc280200700 1 Processor -- start 2024-03-19T03:33:25.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.244+0000 7fc280200700 1 -- start start 2024-03-19T03:33:25.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.245+0000 7fc280200700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc278102550 0x7fc27819f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:25.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.245+0000 7fc280200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc278102f00 0x7fc27819fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:25.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.245+0000 7fc280200700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc278107630 0x7fc27819c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:25.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.245+0000 7fc280200700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc278112640 con 0x7fc278102f00 2024-03-19T03:33:25.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.245+0000 7fc280200700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fc2781124c0 con 0x7fc278102550 2024-03-19T03:33:25.245 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.245+0000 7fc280200700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fc2781127c0 con 0x7fc278107630 2024-03-19T03:33:25.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.245+0000 7fc27e200700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc278107630 0x7fc27819c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:25.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.245+0000 7fc27e200700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc278107630 0x7fc27819c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:45148/0 (socket says 172.21.15.117:45148) 2024-03-19T03:33:25.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.245+0000 7fc27e200700 1 -- 172.21.15.117:0/2774469180 learned_addr learned my addr 172.21.15.117:0/2774469180 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:25.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.245+0000 7fc27d800700 1 --2- 172.21.15.117:0/2774469180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc278102550 0x7fc27819f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:25.246 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.245+0000 7fc27ce00700 1 --2- 172.21.15.117:0/2774469180 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc278102f00 0x7fc27819fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:25.247 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.246+0000 7fc27e200700 1 -- 172.21.15.117:0/2774469180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc278102550 msgr2=0x7fc27819f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:25.247 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.246+0000 7fc27e200700 1 --2- 172.21.15.117:0/2774469180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc278102550 0x7fc27819f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:25.247 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.246+0000 7fc27e200700 1 -- 172.21.15.117:0/2774469180 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc278102f00 msgr2=0x7fc27819fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:25.247 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.246+0000 7fc27e200700 1 --2- 172.21.15.117:0/2774469180 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc278102f00 0x7fc27819fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:25.247 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.246+0000 7fc27e200700 1 -- 172.21.15.117:0/2774469180 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc27819cbe0 con 0x7fc278107630 2024-03-19T03:33:25.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.247+0000 7fc27e200700 1 --2- 172.21.15.117:0/2774469180 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc278107630 0x7fc27819c2e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc270006070 tx=0x7fc270004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:25.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.247+0000 7fc276000700 1 -- 172.21.15.117:0/2774469180 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc270021070 con 0x7fc278107630 2024-03-19T03:33:25.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.247+0000 7fc280200700 1 -- 172.21.15.117:0/2774469180 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc27819cdb0 con 0x7fc278107630 2024-03-19T03:33:25.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.247+0000 7fc276000700 1 -- 172.21.15.117:0/2774469180 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc270003c00 con 0x7fc278107630 2024-03-19T03:33:25.249 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.247+0000 7fc276000700 1 -- 172.21.15.117:0/2774469180 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc270021070 con 0x7fc278107630 2024-03-19T03:33:25.249 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.248+0000 7fc280200700 1 -- 172.21.15.117:0/2774469180 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc27810c360 con 0x7fc278107630 2024-03-19T03:33:25.249 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.249+0000 7fc26b400700 1 -- 172.21.15.117:0/2774469180 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc254005330 con 0x7fc278107630 2024-03-19T03:33:25.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.253+0000 7fc276000700 1 -- 172.21.15.117:0/2774469180 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fc270007980 con 0x7fc278107630 2024-03-19T03:33:25.255 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.255+0000 7fc276000700 1 --2- 172.21.15.117:0/2774469180 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc2600a2d00 0x7fc2600a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:25.255 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.256+0000 7fc276000700 1 -- 172.21.15.117:0/2774469180 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fc27000f020 con 0x7fc278107630 2024-03-19T03:33:25.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.256+0000 7fc27d800700 1 --2- 172.21.15.117:0/2774469180 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc2600a2d00 0x7fc2600a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:25.257 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.257+0000 7fc27d800700 1 --2- 172.21.15.117:0/2774469180 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc2600a2d00 0x7fc2600a51c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc26c009c60 tx=0x7fc26c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:25.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.258+0000 7fc276000700 1 -- 172.21.15.117:0/2774469180 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc27002a020 con 0x7fc278107630 2024-03-19T03:33:25.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:25 smithi119 ceph-mon[66861]: pgmap v406: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:25.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:25 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:33:25.768 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.768+0000 7fc26b400700 1 -- 172.21.15.117:0/2774469180 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc254005e30 con 0x7fc278107630 2024-03-19T03:33:25.768 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:25 smithi117 ceph-mon[31838]: pgmap v406: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:25.768 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:25 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:33:25.769 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.768+0000 7fc276000700 1 -- 172.21.15.117:0/2774469180 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) ==== 53+0+88 (secure 0 0 0) 0x7fc27002d350 con 0x7fc278107630 2024-03-19T03:33:25.769 INFO:teuthology.orchestra.run.smithi117.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-19T03:33:25.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.772+0000 7fc26b400700 1 -- 172.21.15.117:0/2774469180 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc2600a2d00 msgr2=0x7fc2600a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:25.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.772+0000 7fc26b400700 1 --2- 172.21.15.117:0/2774469180 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc2600a2d00 0x7fc2600a51c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc26c009c60 tx=0x7fc26c009260 comp rx=0 tx=0).stop 2024-03-19T03:33:25.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.772+0000 7fc26b400700 1 -- 172.21.15.117:0/2774469180 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc278107630 msgr2=0x7fc27819c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:25.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.772+0000 7fc26b400700 1 --2- 172.21.15.117:0/2774469180 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc278107630 0x7fc27819c2e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc270006070 tx=0x7fc270004d00 comp rx=0 tx=0).stop 2024-03-19T03:33:25.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.772+0000 7fc26b400700 1 -- 172.21.15.117:0/2774469180 shutdown_connections 2024-03-19T03:33:25.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.772+0000 7fc26b400700 1 --2- 172.21.15.117:0/2774469180 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fc2600a2d00 0x7fc2600a51c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:25.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.772+0000 7fc26b400700 1 --2- 172.21.15.117:0/2774469180 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc278107630 0x7fc27819c2e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:25.772 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.772+0000 7fc26b400700 1 --2- 172.21.15.117:0/2774469180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc278102550 0x7fc27819f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:25.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.772+0000 7fc26b400700 1 --2- 172.21.15.117:0/2774469180 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc278102f00 0x7fc27819fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:25.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.772+0000 7fc26b400700 1 -- 172.21.15.117:0/2774469180 >> 172.21.15.117:0/2774469180 conn(0x7fc2780fddc0 msgr2=0x7fc27810d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:25.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.773+0000 7fc26b400700 1 -- 172.21.15.117:0/2774469180 shutdown_connections 2024-03-19T03:33:25.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:25.773+0000 7fc26b400700 1 -- 172.21.15.117:0/2774469180 wait complete. 2024-03-19T03:33:25.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:25 smithi178 ceph-mon[67578]: pgmap v406: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:25.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:25 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:33:26.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:26 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:26.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:26 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:26.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:26 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:26.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:26 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:26.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:26 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2774469180' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-19T03:33:26.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:26 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:26.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:26 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:26.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:26 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:26.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:26 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:26.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:26 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:26.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:26 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:26.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:26 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2774469180' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-19T03:33:26.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:26 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:26.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:26 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:26.672 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-19T03:33:26.672 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-19T03:33:26.672 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi117.front.sepia.ceph.com 2024-03-19T03:33:26.672 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-19T03:33:26.672 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-19T03:33:26.672 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-19T03:33:26.672 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-19T03:33:26.673 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-19T03:33:26.673 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-19T03:33:26.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:33:26.673 DEBUG:teuthology.orchestra.run.smithi117:> ip addr 2024-03-19T03:33:26.715 INFO:teuthology.orchestra.run.smithi117.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-19T03:33:26.715 INFO:teuthology.orchestra.run.smithi117.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-19T03:33:26.715 INFO:teuthology.orchestra.run.smithi117.stdout: inet 127.0.0.1/8 scope host lo 2024-03-19T03:33:26.715 INFO:teuthology.orchestra.run.smithi117.stdout: valid_lft forever preferred_lft forever 2024-03-19T03:33:26.716 INFO:teuthology.orchestra.run.smithi117.stdout: inet6 ::1/128 scope host 2024-03-19T03:33:26.716 INFO:teuthology.orchestra.run.smithi117.stdout: valid_lft forever preferred_lft forever 2024-03-19T03:33:26.716 INFO:teuthology.orchestra.run.smithi117.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T03:33:26.716 INFO:teuthology.orchestra.run.smithi117.stdout: link/ether 0c:c4:7a:d8:a8:8a brd ff:ff:ff:ff:ff:ff 2024-03-19T03:33:26.716 INFO:teuthology.orchestra.run.smithi117.stdout: altname enp7s0f0 2024-03-19T03:33:26.716 INFO:teuthology.orchestra.run.smithi117.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T03:33:26.716 INFO:teuthology.orchestra.run.smithi117.stdout: link/ether 0c:c4:7a:d8:a8:8b brd ff:ff:ff:ff:ff:ff 2024-03-19T03:33:26.716 INFO:teuthology.orchestra.run.smithi117.stdout: altname enp7s0f1 2024-03-19T03:33:26.716 INFO:teuthology.orchestra.run.smithi117.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T03:33:26.716 INFO:teuthology.orchestra.run.smithi117.stdout: link/ether 0c:c4:7a:8f:44:62 brd ff:ff:ff:ff:ff:ff 2024-03-19T03:33:26.716 INFO:teuthology.orchestra.run.smithi117.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-19T03:33:26.716 INFO:teuthology.orchestra.run.smithi117.stdout: link/ether 0c:c4:7a:8f:44:63 brd ff:ff:ff:ff:ff:ff 2024-03-19T03:33:26.716 INFO:teuthology.orchestra.run.smithi117.stdout: inet 172.21.15.117/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-19T03:33:26.716 INFO:teuthology.orchestra.run.smithi117.stdout: valid_lft 40791sec preferred_lft 40791sec 2024-03-19T03:33:26.717 INFO:teuthology.orchestra.run.smithi117.stdout: inet6 fe80::ec4:7aff:fe8f:4463/64 scope link 2024-03-19T03:33:26.717 INFO:teuthology.orchestra.run.smithi117.stdout: valid_lft forever preferred_lft forever 2024-03-19T03:33:26.717 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-19T03:33:26.717 DEBUG:teuthology.orchestra.run.smithi117:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T03:33:26.717 DEBUG:teuthology.orchestra.run.smithi117:> set -e 2024-03-19T03:33:26.717 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip link add name ceph-brx type bridge 2024-03-19T03:33:26.717 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip addr flush dev ceph-brx 2024-03-19T03:33:26.717 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip link set ceph-brx up 2024-03-19T03:33:26.717 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-19T03:33:26.717 DEBUG:teuthology.orchestra.run.smithi117:> ') 2024-03-19T03:33:26.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:26 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:26.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:26 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:26.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:26 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:26.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:26 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:26.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:26 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2774469180' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-19T03:33:26.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:26 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:26.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:26 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:26.857 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T03:33:27.017 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:33:27.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:33:27.027 DEBUG:teuthology.orchestra.run.smithi117:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-19T03:33:27.106 INFO:teuthology.orchestra.run.smithi117.stdout:1 2024-03-19T03:33:27.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:33:27.111 DEBUG:teuthology.orchestra.run.smithi117:> route 2024-03-19T03:33:27.150 INFO:teuthology.orchestra.run.smithi117.stdout:Kernel IP routing table 2024-03-19T03:33:27.150 INFO:teuthology.orchestra.run.smithi117.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-19T03:33:27.150 INFO:teuthology.orchestra.run.smithi117.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-19T03:33:27.150 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-19T03:33:27.150 INFO:teuthology.orchestra.run.smithi117.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-19T03:33:27.150 DEBUG:teuthology.orchestra.run.smithi117:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T03:33:27.150 DEBUG:teuthology.orchestra.run.smithi117:> set -e 2024-03-19T03:33:27.151 DEBUG:teuthology.orchestra.run.smithi117:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-19T03:33:27.151 DEBUG:teuthology.orchestra.run.smithi117:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-19T03:33:27.151 DEBUG:teuthology.orchestra.run.smithi117:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-19T03:33:27.151 DEBUG:teuthology.orchestra.run.smithi117:> ') 2024-03-19T03:33:27.289 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T03:33:27.403 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:33:27.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:33:27.412 DEBUG:teuthology.orchestra.run.smithi117:> ip netns list 2024-03-19T03:33:27.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:33:27.457 DEBUG:teuthology.orchestra.run.smithi117:> ip netns list-id 2024-03-19T03:33:27.541 DEBUG:teuthology.orchestra.run.smithi117:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T03:33:27.542 DEBUG:teuthology.orchestra.run.smithi117:> set -e 2024-03-19T03:33:27.542 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T03:33:27.542 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-19T03:33:27.542 DEBUG:teuthology.orchestra.run.smithi117:> ') 2024-03-19T03:33:27.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:27 smithi117 ceph-mon[31838]: pgmap v407: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:33:27.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:27 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:27.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:27 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:27.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:27 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:27.543 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:27 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:27.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T03:33:27.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:27 smithi119 ceph-mon[66861]: pgmap v407: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:33:27.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:27 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:27.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:27 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:27.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:27 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:27.735 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:27 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:27.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:33:27.790 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-19T03:33:27.790 DEBUG:teuthology.orchestra.run.smithi117:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T03:33:27.790 DEBUG:teuthology.orchestra.run.smithi117:> set -e 2024-03-19T03:33:27.790 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-19T03:33:27.790 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-19T03:33:27.790 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-19T03:33:27.790 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-19T03:33:27.790 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-19T03:33:27.790 DEBUG:teuthology.orchestra.run.smithi117:> ') 2024-03-19T03:33:27.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:27 smithi178 ceph-mon[67578]: pgmap v407: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:33:27.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:27 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:27.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:27 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:27.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:27 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:27.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:27 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:27.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T03:33:28.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:33:28.123 DEBUG:teuthology.orchestra.run.smithi117:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T03:33:28.124 DEBUG:teuthology.orchestra.run.smithi117:> set -e 2024-03-19T03:33:28.124 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip link set brx.0 up 2024-03-19T03:33:28.124 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip link set dev brx.0 master ceph-brx 2024-03-19T03:33:28.124 DEBUG:teuthology.orchestra.run.smithi117:> ') 2024-03-19T03:33:28.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T03:33:28.324 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:33:28.334 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T03:33:28.334 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-19T03:33:28.376 INFO:teuthology.orchestra.run.smithi117.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-19T03:33:28.376 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T03:33:28.376 DEBUG:teuthology.orchestra.run.smithi117:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-19T03:33:28.459 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-19T03:33:28.459 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-19T03:33:28.459 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/8f1a3abf-f504-494e-80fd-bcaac3a6e05c 2024-03-19T03:33:28.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:33:28.459 DEBUG:teuthology.orchestra.run.smithi117:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/8f1a3abf-f504-494e-80fd-bcaac3a6e05c /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync 2024-03-19T03:33:28.612 INFO:teuthology.orchestra.run.smithi117.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync 2024-03-19T03:33:28.613 INFO:teuthology.orchestra.run.smithi117.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync". 2024-03-19T03:33:28.613 INFO:teuthology.orchestra.run.smithi117.stdout:invalid new device string format 2024-03-19T03:33:29.231 INFO:teuthology.orchestra.run.smithi117.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync 2024-03-19T03:33:29.231 INFO:teuthology.orchestra.run.smithi117.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync". 2024-03-19T03:33:29.231 INFO:teuthology.orchestra.run.smithi117.stdout:invalid new device string format 2024-03-19T03:33:29.231 INFO:teuthology.orchestra.run.smithi117.stdout:mount.ceph: resolved to: "172.21.15.117:6789,172.21.15.119:6789,172.21.15.178:6789" 2024-03-19T03:33:29.232 INFO:teuthology.orchestra.run.smithi117.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.117:6789,172.21.15.119:6789,172.21.15.178:6789:/volumes/qa/sv_1/8f1a3abf-f504-494e-80fd-bcaac3a6e05c 2024-03-19T03:33:29.232 INFO:teuthology.orchestra.run.smithi117.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync,key=0,fsid=a3a82456-e59f-11ee-95c9-87774f69a715" will pass to kernel 2024-03-19T03:33:29.236 INFO:teuthology.orchestra.run.smithi117.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-19T03:33:29.236 INFO:teuthology.orchestra.run.smithi117.stdout: You just mounted a file system that supports labels which does not 2024-03-19T03:33:29.236 INFO:teuthology.orchestra.run.smithi117.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-19T03:33:29.236 INFO:teuthology.orchestra.run.smithi117.stdout: applications will generate AVC messages and not be allowed access to 2024-03-19T03:33:29.236 INFO:teuthology.orchestra.run.smithi117.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-19T03:33:29.241 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-19T03:33:29.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:33:29.241 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-19T03:33:29.325 DEBUG:teuthology.orchestra.run.smithi117:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-19T03:33:29.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:29 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-19T03:33:29.431 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:33:29.443 INFO:teuthology.orchestra.run.smithi117.stdout:client25463 2024-03-19T03:33:29.443 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-19T03:33:29.443 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T03:33:29.484 DEBUG:teuthology.orchestra.run.smithi117:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-19T03:33:29.633 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:29 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-19T03:33:29.641 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:33:29.652 INFO:teuthology.orchestra.run.smithi117.stdout:client25463 2024-03-19T03:33:29.653 DEBUG:teuthology.orchestra.run.smithi117:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-19T03:33:29.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:29 smithi119 ceph-mon[66861]: pgmap v408: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:29.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:29 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:29.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:29 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:29.748 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:29 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-19T03:33:29.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:33:29.764 INFO:teuthology.orchestra.run.smithi117.stdout:client25463 2024-03-19T03:33:29.765 DEBUG:teuthology.orchestra.run.smithi117:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/a3a82456-e59f-11ee-95c9-87774f69a715.client25463/status') 2024-03-19T03:33:29.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:29 smithi117 ceph-mon[31838]: pgmap v408: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:29.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:29 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:29.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:29 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:29.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:29 smithi178 ceph-mon[67578]: pgmap v408: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:29.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:29 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:29.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:29 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:29.861 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T03:33:29.902 INFO:teuthology.orchestra.run.smithi117.stdout:instance: client.25463 (3)172.21.15.117:0/2975909292 2024-03-19T03:33:29.903 INFO:teuthology.orchestra.run.smithi117.stdout:blocklisted: false 2024-03-19T03:33:29.903 INFO:teuthology.orchestra.run.smithi117.stderr:0+1 records in 2024-03-19T03:33:29.903 INFO:teuthology.orchestra.run.smithi117.stderr:0+1 records out 2024-03-19T03:33:29.903 INFO:teuthology.orchestra.run.smithi117.stderr:72 bytes copied, 6.0688e-05 s, 1.2 MB/s 2024-03-19T03:33:29.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T03:33:29.912 INFO:teuthology.run_tasks:Running task exec... 2024-03-19T03:33:29.924 INFO:teuthology.task.exec:Executing custom commands... 2024-03-19T03:33:29.924 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi117.front.sepia.ceph.com 2024-03-19T03:33:29.924 DEBUG:teuthology.orchestra.run.smithi117:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-03-19T03:33:30.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.154+0000 7fa374200640 1 -- 172.21.15.117:0/1237789133 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa36c102a60 msgr2=0x7fa36c102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:30.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.154+0000 7fa374200640 1 --2- 172.21.15.117:0/1237789133 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa36c102a60 0x7fa36c102ee0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa358009900 tx=0x7fa35802bc30 comp rx=0 tx=0).stop 2024-03-19T03:33:30.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.154+0000 7fa374200640 1 -- 172.21.15.117:0/1237789133 shutdown_connections 2024-03-19T03:33:30.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.154+0000 7fa374200640 1 --2- 172.21.15.117:0/1237789133 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa36c103420 0x7fa36c109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:30.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.154+0000 7fa374200640 1 --2- 172.21.15.117:0/1237789133 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa36c102a60 0x7fa36c102ee0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:30.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.154+0000 7fa374200640 1 --2- 172.21.15.117:0/1237789133 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa36c101870 0x7fa36c101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:30.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.154+0000 7fa374200640 1 -- 172.21.15.117:0/1237789133 >> 172.21.15.117:0/1237789133 conn(0x7fa36c0fcde0 msgr2=0x7fa36c0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:30.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.155+0000 7fa374200640 1 -- 172.21.15.117:0/1237789133 shutdown_connections 2024-03-19T03:33:30.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.155+0000 7fa374200640 1 -- 172.21.15.117:0/1237789133 wait complete. 2024-03-19T03:33:30.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.156+0000 7fa374200640 1 Processor -- start 2024-03-19T03:33:30.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.156+0000 7fa374200640 1 -- start start 2024-03-19T03:33:30.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.156+0000 7fa374200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa36c101870 0x7fa36c199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:30.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.156+0000 7fa374200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa36c102a60 0x7fa36c19a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:30.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.157+0000 7fa374200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa36c103420 0x7fa36c1a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:30.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.157+0000 7fa374200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa36c10d4d0 con 0x7fa36c102a60 2024-03-19T03:33:30.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.157+0000 7fa374200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa36c10d350 con 0x7fa36c103420 2024-03-19T03:33:30.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.157+0000 7fa374200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fa36c10d650 con 0x7fa36c101870 2024-03-19T03:33:30.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.157+0000 7fa372200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa36c103420 0x7fa36c1a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:30.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.157+0000 7fa370e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa36c102a60 0x7fa36c19a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:30.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.157+0000 7fa372200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa36c103420 0x7fa36c1a1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:50128/0 (socket says 172.21.15.117:50128) 2024-03-19T03:33:30.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.157+0000 7fa372200640 1 -- 172.21.15.117:0/2397330278 learned_addr learned my addr 172.21.15.117:0/2397330278 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:30.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.157+0000 7fa371800640 1 --2- 172.21.15.117:0/2397330278 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa36c101870 0x7fa36c199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:30.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.157+0000 7fa372200640 1 -- 172.21.15.117:0/2397330278 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa36c101870 msgr2=0x7fa36c199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:30.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.157+0000 7fa372200640 1 --2- 172.21.15.117:0/2397330278 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa36c101870 0x7fa36c199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:30.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.157+0000 7fa372200640 1 -- 172.21.15.117:0/2397330278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa36c102a60 msgr2=0x7fa36c19a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:30.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.157+0000 7fa372200640 1 --2- 172.21.15.117:0/2397330278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa36c102a60 0x7fa36c19a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:30.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.157+0000 7fa372200640 1 -- 172.21.15.117:0/2397330278 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa36c1a1910 con 0x7fa36c103420 2024-03-19T03:33:30.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.158+0000 7fa372200640 1 --2- 172.21.15.117:0/2397330278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa36c103420 0x7fa36c1a1270 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa36400eb10 tx=0x7fa36400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:30.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.158+0000 7fa36a000640 1 -- 172.21.15.117:0/2397330278 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa364004b20 con 0x7fa36c103420 2024-03-19T03:33:30.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.158+0000 7fa36a000640 1 -- 172.21.15.117:0/2397330278 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa364004cc0 con 0x7fa36c103420 2024-03-19T03:33:30.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.158+0000 7fa374200640 1 -- 172.21.15.117:0/2397330278 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa36c1a1c00 con 0x7fa36c103420 2024-03-19T03:33:30.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.158+0000 7fa36a000640 1 -- 172.21.15.117:0/2397330278 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa364010760 con 0x7fa36c103420 2024-03-19T03:33:30.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.158+0000 7fa374200640 1 -- 172.21.15.117:0/2397330278 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa36c1a2150 con 0x7fa36c103420 2024-03-19T03:33:30.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.159+0000 7fa374200640 1 -- 172.21.15.117:0/2397330278 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa36c1a2320 con 0x7fa36c103420 2024-03-19T03:33:30.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.163+0000 7fa36a000640 1 -- 172.21.15.117:0/2397330278 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fa364010900 con 0x7fa36c103420 2024-03-19T03:33:30.165 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.164+0000 7fa36a000640 1 --2- 172.21.15.117:0/2397330278 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa3540a2920 0x7fa3540a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:30.165 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.164+0000 7fa36a000640 1 -- 172.21.15.117:0/2397330278 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fa3640cf600 con 0x7fa36c103420 2024-03-19T03:33:30.165 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.164+0000 7fa371800640 1 --2- 172.21.15.117:0/2397330278 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa3540a2920 0x7fa3540a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:30.166 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.165+0000 7fa371800640 1 --2- 172.21.15.117:0/2397330278 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa3540a2920 0x7fa3540a4de0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa36c19b1d0 tx=0x7fa360005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:30.169 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.169+0000 7fa36a000640 1 -- 172.21.15.117:0/2397330278 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa364053740 con 0x7fa36c103420 2024-03-19T03:33:30.415 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:30.414+0000 7fa374200640 1 -- 172.21.15.117:0/2397330278 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7fa36c1a24f0 con 0x7fa36c103420 2024-03-19T03:33:30.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:30 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:33:30.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:30 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:33:30.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:30 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:30.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:30 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2397330278' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-19T03:33:30.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:30 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-19T03:33:30.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:30 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:33:30.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:30 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:33:30.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:30 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:30.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:30 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2397330278' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-19T03:33:30.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:30 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-19T03:33:30.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:30 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:33:30.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:30 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:33:30.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:30 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:33:30.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:30 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2397330278' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-19T03:33:30.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:30 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-19T03:33:31.004 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.003+0000 7fa36a000640 1 -- 172.21.15.117:0/2397330278 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v29) ==== 118+0+0 (secure 0 0 0) 0x7fa36409c2b0 con 0x7fa36c103420 2024-03-19T03:33:31.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.008+0000 7fa374200640 1 -- 172.21.15.117:0/2397330278 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa3540a2920 msgr2=0x7fa3540a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:31.009 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.008+0000 7fa374200640 1 --2- 172.21.15.117:0/2397330278 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa3540a2920 0x7fa3540a4de0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa36c19b1d0 tx=0x7fa360005f90 comp rx=0 tx=0).stop 2024-03-19T03:33:31.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.008+0000 7fa374200640 1 -- 172.21.15.117:0/2397330278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa36c103420 msgr2=0x7fa36c1a1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:31.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.008+0000 7fa374200640 1 --2- 172.21.15.117:0/2397330278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa36c103420 0x7fa36c1a1270 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa36400eb10 tx=0x7fa36400c2d0 comp rx=0 tx=0).stop 2024-03-19T03:33:31.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.008+0000 7fa374200640 1 -- 172.21.15.117:0/2397330278 shutdown_connections 2024-03-19T03:33:31.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.008+0000 7fa374200640 1 --2- 172.21.15.117:0/2397330278 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa3540a2920 0x7fa3540a4de0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:31.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.008+0000 7fa374200640 1 --2- 172.21.15.117:0/2397330278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa36c103420 0x7fa36c1a1270 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:31.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.008+0000 7fa374200640 1 --2- 172.21.15.117:0/2397330278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa36c102a60 0x7fa36c19a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:31.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.008+0000 7fa374200640 1 --2- 172.21.15.117:0/2397330278 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa36c101870 0x7fa36c199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:31.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.008+0000 7fa374200640 1 -- 172.21.15.117:0/2397330278 >> 172.21.15.117:0/2397330278 conn(0x7fa36c0fcde0 msgr2=0x7fa36c073a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:31.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.009+0000 7fa374200640 1 -- 172.21.15.117:0/2397330278 shutdown_connections 2024-03-19T03:33:31.010 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.009+0000 7fa374200640 1 -- 172.21.15.117:0/2397330278 wait complete. 2024-03-19T03:33:31.029 DEBUG:teuthology.orchestra.run.smithi117:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-03-19T03:33:31.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.219+0000 7f7918800640 1 -- 172.21.15.117:0/1822929853 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f79101075a0 msgr2=0x7f7910109990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:31.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.219+0000 7f7918800640 1 --2- 172.21.15.117:0/1822929853 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f79101075a0 0x7f7910109990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f790c009ca0 tx=0x7f790c02c0f0 comp rx=0 tx=0).stop 2024-03-19T03:33:31.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.219+0000 7f7918800640 1 -- 172.21.15.117:0/1822929853 shutdown_connections 2024-03-19T03:33:31.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.219+0000 7f7918800640 1 --2- 172.21.15.117:0/1822929853 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f79101075a0 0x7f7910109990 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:31.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.219+0000 7f7918800640 1 --2- 172.21.15.117:0/1822929853 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7910104c70 0x7f7910107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:31.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.219+0000 7f7918800640 1 --2- 172.21.15.117:0/1822929853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7910102340 0x7f7910104730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:31.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.219+0000 7f7918800640 1 -- 172.21.15.117:0/1822929853 >> 172.21.15.117:0/1822929853 conn(0x7f79100fbcd0 msgr2=0x7f79100fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:31.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.219+0000 7f7918800640 1 -- 172.21.15.117:0/1822929853 shutdown_connections 2024-03-19T03:33:31.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.219+0000 7f7918800640 1 -- 172.21.15.117:0/1822929853 wait complete. 2024-03-19T03:33:31.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.221+0000 7f7918800640 1 Processor -- start 2024-03-19T03:33:31.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.221+0000 7f7918800640 1 -- start start 2024-03-19T03:33:31.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.222+0000 7f7918800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7910102340 0x7f7910197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:31.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.222+0000 7f7918800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7910104c70 0x7f7910197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:31.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.222+0000 7f7918800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79101075a0 0x7f791019f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:31.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.222+0000 7f7918800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f791010d4e0 con 0x7f79101075a0 2024-03-19T03:33:31.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.222+0000 7f7918800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f791010d360 con 0x7f7910102340 2024-03-19T03:33:31.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.222+0000 7f7918800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f791010d660 con 0x7f7910104c70 2024-03-19T03:33:31.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.222+0000 7f7915e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7910102340 0x7f7910197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:31.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.222+0000 7f7915e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7910102340 0x7f7910197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:50136/0 (socket says 172.21.15.117:50136) 2024-03-19T03:33:31.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.222+0000 7f7915e00640 1 -- 172.21.15.117:0/3491319446 learned_addr learned my addr 172.21.15.117:0/3491319446 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:33:31.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.222+0000 7f7916800640 1 --2- 172.21.15.117:0/3491319446 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79101075a0 0x7f791019f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:31.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.222+0000 7f7915400640 1 --2- 172.21.15.117:0/3491319446 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7910104c70 0x7f7910197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:31.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.222+0000 7f7915e00640 1 -- 172.21.15.117:0/3491319446 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7910104c70 msgr2=0x7f7910197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:31.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.222+0000 7f7915e00640 1 --2- 172.21.15.117:0/3491319446 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7910104c70 0x7f7910197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:31.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.222+0000 7f7915e00640 1 -- 172.21.15.117:0/3491319446 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79101075a0 msgr2=0x7f791019f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:31.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.222+0000 7f7915e00640 1 --2- 172.21.15.117:0/3491319446 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79101075a0 0x7f791019f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:31.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.222+0000 7f7915e00640 1 -- 172.21.15.117:0/3491319446 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f791019f690 con 0x7f7910102340 2024-03-19T03:33:31.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.223+0000 7f7915e00640 1 --2- 172.21.15.117:0/3491319446 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7910102340 0x7f7910197a70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f790400e9f0 tx=0x7f790400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:31.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.223+0000 7f790aa00640 1 -- 172.21.15.117:0/3491319446 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790400cd80 con 0x7f7910102340 2024-03-19T03:33:31.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.223+0000 7f790aa00640 1 -- 172.21.15.117:0/3491319446 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7904010910 con 0x7f7910102340 2024-03-19T03:33:31.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.223+0000 7f790aa00640 1 -- 172.21.15.117:0/3491319446 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904018990 con 0x7f7910102340 2024-03-19T03:33:31.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.223+0000 7f7918800640 1 -- 172.21.15.117:0/3491319446 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f791019f980 con 0x7f7910102340 2024-03-19T03:33:31.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.223+0000 7f7918800640 1 -- 172.21.15.117:0/3491319446 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f791019fe80 con 0x7f7910102340 2024-03-19T03:33:31.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.224+0000 7f7918800640 1 -- 172.21.15.117:0/3491319446 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79100fd7c0 con 0x7f7910102340 2024-03-19T03:33:31.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.228+0000 7f790aa00640 1 -- 172.21.15.117:0/3491319446 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f7904010ac0 con 0x7f7910102340 2024-03-19T03:33:31.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.229+0000 7f790aa00640 1 --2- 172.21.15.117:0/3491319446 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f78f80a2920 0x7f78f80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:33:31.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.229+0000 7f790aa00640 1 -- 172.21.15.117:0/3491319446 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f79040cf1b0 con 0x7f7910102340 2024-03-19T03:33:31.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.229+0000 7f7915400640 1 --2- 172.21.15.117:0/3491319446 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f78f80a2920 0x7f78f80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:33:31.231 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.230+0000 7f7915400640 1 --2- 172.21.15.117:0/3491319446 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f78f80a2920 0x7f78f80a4de0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7910198f80 tx=0x7f7900005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:33:31.234 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.233+0000 7f790aa00640 1 -- 172.21.15.117:0/3491319446 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f790409be60 con 0x7f7910102340 2024-03-19T03:33:31.504 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.503+0000 7f7918800640 1 -- 172.21.15.117:0/3491319446 --> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f7910061d00 con 0x7f78f80a2920 2024-03-19T03:33:31.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.532+0000 7f790aa00640 1 -- 172.21.15.117:0/3491319446 <== mgr.14152 v2:172.21.15.117:6800/3144614841 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f7910061d00 con 0x7f78f80a2920 2024-03-19T03:33:31.533 INFO:teuthology.orchestra.run.smithi117.stdout:{} 2024-03-19T03:33:31.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.535+0000 7f7918800640 1 -- 172.21.15.117:0/3491319446 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f78f80a2920 msgr2=0x7f78f80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:31.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.535+0000 7f7918800640 1 --2- 172.21.15.117:0/3491319446 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f78f80a2920 0x7f78f80a4de0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7910198f80 tx=0x7f7900005f90 comp rx=0 tx=0).stop 2024-03-19T03:33:31.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.535+0000 7f7918800640 1 -- 172.21.15.117:0/3491319446 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7910102340 msgr2=0x7f7910197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:33:31.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.535+0000 7f7918800640 1 --2- 172.21.15.117:0/3491319446 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7910102340 0x7f7910197a70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f790400e9f0 tx=0x7f790400eeb0 comp rx=0 tx=0).stop 2024-03-19T03:33:31.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.535+0000 7f7918800640 1 -- 172.21.15.117:0/3491319446 shutdown_connections 2024-03-19T03:33:31.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.535+0000 7f7918800640 1 --2- 172.21.15.117:0/3491319446 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79101075a0 0x7f791019f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:31.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.535+0000 7f7918800640 1 --2- 172.21.15.117:0/3491319446 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7910104c70 0x7f7910197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:31.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.535+0000 7f7918800640 1 --2- 172.21.15.117:0/3491319446 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f78f80a2920 0x7f78f80a4de0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:31.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.535+0000 7f7918800640 1 --2- 172.21.15.117:0/3491319446 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7910102340 0x7f7910197a70 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:33:31.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.535+0000 7f7918800640 1 -- 172.21.15.117:0/3491319446 >> 172.21.15.117:0/3491319446 conn(0x7f79100fbcd0 msgr2=0x7f7910102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:33:31.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.535+0000 7f7918800640 1 -- 172.21.15.117:0/3491319446 shutdown_connections 2024-03-19T03:33:31.539 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:33:31.536+0000 7f7918800640 1 -- 172.21.15.117:0/3491319446 wait complete. 2024-03-19T03:33:31.558 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-19T03:33:31.569 INFO:teuthology.run_tasks:Running task workunit... 2024-03-19T03:33:31.581 INFO:tasks.workunit:Pulling workunits from ref e48b1f6c0608b2615e1890cc46286c3877d6e910 2024-03-19T03:33:31.582 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-19T03:33:31.582 INFO:tasks.workunit:timeout=3h 2024-03-19T03:33:31.582 INFO:tasks.workunit:cleanup=True 2024-03-19T03:33:31.582 DEBUG:teuthology.orchestra.run.smithi117:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-19T03:33:31.625 INFO:teuthology.orchestra.run.smithi117.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-19T03:33:31.625 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-19T03:33:31.625 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 76h/118d Inode: 1099511628283 Links: 2 2024-03-19T03:33:31.625 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T03:33:31.625 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-19T03:33:31.625 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-19 03:31:10.619923007 +0000 2024-03-19T03:33:31.625 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-19 03:31:10.619923007 +0000 2024-03-19T03:33:31.625 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-19 03:33:29.319007052 +0000 2024-03-19T03:33:31.625 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-19 03:31:10.619923007 +0000 2024-03-19T03:33:31.626 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-19T03:33:31.626 DEBUG:teuthology.orchestra.run.smithi117:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-19T03:33:31.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:31 smithi119 ceph-mon[66861]: pgmap v409: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:33:31.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:31 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-19T03:33:31.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:31 smithi119 ceph-mon[66861]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:33:31.737 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:31 smithi117 ceph-mon[31838]: pgmap v409: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:33:31.737 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:31 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-19T03:33:31.737 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:31 smithi117 ceph-mon[31838]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:33:31.738 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:33:31 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:33:31.511+0000 7f2510400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T03:33:31.738 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:33:31 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:33:31.511+0000 7f2510400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T03:33:31.738 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:33:31 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:33:31.511+0000 7f2510400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T03:33:31.738 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:33:31 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:33:31.511+0000 7f2510400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T03:33:31.738 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:33:31 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:33:31.511+0000 7f2510400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T03:33:31.758 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout e48b1f6c0608b2615e1890cc46286c3877d6e910 2024-03-19T03:33:31.811 INFO:tasks.workunit.client.0.smithi117.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-19T03:33:31.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:31 smithi178 ceph-mon[67578]: pgmap v409: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:33:31.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:31 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-19T03:33:31.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:31 smithi178 ceph-mon[67578]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:33:33.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:33 smithi119 ceph-mon[66861]: from='client.25475 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:33:33.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:33 smithi119 ceph-mon[66861]: pgmap v410: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T03:33:33.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:33 smithi117 ceph-mon[31838]: from='client.25475 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:33:33.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:33 smithi117 ceph-mon[31838]: pgmap v410: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T03:33:33.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:33 smithi178 ceph-mon[67578]: from='client.25475 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T03:33:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:33 smithi178 ceph-mon[67578]: pgmap v410: 129 pgs: 129 active+clean; 647 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T03:33:34.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:34 smithi119 ceph-mon[66861]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-03-19T03:33:34.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:34 smithi117 ceph-mon[31838]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-03-19T03:33:34.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:34 smithi178 ceph-mon[67578]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-03-19T03:33:35.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:35 smithi119 ceph-mon[66861]: pgmap v411: 129 pgs: 129 active+clean; 660 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-03-19T03:33:35.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:35 smithi117 ceph-mon[31838]: pgmap v411: 129 pgs: 129 active+clean; 660 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-03-19T03:33:35.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:35 smithi178 ceph-mon[67578]: pgmap v411: 129 pgs: 129 active+clean; 660 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-03-19T03:33:37.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:37 smithi119 ceph-mon[66861]: pgmap v412: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-03-19T03:33:37.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:37 smithi117 ceph-mon[31838]: pgmap v412: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-03-19T03:33:37.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:37 smithi178 ceph-mon[67578]: pgmap v412: 129 pgs: 129 active+clean; 661 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-03-19T03:33:39.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:39 smithi119 ceph-mon[66861]: pgmap v413: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-03-19T03:33:39.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:39 smithi117 ceph-mon[31838]: pgmap v413: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-03-19T03:33:39.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:39 smithi178 ceph-mon[67578]: pgmap v413: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-03-19T03:33:41.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:41 smithi119 ceph-mon[66861]: pgmap v414: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.4 KiB/s wr, 11 op/s 2024-03-19T03:33:41.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:41 smithi117 ceph-mon[31838]: pgmap v414: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.4 KiB/s wr, 11 op/s 2024-03-19T03:33:41.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:41 smithi178 ceph-mon[67578]: pgmap v414: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.4 KiB/s wr, 11 op/s 2024-03-19T03:33:43.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:43 smithi119 ceph-mon[66861]: pgmap v415: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-03-19T03:33:43.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:43 smithi117 ceph-mon[31838]: pgmap v415: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-03-19T03:33:43.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:43 smithi178 ceph-mon[67578]: pgmap v415: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-03-19T03:33:45.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:45 smithi119 ceph-mon[66861]: pgmap v416: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-03-19T03:33:45.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:45 smithi117 ceph-mon[31838]: pgmap v416: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-03-19T03:33:45.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:45 smithi178 ceph-mon[67578]: pgmap v416: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-03-19T03:33:46.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:46 smithi117 ceph-mon[31838]: pgmap v417: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-19T03:33:46.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:46 smithi178 ceph-mon[67578]: pgmap v417: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-19T03:33:46.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:46 smithi119 ceph-mon[66861]: pgmap v417: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-19T03:33:49.731 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:49 smithi117 ceph-mon[31838]: pgmap v418: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-19T03:33:49.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:49 smithi119 ceph-mon[66861]: pgmap v418: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-19T03:33:49.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:49 smithi178 ceph-mon[67578]: pgmap v418: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-19T03:33:51.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:51 smithi119 ceph-mon[66861]: pgmap v419: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-19T03:33:51.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:51 smithi117 ceph-mon[31838]: pgmap v419: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-19T03:33:51.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:51 smithi178 ceph-mon[67578]: pgmap v419: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-19T03:33:53.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:53 smithi119 ceph-mon[66861]: pgmap v420: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-19T03:33:53.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:53 smithi117 ceph-mon[31838]: pgmap v420: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-19T03:33:53.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:53 smithi178 ceph-mon[67578]: pgmap v420: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-19T03:33:55.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:55 smithi119 ceph-mon[66861]: pgmap v421: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-03-19T03:33:55.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:55 smithi117 ceph-mon[31838]: pgmap v421: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-03-19T03:33:55.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:55 smithi178 ceph-mon[67578]: pgmap v421: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-03-19T03:33:57.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:57 smithi119 ceph-mon[66861]: pgmap v422: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:33:57.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:57 smithi117 ceph-mon[31838]: pgmap v422: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:33:57.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:57 smithi178 ceph-mon[67578]: pgmap v422: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:33:59.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:33:59 smithi119 ceph-mon[66861]: pgmap v423: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:59.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:33:59 smithi117 ceph-mon[31838]: pgmap v423: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:33:59.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:33:59 smithi178 ceph-mon[67578]: pgmap v423: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:34:01.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:01 smithi119 ceph-mon[66861]: pgmap v424: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:34:01.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:01 smithi117 ceph-mon[31838]: pgmap v424: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:34:01.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:01 smithi178 ceph-mon[67578]: pgmap v424: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:34:03.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:03 smithi119 ceph-mon[66861]: pgmap v425: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T03:34:03.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:03 smithi117 ceph-mon[31838]: pgmap v425: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T03:34:03.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:03 smithi178 ceph-mon[67578]: pgmap v425: 129 pgs: 129 active+clean; 663 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T03:34:05.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:05 smithi119 ceph-mon[66861]: pgmap v426: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-19T03:34:05.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:05 smithi117 ceph-mon[31838]: pgmap v426: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-19T03:34:05.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:05 smithi178 ceph-mon[67578]: pgmap v426: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-19T03:34:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:07 smithi119 ceph-mon[66861]: pgmap v427: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-19T03:34:07.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:07 smithi117 ceph-mon[31838]: pgmap v427: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-19T03:34:07.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:07 smithi178 ceph-mon[67578]: pgmap v427: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-19T03:34:09.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:09 smithi119 ceph-mon[66861]: pgmap v428: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-19T03:34:09.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:09 smithi117 ceph-mon[31838]: pgmap v428: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-19T03:34:09.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:09 smithi178 ceph-mon[67578]: pgmap v428: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-19T03:34:11.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:11 smithi119 ceph-mon[66861]: pgmap v429: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-19T03:34:11.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:11 smithi117 ceph-mon[31838]: pgmap v429: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-19T03:34:11.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:11 smithi178 ceph-mon[67578]: pgmap v429: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-19T03:34:13.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:13 smithi119 ceph-mon[66861]: pgmap v430: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-19T03:34:13.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:13 smithi117 ceph-mon[31838]: pgmap v430: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-19T03:34:13.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:13 smithi178 ceph-mon[67578]: pgmap v430: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-19T03:34:15.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:15 smithi119 ceph-mon[66861]: pgmap v431: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-19T03:34:15.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:15 smithi117 ceph-mon[31838]: pgmap v431: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-19T03:34:15.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:15 smithi178 ceph-mon[67578]: pgmap v431: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-19T03:34:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:17 smithi119 ceph-mon[66861]: pgmap v432: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-19T03:34:17.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:17 smithi117 ceph-mon[31838]: pgmap v432: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-19T03:34:17.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:17 smithi178 ceph-mon[67578]: pgmap v432: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-19T03:34:19.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:19 smithi119 ceph-mon[66861]: pgmap v433: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 10 op/s 2024-03-19T03:34:19.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:19 smithi117 ceph-mon[31838]: pgmap v433: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 10 op/s 2024-03-19T03:34:19.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:19 smithi178 ceph-mon[67578]: pgmap v433: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 10 op/s 2024-03-19T03:34:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:21 smithi119 ceph-mon[66861]: pgmap v434: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:34:21.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:21 smithi117 ceph-mon[31838]: pgmap v434: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:34:21.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:21 smithi178 ceph-mon[67578]: pgmap v434: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:34:23.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:23 smithi119 ceph-mon[66861]: pgmap v435: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T03:34:23.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:23 smithi117 ceph-mon[31838]: pgmap v435: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T03:34:23.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:23 smithi178 ceph-mon[67578]: pgmap v435: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-19T03:34:24.464 INFO:tasks.workunit.client.0.smithi117.stderr:Note: switching to 'e48b1f6c0608b2615e1890cc46286c3877d6e910'. 2024-03-19T03:34:24.464 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-03-19T03:34:24.464 INFO:tasks.workunit.client.0.smithi117.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-19T03:34:24.464 INFO:tasks.workunit.client.0.smithi117.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-19T03:34:24.464 INFO:tasks.workunit.client.0.smithi117.stderr:state without impacting any branches by switching back to a branch. 2024-03-19T03:34:24.464 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-03-19T03:34:24.464 INFO:tasks.workunit.client.0.smithi117.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-19T03:34:24.464 INFO:tasks.workunit.client.0.smithi117.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-19T03:34:24.464 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-03-19T03:34:24.464 INFO:tasks.workunit.client.0.smithi117.stderr: git switch -c 2024-03-19T03:34:24.464 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-03-19T03:34:24.465 INFO:tasks.workunit.client.0.smithi117.stderr:Or undo this operation with: 2024-03-19T03:34:24.465 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-03-19T03:34:24.465 INFO:tasks.workunit.client.0.smithi117.stderr: git switch - 2024-03-19T03:34:24.465 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-03-19T03:34:24.465 INFO:tasks.workunit.client.0.smithi117.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-19T03:34:24.465 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-03-19T03:34:24.465 INFO:tasks.workunit.client.0.smithi117.stderr:HEAD is now at e48b1f6c060 testfix 2024-03-19T03:34:24.481 DEBUG:teuthology.orchestra.run.smithi117:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-19T03:34:24.541 INFO:tasks.workunit.client.0.smithi117.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-19T03:34:24.546 INFO:tasks.workunit.client.0.smithi117.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-19T03:34:24.547 INFO:tasks.workunit.client.0.smithi117.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-19T03:34:25.458 INFO:tasks.workunit.client.0.smithi117.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-19T03:34:25.603 INFO:tasks.workunit.client.0.smithi117.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-19T03:34:25.683 INFO:tasks.workunit.client.0.smithi117.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-19T03:34:25.690 INFO:tasks.workunit.client.0.smithi117.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-19T03:34:25.691 INFO:tasks.workunit.client.0.smithi117.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-19T03:34:25.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:25 smithi119 ceph-mon[66861]: pgmap v436: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:34:25.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:25 smithi117 ceph-mon[31838]: pgmap v436: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:34:25.805 INFO:tasks.workunit.client.0.smithi117.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-19T03:34:25.815 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-19T03:34:25.815 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-19T03:34:25.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:25 smithi178 ceph-mon[67578]: pgmap v436: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-19T03:34:25.858 INFO:tasks.workunit:Running workunits matching suites/blogbench.sh on client.0... 2024-03-19T03:34:25.859 INFO:tasks.workunit:Running workunit suites/blogbench.sh... 2024-03-19T03:34:25.859 DEBUG:teuthology.orchestra.run.smithi117:workunit test suites/blogbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e48b1f6c0608b2615e1890cc46286c3877d6e910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh 2024-03-19T03:34:25.960 INFO:tasks.workunit.client.0.smithi117.stdout:getting blogbench 2024-03-19T03:34:25.960 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'getting blogbench' 2024-03-19T03:34:25.961 INFO:tasks.workunit.client.0.smithi117.stderr:+ wget http://download.ceph.com/qa/blogbench-1.0.tar.bz2 2024-03-19T03:34:25.997 INFO:tasks.workunit.client.0.smithi117.stderr:--2024-03-19 03:34:25-- http://download.ceph.com/qa/blogbench-1.0.tar.bz2 2024-03-19T03:34:25.998 INFO:tasks.workunit.client.0.smithi117.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-03-19T03:34:26.034 INFO:tasks.workunit.client.0.smithi117.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-03-19T03:34:26.069 INFO:tasks.workunit.client.0.smithi117.stderr:HTTP request sent, awaiting response... 200 OK 2024-03-19T03:34:26.070 INFO:tasks.workunit.client.0.smithi117.stderr:Length: 103355 (101K) [application/octet-stream] 2024-03-19T03:34:26.072 INFO:tasks.workunit.client.0.smithi117.stderr:Saving to: ‘blogbench-1.0.tar.bz2’ 2024-03-19T03:34:26.077 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-03-19T03:34:26.143 INFO:tasks.workunit.client.0.smithi117.stderr: 0K .......... .......... .......... .......... .......... 49% 721K 0s 2024-03-19T03:34:26.178 INFO:tasks.workunit.client.0.smithi117.stderr: 50K .......... .......... .......... .......... .......... 99% 1.39M 0s 2024-03-19T03:34:26.178 INFO:tasks.workunit.client.0.smithi117.stderr: 100K 100% 1.74T=0.1s 2024-03-19T03:34:26.178 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-03-19T03:34:26.178 INFO:tasks.workunit.client.0.smithi117.stderr:2024-03-19 03:34:26 (965 KB/s) - ‘blogbench-1.0.tar.bz2’ saved [103355/103355] 2024-03-19T03:34:26.178 INFO:tasks.workunit.client.0.smithi117.stderr: 2024-03-19T03:34:26.180 INFO:tasks.workunit.client.0.smithi117.stderr:+ tar -xvf blogbench-1.0.tar.bz2 2024-03-19T03:34:26.215 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/ 2024-03-19T03:34:26.217 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/m4/ 2024-03-19T03:34:26.219 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/m4/acx_pthread.m4 2024-03-19T03:34:26.221 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/man/ 2024-03-19T03:34:26.222 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/man/blogbench.8 2024-03-19T03:34:26.224 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/man/Makefile.am 2024-03-19T03:34:26.225 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/man/Makefile.in 2024-03-19T03:34:26.226 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/src/ 2024-03-19T03:34:26.227 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/src/mysnprintf.c 2024-03-19T03:34:26.228 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/src/mysnprintf.h 2024-03-19T03:34:26.229 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/src/blogbench.c 2024-03-19T03:34:26.231 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/src/blogbench.h 2024-03-19T03:34:26.232 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/src/fakesnprintf.c 2024-03-19T03:34:26.234 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/src/fakesnprintf.h 2024-03-19T03:34:26.236 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/src/blogbench_p.h 2024-03-19T03:34:26.238 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/src/bsd-getopt_long.c 2024-03-19T03:34:26.239 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/src/bsd-getopt_long.h 2024-03-19T03:34:26.241 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/src/Makefile.am 2024-03-19T03:34:26.243 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/src/Makefile.in 2024-03-19T03:34:26.245 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/src/process.c 2024-03-19T03:34:26.247 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/src/globals.h 2024-03-19T03:34:26.249 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/src/commenter.c 2024-03-19T03:34:26.250 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/src/rewriter.c 2024-03-19T03:34:26.252 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/src/reader.c 2024-03-19T03:34:26.253 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/src/writer.c 2024-03-19T03:34:26.255 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/src/helpers.c 2024-03-19T03:34:26.256 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/NEWS 2024-03-19T03:34:26.258 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/compile 2024-03-19T03:34:26.259 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/depcomp 2024-03-19T03:34:26.261 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/aclocal.m4 2024-03-19T03:34:26.262 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/README 2024-03-19T03:34:26.264 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/configure 2024-03-19T03:34:26.272 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/configure.ac 2024-03-19T03:34:26.273 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/config.guess 2024-03-19T03:34:26.275 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/install-sh 2024-03-19T03:34:26.276 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/config.sub 2024-03-19T03:34:26.278 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/missing 2024-03-19T03:34:26.279 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/mkinstalldirs 2024-03-19T03:34:26.281 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/Makefile.am 2024-03-19T03:34:26.283 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/Makefile.in 2024-03-19T03:34:26.284 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/config.h.in 2024-03-19T03:34:26.286 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/AUTHORS 2024-03-19T03:34:26.287 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/INSTALL 2024-03-19T03:34:26.289 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/ChangeLog 2024-03-19T03:34:26.291 INFO:tasks.workunit.client.0.smithi117.stdout:blogbench-1.0/COPYING 2024-03-19T03:34:26.294 INFO:tasks.workunit.client.0.smithi117.stderr:+ cd blogbench-1.0/ 2024-03-19T03:34:26.294 INFO:tasks.workunit.client.0.smithi117.stdout:making blogbench 2024-03-19T03:34:26.295 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'making blogbench' 2024-03-19T03:34:26.295 INFO:tasks.workunit.client.0.smithi117.stderr:+ ./configure 2024-03-19T03:34:26.563 INFO:tasks.workunit.client.0.smithi117.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-19T03:34:27.583 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether build environment is sane... yes 2024-03-19T03:34:27.603 INFO:tasks.workunit.client.0.smithi117.stdout:checking for gawk... gawk 2024-03-19T03:34:27.630 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether make sets $(MAKE)... yes 2024-03-19T03:34:27.651 INFO:tasks.workunit.client.0.smithi117.stdout:checking for gcc... gcc 2024-03-19T03:34:27.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:27 smithi119 ceph-mon[66861]: pgmap v437: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:34:27.763 INFO:tasks.workunit.client.0.smithi117.stdout:checking for C compiler default output file name... a.out 2024-03-19T03:34:27.770 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether the C compiler works... yes 2024-03-19T03:34:27.776 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether we are cross compiling... no 2024-03-19T03:34:27.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:27 smithi117 ceph-mon[31838]: pgmap v437: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:34:27.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:27 smithi178 ceph-mon[67578]: pgmap v437: 129 pgs: 129 active+clean; 668 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-19T03:34:27.848 INFO:tasks.workunit.client.0.smithi117.stdout:checking for suffix of executables... 2024-03-19T03:34:27.924 INFO:tasks.workunit.client.0.smithi117.stdout:checking for suffix of object files... o 2024-03-19T03:34:27.995 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether we are using the GNU C compiler... yes 2024-03-19T03:34:28.062 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether gcc accepts -g... yes 2024-03-19T03:34:28.177 INFO:tasks.workunit.client.0.smithi117.stdout:checking for gcc option to accept ANSI C... none needed 2024-03-19T03:34:28.254 INFO:tasks.workunit.client.0.smithi117.stdout:checking for style of include used by make... GNU 2024-03-19T03:34:28.439 INFO:tasks.workunit.client.0.smithi117.stdout:checking dependency style of gcc... gcc3 2024-03-19T03:34:28.440 INFO:tasks.workunit.client.0.smithi117.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-19T03:34:28.575 INFO:tasks.workunit.client.0.smithi117.stdout:checking how to run the C preprocessor... gcc -E 2024-03-19T03:34:28.683 INFO:tasks.workunit.client.0.smithi117.stdout:checking for egrep... grep -E 2024-03-19T03:34:28.707 INFO:tasks.workunit.client.0.smithi117.stdout:checking for AIX... no 2024-03-19T03:34:28.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:28 smithi117 ceph-mon[31838]: pgmap v438: 129 pgs: 129 active+clean; 1004 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 29 KiB/s wr, 11 op/s 2024-03-19T03:34:28.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:28 smithi178 ceph-mon[67578]: pgmap v438: 129 pgs: 129 active+clean; 1004 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 29 KiB/s wr, 11 op/s 2024-03-19T03:34:28.847 INFO:tasks.workunit.client.0.smithi117.stdout:checking for library containing strerror... none required 2024-03-19T03:34:28.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:28 smithi119 ceph-mon[66861]: pgmap v438: 129 pgs: 129 active+clean; 1004 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 29 KiB/s wr, 11 op/s 2024-03-19T03:34:29.186 INFO:tasks.workunit.client.0.smithi117.stdout:checking for ANSI C header files... yes 2024-03-19T03:34:29.278 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-03-19T03:34:29.386 INFO:tasks.workunit.client.0.smithi117.stdout:checking for sys/types.h... yes 2024-03-19T03:34:29.496 INFO:tasks.workunit.client.0.smithi117.stdout:checking for sys/stat.h... yes 2024-03-19T03:34:29.616 INFO:tasks.workunit.client.0.smithi117.stdout:checking for stdlib.h... yes 2024-03-19T03:34:29.734 INFO:tasks.workunit.client.0.smithi117.stdout:checking for string.h... yes 2024-03-19T03:34:29.858 INFO:tasks.workunit.client.0.smithi117.stdout:checking for memory.h... yes 2024-03-19T03:34:29.972 INFO:tasks.workunit.client.0.smithi117.stdout:checking for strings.h... yes 2024-03-19T03:34:30.081 INFO:tasks.workunit.client.0.smithi117.stdout:checking for inttypes.h... yes 2024-03-19T03:34:30.202 INFO:tasks.workunit.client.0.smithi117.stdout:checking for stdint.h... yes 2024-03-19T03:34:30.331 INFO:tasks.workunit.client.0.smithi117.stdout:checking for unistd.h... yes 2024-03-19T03:34:30.359 INFO:tasks.workunit.client.0.smithi117.stdout:checking for memory.h... (cached) yes 2024-03-19T03:34:30.375 INFO:tasks.workunit.client.0.smithi117.stdout:checking for strings.h... (cached) yes 2024-03-19T03:34:30.393 INFO:tasks.workunit.client.0.smithi117.stdout:checking for unistd.h... (cached) yes 2024-03-19T03:34:30.507 INFO:tasks.workunit.client.0.smithi117.stdout:checking locale.h usability... yes 2024-03-19T03:34:30.556 INFO:tasks.workunit.client.0.smithi117.stdout:checking locale.h presence... yes 2024-03-19T03:34:30.558 INFO:tasks.workunit.client.0.smithi117.stdout:checking for locale.h... yes 2024-03-19T03:34:30.589 INFO:tasks.workunit.client.0.smithi117.stdout:checking for stdlib.h... (cached) yes 2024-03-19T03:34:30.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:30 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:34:30.605 INFO:tasks.workunit.client.0.smithi117.stdout:checking for string.h... (cached) yes 2024-03-19T03:34:30.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:30 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:34:30.737 INFO:tasks.workunit.client.0.smithi117.stdout:checking sys/socket.h usability... yes 2024-03-19T03:34:30.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:30 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:34:30.799 INFO:tasks.workunit.client.0.smithi117.stdout:checking sys/socket.h presence... yes 2024-03-19T03:34:30.802 INFO:tasks.workunit.client.0.smithi117.stdout:checking for sys/socket.h... yes 2024-03-19T03:34:30.917 INFO:tasks.workunit.client.0.smithi117.stdout:checking sys/time.h usability... yes 2024-03-19T03:34:30.962 INFO:tasks.workunit.client.0.smithi117.stdout:checking sys/time.h presence... yes 2024-03-19T03:34:30.963 INFO:tasks.workunit.client.0.smithi117.stdout:checking for sys/time.h... yes 2024-03-19T03:34:31.078 INFO:tasks.workunit.client.0.smithi117.stdout:checking fcntl.h usability... yes 2024-03-19T03:34:31.121 INFO:tasks.workunit.client.0.smithi117.stdout:checking fcntl.h presence... yes 2024-03-19T03:34:31.123 INFO:tasks.workunit.client.0.smithi117.stdout:checking for fcntl.h... yes 2024-03-19T03:34:31.312 INFO:tasks.workunit.client.0.smithi117.stdout:checking sys/fcntl.h usability... yes 2024-03-19T03:34:31.355 INFO:tasks.workunit.client.0.smithi117.stdout:checking sys/fcntl.h presence... yes 2024-03-19T03:34:31.357 INFO:tasks.workunit.client.0.smithi117.stdout:checking for sys/fcntl.h... yes 2024-03-19T03:34:31.535 INFO:tasks.workunit.client.0.smithi117.stdout:checking sys/param.h usability... yes 2024-03-19T03:34:31.583 INFO:tasks.workunit.client.0.smithi117.stdout:checking sys/param.h presence... yes 2024-03-19T03:34:31.587 INFO:tasks.workunit.client.0.smithi117.stdout:checking for sys/param.h... yes 2024-03-19T03:34:31.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:31 smithi178 ceph-mon[67578]: pgmap v439: 129 pgs: 129 active+clean; 1004 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 9 op/s 2024-03-19T03:34:31.690 INFO:tasks.workunit.client.0.smithi117.stdout:checking ioctl.h usability... no 2024-03-19T03:34:31.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:31 smithi119 ceph-mon[66861]: pgmap v439: 129 pgs: 129 active+clean; 1004 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 9 op/s 2024-03-19T03:34:31.734 INFO:tasks.workunit.client.0.smithi117.stdout:checking ioctl.h presence... no 2024-03-19T03:34:31.736 INFO:tasks.workunit.client.0.smithi117.stdout:checking for ioctl.h... no 2024-03-19T03:34:31.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:31 smithi117 ceph-mon[31838]: pgmap v439: 129 pgs: 129 active+clean; 1004 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 29 KiB/s wr, 9 op/s 2024-03-19T03:34:31.843 INFO:tasks.workunit.client.0.smithi117.stdout:checking sys/ioctl.h usability... yes 2024-03-19T03:34:31.889 INFO:tasks.workunit.client.0.smithi117.stdout:checking sys/ioctl.h presence... yes 2024-03-19T03:34:31.892 INFO:tasks.workunit.client.0.smithi117.stdout:checking for sys/ioctl.h... yes 2024-03-19T03:34:32.000 INFO:tasks.workunit.client.0.smithi117.stdout:checking return type of signal handlers... void 2024-03-19T03:34:32.099 INFO:tasks.workunit.client.0.smithi117.stdout:checking for size_t... yes 2024-03-19T03:34:32.200 INFO:tasks.workunit.client.0.smithi117.stdout:checking for dev_t... yes 2024-03-19T03:34:32.294 INFO:tasks.workunit.client.0.smithi117.stdout:checking for short... yes 2024-03-19T03:34:32.466 INFO:tasks.workunit.client.0.smithi117.stdout:checking size of short... 2 2024-03-19T03:34:32.589 INFO:tasks.workunit.client.0.smithi117.stdout:checking for int... yes 2024-03-19T03:34:32.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:34:32.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:34:32.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:32 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:34:32.728 INFO:tasks.workunit.client.0.smithi117.stdout:checking size of int... 4 2024-03-19T03:34:32.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:32 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:34:32.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:32 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:34:32.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:32 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:34:32.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:32 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:34:32.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:32 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:34:32.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:32 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:34:32.842 INFO:tasks.workunit.client.0.smithi117.stdout:checking for long... yes 2024-03-19T03:34:32.988 INFO:tasks.workunit.client.0.smithi117.stdout:checking size of long... 8 2024-03-19T03:34:33.084 INFO:tasks.workunit.client.0.smithi117.stdout:checking for long long... yes 2024-03-19T03:34:33.232 INFO:tasks.workunit.client.0.smithi117.stdout:checking size of long long... 8 2024-03-19T03:34:33.349 INFO:tasks.workunit.client.0.smithi117.stdout:checking for ssize_t... yes 2024-03-19T03:34:33.452 INFO:tasks.workunit.client.0.smithi117.stdout:checking for sig_atomic_t... yes 2024-03-19T03:34:33.552 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether gcc needs -traditional... no 2024-03-19T03:34:33.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:33 smithi178 ceph-mon[67578]: pgmap v440: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 42 KiB/s wr, 15 op/s 2024-03-19T03:34:33.641 INFO:tasks.workunit.client.0.smithi117.stdout:checking for an ANSI C-conforming const... yes 2024-03-19T03:34:33.713 INFO:tasks.workunit.client.0.smithi117.stdout:checking for inline... inline 2024-03-19T03:34:33.733 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:33 smithi119 ceph-mon[66861]: pgmap v440: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 42 KiB/s wr, 15 op/s 2024-03-19T03:34:33.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:33 smithi117 ceph-mon[31838]: pgmap v440: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 42 KiB/s wr, 15 op/s 2024-03-19T03:34:33.843 INFO:tasks.workunit.client.0.smithi117.stdout:checking for working alloca.h... yes 2024-03-19T03:34:33.958 INFO:tasks.workunit.client.0.smithi117.stdout:checking for alloca... yes 2024-03-19T03:34:34.087 INFO:tasks.workunit.client.0.smithi117.stdout:checking for working memcmp... yes 2024-03-19T03:34:34.210 INFO:tasks.workunit.client.0.smithi117.stdout:checking for strftime... yes 2024-03-19T03:34:34.363 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether lstat dereferences a symlink specified with a trailing slash... yes 2024-03-19T03:34:34.490 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether stat accepts an empty string... no 2024-03-19T03:34:34.625 INFO:tasks.workunit.client.0.smithi117.stdout:checking for vprintf... yes 2024-03-19T03:34:34.750 INFO:tasks.workunit.client.0.smithi117.stdout:checking for _doprnt... no 2024-03-19T03:34:34.818 INFO:tasks.workunit.client.0.smithi117.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-03-19T03:34:34.854 INFO:tasks.workunit.client.0.smithi117.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-03-19T03:34:34.980 INFO:tasks.workunit.client.0.smithi117.stdout:checking for the pthreads library -lpthreads... no 2024-03-19T03:34:35.102 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether pthreads work without any flags... yes 2024-03-19T03:34:35.215 INFO:tasks.workunit.client.0.smithi117.stdout:checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE 2024-03-19T03:34:35.216 INFO:tasks.workunit.client.0.smithi117.stdout:checking if more special flags are required for pthreads... no 2024-03-19T03:34:35.216 INFO:tasks.workunit.client.0.smithi117.stdout:checking for cc_r... gcc 2024-03-19T03:34:35.381 INFO:tasks.workunit.client.0.smithi117.stdout:checking for setrlimit... yes 2024-03-19T03:34:35.513 INFO:tasks.workunit.client.0.smithi117.stdout:checking for setproctitle... no 2024-03-19T03:34:35.665 INFO:tasks.workunit.client.0.smithi117.stdout:checking for getopt_long... yes 2024-03-19T03:34:35.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:35 smithi119 ceph-mon[66861]: pgmap v441: 129 pgs: 129 active+clean; 2.6 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 172 KiB/s wr, 20 op/s 2024-03-19T03:34:35.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:35 smithi117 ceph-mon[31838]: pgmap v441: 129 pgs: 129 active+clean; 2.6 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 172 KiB/s wr, 20 op/s 2024-03-19T03:34:35.799 INFO:tasks.workunit.client.0.smithi117.stdout:checking for strtoull... yes 2024-03-19T03:34:35.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:35 smithi178 ceph-mon[67578]: pgmap v441: 129 pgs: 129 active+clean; 2.6 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 172 KiB/s wr, 20 op/s 2024-03-19T03:34:35.950 INFO:tasks.workunit.client.0.smithi117.stdout:checking for strtoq... yes 2024-03-19T03:34:36.082 INFO:tasks.workunit.client.0.smithi117.stdout:checking for snprintf... yes 2024-03-19T03:34:36.219 INFO:tasks.workunit.client.0.smithi117.stdout:checking for vsnprintf... yes 2024-03-19T03:34:36.400 INFO:tasks.workunit.client.0.smithi117.stdout:checking for vfprintf... yes 2024-03-19T03:34:36.530 INFO:tasks.workunit.client.0.smithi117.stdout:checking for gethostname... yes 2024-03-19T03:34:36.664 INFO:tasks.workunit.client.0.smithi117.stdout:checking for setlocale... yes 2024-03-19T03:34:36.811 INFO:tasks.workunit.client.0.smithi117.stdout:checking for sync... yes 2024-03-19T03:34:36.999 INFO:tasks.workunit.client.0.smithi117.stdout:checking whether snprintf is C99 conformant... done 2024-03-19T03:34:37.025 INFO:tasks.workunit.client.0.smithi117.stdout:configure: creating ./config.status 2024-03-19T03:34:37.382 INFO:tasks.workunit.client.0.smithi117.stdout:config.status: creating Makefile 2024-03-19T03:34:37.437 INFO:tasks.workunit.client.0.smithi117.stdout:config.status: creating src/Makefile 2024-03-19T03:34:37.483 INFO:tasks.workunit.client.0.smithi117.stdout:config.status: creating man/Makefile 2024-03-19T03:34:37.508 INFO:tasks.workunit.client.0.smithi117.stdout:config.status: creating config.h 2024-03-19T03:34:37.625 INFO:tasks.workunit.client.0.smithi117.stdout:config.status: executing depfiles commands 2024-03-19T03:34:37.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:37 smithi119 ceph-mon[66861]: pgmap v442: 129 pgs: 129 active+clean; 3.2 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 227 KiB/s wr, 26 op/s 2024-03-19T03:34:37.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:37 smithi117 ceph-mon[31838]: pgmap v442: 129 pgs: 129 active+clean; 3.2 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 227 KiB/s wr, 26 op/s 2024-03-19T03:34:37.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:37 smithi178 ceph-mon[67578]: pgmap v442: 129 pgs: 129 active+clean; 3.2 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 227 KiB/s wr, 26 op/s 2024-03-19T03:34:37.976 INFO:tasks.workunit.client.0.smithi117.stderr:+ make 2024-03-19T03:34:37.992 INFO:tasks.workunit.client.0.smithi117.stdout:make all-recursive 2024-03-19T03:34:37.997 INFO:tasks.workunit.client.0.smithi117.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-19T03:34:38.004 INFO:tasks.workunit.client.0.smithi117.stdout:Making all in src 2024-03-19T03:34:38.012 INFO:tasks.workunit.client.0.smithi117.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/src' 2024-03-19T03:34:38.013 INFO:tasks.workunit.client.0.smithi117.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT blogbench.o -MD -MP -MF ".deps/blogbench.Tpo" -c -o blogbench.o blogbench.c; \ 2024-03-19T03:34:38.013 INFO:tasks.workunit.client.0.smithi117.stdout:then mv -f ".deps/blogbench.Tpo" ".deps/blogbench.Po"; else rm -f ".deps/blogbench.Tpo"; exit 1; fi 2024-03-19T03:34:38.185 INFO:tasks.workunit.client.0.smithi117.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT process.o -MD -MP -MF ".deps/process.Tpo" -c -o process.o process.c; \ 2024-03-19T03:34:38.185 INFO:tasks.workunit.client.0.smithi117.stdout:then mv -f ".deps/process.Tpo" ".deps/process.Po"; else rm -f ".deps/process.Tpo"; exit 1; fi 2024-03-19T03:34:38.301 INFO:tasks.workunit.client.0.smithi117.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT writer.o -MD -MP -MF ".deps/writer.Tpo" -c -o writer.o writer.c; \ 2024-03-19T03:34:38.301 INFO:tasks.workunit.client.0.smithi117.stdout:then mv -f ".deps/writer.Tpo" ".deps/writer.Po"; else rm -f ".deps/writer.Tpo"; exit 1; fi 2024-03-19T03:34:38.404 INFO:tasks.workunit.client.0.smithi117.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT rewriter.o -MD -MP -MF ".deps/rewriter.Tpo" -c -o rewriter.o rewriter.c; \ 2024-03-19T03:34:38.404 INFO:tasks.workunit.client.0.smithi117.stdout:then mv -f ".deps/rewriter.Tpo" ".deps/rewriter.Po"; else rm -f ".deps/rewriter.Tpo"; exit 1; fi 2024-03-19T03:34:38.500 INFO:tasks.workunit.client.0.smithi117.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT reader.o -MD -MP -MF ".deps/reader.Tpo" -c -o reader.o reader.c; \ 2024-03-19T03:34:38.500 INFO:tasks.workunit.client.0.smithi117.stdout:then mv -f ".deps/reader.Tpo" ".deps/reader.Po"; else rm -f ".deps/reader.Tpo"; exit 1; fi 2024-03-19T03:34:38.591 INFO:tasks.workunit.client.0.smithi117.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT commenter.o -MD -MP -MF ".deps/commenter.Tpo" -c -o commenter.o commenter.c; \ 2024-03-19T03:34:38.592 INFO:tasks.workunit.client.0.smithi117.stdout:then mv -f ".deps/commenter.Tpo" ".deps/commenter.Po"; else rm -f ".deps/commenter.Tpo"; exit 1; fi 2024-03-19T03:34:38.685 INFO:tasks.workunit.client.0.smithi117.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT helpers.o -MD -MP -MF ".deps/helpers.Tpo" -c -o helpers.o helpers.c; \ 2024-03-19T03:34:38.685 INFO:tasks.workunit.client.0.smithi117.stdout:then mv -f ".deps/helpers.Tpo" ".deps/helpers.Po"; else rm -f ".deps/helpers.Tpo"; exit 1; fi 2024-03-19T03:34:38.824 INFO:tasks.workunit.client.0.smithi117.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT bsd-getopt_long.o -MD -MP -MF ".deps/bsd-getopt_long.Tpo" -c -o bsd-getopt_long.o bsd-getopt_long.c; \ 2024-03-19T03:34:38.824 INFO:tasks.workunit.client.0.smithi117.stdout:then mv -f ".deps/bsd-getopt_long.Tpo" ".deps/bsd-getopt_long.Po"; else rm -f ".deps/bsd-getopt_long.Tpo"; exit 1; fi 2024-03-19T03:34:38.893 INFO:tasks.workunit.client.0.smithi117.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT fakesnprintf.o -MD -MP -MF ".deps/fakesnprintf.Tpo" -c -o fakesnprintf.o fakesnprintf.c; \ 2024-03-19T03:34:38.893 INFO:tasks.workunit.client.0.smithi117.stdout:then mv -f ".deps/fakesnprintf.Tpo" ".deps/fakesnprintf.Po"; else rm -f ".deps/fakesnprintf.Tpo"; exit 1; fi 2024-03-19T03:34:38.961 INFO:tasks.workunit.client.0.smithi117.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT mysnprintf.o -MD -MP -MF ".deps/mysnprintf.Tpo" -c -o mysnprintf.o mysnprintf.c; \ 2024-03-19T03:34:38.962 INFO:tasks.workunit.client.0.smithi117.stdout:then mv -f ".deps/mysnprintf.Tpo" ".deps/mysnprintf.Po"; else rm -f ".deps/mysnprintf.Tpo"; exit 1; fi 2024-03-19T03:34:39.042 INFO:tasks.workunit.client.0.smithi117.stdout:gcc -g -O2 -o blogbench blogbench.o process.o writer.o rewriter.o reader.o commenter.o helpers.o bsd-getopt_long.o fakesnprintf.o mysnprintf.o 2024-03-19T03:34:39.078 INFO:tasks.workunit.client.0.smithi117.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/src' 2024-03-19T03:34:39.079 INFO:tasks.workunit.client.0.smithi117.stdout:Making all in man 2024-03-19T03:34:39.108 INFO:tasks.workunit.client.0.smithi117.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/man' 2024-03-19T03:34:39.108 INFO:tasks.workunit.client.0.smithi117.stdout:make[2]: Nothing to be done for 'all'. 2024-03-19T03:34:39.108 INFO:tasks.workunit.client.0.smithi117.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/man' 2024-03-19T03:34:39.114 INFO:tasks.workunit.client.0.smithi117.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-19T03:34:39.117 INFO:tasks.workunit.client.0.smithi117.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-19T03:34:39.118 INFO:tasks.workunit.client.0.smithi117.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-19T03:34:39.119 INFO:tasks.workunit.client.0.smithi117.stderr:+ cd src 2024-03-19T03:34:39.119 INFO:tasks.workunit.client.0.smithi117.stderr:+ mkdir blogtest_in 2024-03-19T03:34:39.124 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'running blogbench' 2024-03-19T03:34:39.124 INFO:tasks.workunit.client.0.smithi117.stdout:running blogbench 2024-03-19T03:34:39.124 INFO:tasks.workunit.client.0.smithi117.stderr:+ ./blogbench -d blogtest_in 2024-03-19T03:34:39.125 INFO:tasks.workunit.client.0.smithi117.stdout: 2024-03-19T03:34:39.126 INFO:tasks.workunit.client.0.smithi117.stdout:Frequency = 10 secs 2024-03-19T03:34:39.126 INFO:tasks.workunit.client.0.smithi117.stdout:Scratch dir = [blogtest_in] 2024-03-19T03:34:39.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:39 smithi119 ceph-mon[66861]: pgmap v443: 129 pgs: 129 active+clean; 4.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 338 KiB/s wr, 34 op/s 2024-03-19T03:34:39.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:39 smithi117 ceph-mon[31838]: pgmap v443: 129 pgs: 129 active+clean; 4.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 338 KiB/s wr, 34 op/s 2024-03-19T03:34:39.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:39 smithi178 ceph-mon[67578]: pgmap v443: 129 pgs: 129 active+clean; 4.4 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 338 KiB/s wr, 34 op/s 2024-03-19T03:34:40.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:40 smithi119 ceph-mon[66861]: mgrmap e20: x(active, since 12m), standbys: z, y 2024-03-19T03:34:40.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:40 smithi117 ceph-mon[31838]: mgrmap e20: x(active, since 12m), standbys: z, y 2024-03-19T03:34:40.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:40 smithi178 ceph-mon[67578]: mgrmap e20: x(active, since 12m), standbys: z, y 2024-03-19T03:34:41.095 INFO:tasks.workunit.client.0.smithi117.stdout:Spawning 3 writers... 2024-03-19T03:34:41.096 INFO:tasks.workunit.client.0.smithi117.stdout:Spawning 1 rewriters... 2024-03-19T03:34:41.096 INFO:tasks.workunit.client.0.smithi117.stdout:Spawning 5 commenters... 2024-03-19T03:34:41.096 INFO:tasks.workunit.client.0.smithi117.stdout:Spawning 100 readers... 2024-03-19T03:34:41.104 INFO:tasks.workunit.client.0.smithi117.stdout:Benchmarking for 30 iterations. 2024-03-19T03:34:41.104 INFO:tasks.workunit.client.0.smithi117.stdout:The test will run during 5 minutes. 2024-03-19T03:34:41.104 INFO:tasks.workunit.client.0.smithi117.stdout: 2024-03-19T03:34:41.104 INFO:tasks.workunit.client.0.smithi117.stdout: Nb blogs R articles W articles R pictures W pictures R comments W comments 2024-03-19T03:34:41.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:41 smithi119 ceph-mon[66861]: pgmap v444: 129 pgs: 129 active+clean; 4.5 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 327 KiB/s wr, 38 op/s 2024-03-19T03:34:41.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:41 smithi117 ceph-mon[31838]: pgmap v444: 129 pgs: 129 active+clean; 4.5 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 327 KiB/s wr, 38 op/s 2024-03-19T03:34:41.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:41 smithi178 ceph-mon[67578]: pgmap v444: 129 pgs: 129 active+clean; 4.5 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 327 KiB/s wr, 38 op/s 2024-03-19T03:34:43.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:43 smithi119 ceph-mon[66861]: pgmap v445: 129 pgs: 129 active+clean; 4.9 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 393 KiB/s wr, 57 op/s 2024-03-19T03:34:43.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:43 smithi117 ceph-mon[31838]: pgmap v445: 129 pgs: 129 active+clean; 4.9 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 393 KiB/s wr, 57 op/s 2024-03-19T03:34:43.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:43 smithi178 ceph-mon[67578]: pgmap v445: 129 pgs: 129 active+clean; 4.9 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 393 KiB/s wr, 57 op/s 2024-03-19T03:34:45.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:45 smithi119 ceph-mon[66861]: pgmap v446: 129 pgs: 129 active+clean; 6.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 492 KiB/s wr, 55 op/s 2024-03-19T03:34:45.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:45 smithi117 ceph-mon[31838]: pgmap v446: 129 pgs: 129 active+clean; 6.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 492 KiB/s wr, 55 op/s 2024-03-19T03:34:45.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:45 smithi178 ceph-mon[67578]: pgmap v446: 129 pgs: 129 active+clean; 6.1 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 492 KiB/s wr, 55 op/s 2024-03-19T03:34:47.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:47 smithi119 ceph-mon[66861]: pgmap v447: 129 pgs: 129 active+clean; 11 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 847 KiB/s wr, 60 op/s 2024-03-19T03:34:47.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:47 smithi117 ceph-mon[31838]: pgmap v447: 129 pgs: 129 active+clean; 11 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 847 KiB/s wr, 60 op/s 2024-03-19T03:34:47.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:47 smithi178 ceph-mon[67578]: pgmap v447: 129 pgs: 129 active+clean; 11 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 847 KiB/s wr, 60 op/s 2024-03-19T03:34:49.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:49 smithi119 ceph-mon[66861]: pgmap v448: 129 pgs: 129 active+clean; 12 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 838 KiB/s wr, 58 op/s 2024-03-19T03:34:49.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:49 smithi119 ceph-mon[66861]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:34:49.794 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:49 smithi117 ceph-mon[31838]: pgmap v448: 129 pgs: 129 active+clean; 12 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 838 KiB/s wr, 58 op/s 2024-03-19T03:34:49.794 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:49 smithi117 ceph-mon[31838]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:34:49.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:49 smithi178 ceph-mon[67578]: pgmap v448: 129 pgs: 129 active+clean; 12 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 838 KiB/s wr, 58 op/s 2024-03-19T03:34:49.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:49 smithi178 ceph-mon[67578]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:34:50.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:50 smithi117 ceph-mon[31838]: pgmap v449: 129 pgs: 129 active+clean; 15 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 1.0 MiB/s wr, 66 op/s 2024-03-19T03:34:50.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:50 smithi178 ceph-mon[67578]: pgmap v449: 129 pgs: 129 active+clean; 15 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 1.0 MiB/s wr, 66 op/s 2024-03-19T03:34:50.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:50 smithi119 ceph-mon[66861]: pgmap v449: 129 pgs: 129 active+clean; 15 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 1.0 MiB/s wr, 66 op/s 2024-03-19T03:34:51.104 INFO:tasks.workunit.client.0.smithi117.stdout: 16 244278 713 142158 808 34826 676 2024-03-19T03:34:53.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:53 smithi119 ceph-mon[66861]: pgmap v450: 129 pgs: 129 active+clean; 32 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 119 op/s 2024-03-19T03:34:53.795 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:53 smithi117 ceph-mon[31838]: pgmap v450: 129 pgs: 129 active+clean; 32 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 119 op/s 2024-03-19T03:34:53.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:53 smithi178 ceph-mon[67578]: pgmap v450: 129 pgs: 129 active+clean; 32 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 119 op/s 2024-03-19T03:34:55.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:55 smithi119 ceph-mon[66861]: pgmap v451: 129 pgs: 129 active+clean; 39 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.1 MiB/s wr, 117 op/s 2024-03-19T03:34:55.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:55 smithi117 ceph-mon[31838]: pgmap v451: 129 pgs: 129 active+clean; 39 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.1 MiB/s wr, 117 op/s 2024-03-19T03:34:55.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:55 smithi178 ceph-mon[67578]: pgmap v451: 129 pgs: 129 active+clean; 39 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.1 MiB/s wr, 117 op/s 2024-03-19T03:34:57.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:57 smithi119 ceph-mon[66861]: pgmap v452: 129 pgs: 129 active+clean; 49 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 154 op/s 2024-03-19T03:34:57.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:57 smithi117 ceph-mon[31838]: pgmap v452: 129 pgs: 129 active+clean; 49 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 154 op/s 2024-03-19T03:34:57.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:57 smithi178 ceph-mon[67578]: pgmap v452: 129 pgs: 129 active+clean; 49 MiB data, 433 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 154 op/s 2024-03-19T03:34:59.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:34:59 smithi119 ceph-mon[66861]: pgmap v453: 129 pgs: 129 active+clean; 52 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 159 op/s 2024-03-19T03:34:59.814 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:34:59 smithi117 ceph-mon[31838]: pgmap v453: 129 pgs: 129 active+clean; 52 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 159 op/s 2024-03-19T03:34:59.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:34:59 smithi178 ceph-mon[67578]: pgmap v453: 129 pgs: 129 active+clean; 52 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 159 op/s 2024-03-19T03:35:01.109 INFO:tasks.workunit.client.0.smithi117.stdout: 22 318015 420 203937 423 147777 1108 2024-03-19T03:35:01.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:01 smithi119 ceph-mon[66861]: pgmap v454: 129 pgs: 129 active+clean; 56 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 164 op/s 2024-03-19T03:35:01.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:01 smithi117 ceph-mon[31838]: pgmap v454: 129 pgs: 129 active+clean; 56 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 164 op/s 2024-03-19T03:35:01.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:01 smithi178 ceph-mon[67578]: pgmap v454: 129 pgs: 129 active+clean; 56 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 164 op/s 2024-03-19T03:35:03.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:03 smithi119 ceph-mon[66861]: pgmap v455: 129 pgs: 129 active+clean; 56 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 170 op/s 2024-03-19T03:35:03.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:03 smithi117 ceph-mon[31838]: pgmap v455: 129 pgs: 129 active+clean; 56 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 170 op/s 2024-03-19T03:35:03.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:03 smithi178 ceph-mon[67578]: pgmap v455: 129 pgs: 129 active+clean; 56 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 170 op/s 2024-03-19T03:35:05.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:05 smithi119 ceph-mon[66861]: pgmap v456: 129 pgs: 129 active+clean; 62 MiB data, 443 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 122 op/s 2024-03-19T03:35:05.801 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:05 smithi117 ceph-mon[31838]: pgmap v456: 129 pgs: 129 active+clean; 62 MiB data, 443 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 122 op/s 2024-03-19T03:35:05.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:05 smithi178 ceph-mon[67578]: pgmap v456: 129 pgs: 129 active+clean; 62 MiB data, 443 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 122 op/s 2024-03-19T03:35:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:07 smithi119 ceph-mon[66861]: pgmap v457: 129 pgs: 129 active+clean; 62 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.3 MiB/s wr, 115 op/s 2024-03-19T03:35:07.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:07 smithi117 ceph-mon[31838]: pgmap v457: 129 pgs: 129 active+clean; 62 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.3 MiB/s wr, 115 op/s 2024-03-19T03:35:07.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:07 smithi178 ceph-mon[67578]: pgmap v457: 129 pgs: 129 active+clean; 62 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.3 MiB/s wr, 115 op/s 2024-03-19T03:35:09.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:09 smithi117 ceph-mon[31838]: pgmap v458: 129 pgs: 129 active+clean; 63 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-03-19T03:35:09.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:09 smithi178 ceph-mon[67578]: pgmap v458: 129 pgs: 129 active+clean; 63 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-03-19T03:35:09.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:09 smithi119 ceph-mon[66861]: pgmap v458: 129 pgs: 129 active+clean; 63 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-03-19T03:35:11.129 INFO:tasks.workunit.client.0.smithi117.stdout: 29 434520 377 304200 397 222678 1124 2024-03-19T03:35:11.801 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:11 smithi117 ceph-mon[31838]: pgmap v459: 129 pgs: 129 active+clean; 69 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 88 op/s 2024-03-19T03:35:11.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:11 smithi178 ceph-mon[67578]: pgmap v459: 129 pgs: 129 active+clean; 69 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 88 op/s 2024-03-19T03:35:11.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:11 smithi119 ceph-mon[66861]: pgmap v459: 129 pgs: 129 active+clean; 69 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 88 op/s 2024-03-19T03:35:12.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:12 smithi117 ceph-mon[31838]: pgmap v460: 129 pgs: 129 active+clean; 82 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 140 op/s 2024-03-19T03:35:12.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:12 smithi178 ceph-mon[67578]: pgmap v460: 129 pgs: 129 active+clean; 82 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 140 op/s 2024-03-19T03:35:12.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:12 smithi119 ceph-mon[66861]: pgmap v460: 129 pgs: 129 active+clean; 82 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 140 op/s 2024-03-19T03:35:15.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:15 smithi119 ceph-mon[66861]: pgmap v461: 129 pgs: 129 active+clean; 97 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 197 op/s 2024-03-19T03:35:15.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:15 smithi117 ceph-mon[31838]: pgmap v461: 129 pgs: 129 active+clean; 97 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 197 op/s 2024-03-19T03:35:15.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:15 smithi178 ceph-mon[67578]: pgmap v461: 129 pgs: 129 active+clean; 97 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 197 op/s 2024-03-19T03:35:17.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:17 smithi119 ceph-mon[66861]: pgmap v462: 129 pgs: 129 active+clean; 127 MiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.9 MiB/s wr, 357 op/s 2024-03-19T03:35:17.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:17 smithi117 ceph-mon[31838]: pgmap v462: 129 pgs: 129 active+clean; 127 MiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.9 MiB/s wr, 357 op/s 2024-03-19T03:35:17.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:17 smithi178 ceph-mon[67578]: pgmap v462: 129 pgs: 129 active+clean; 127 MiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.9 MiB/s wr, 357 op/s 2024-03-19T03:35:19.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:19 smithi119 ceph-mon[66861]: pgmap v463: 129 pgs: 129 active+clean; 137 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.8 MiB/s wr, 421 op/s 2024-03-19T03:35:19.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:19 smithi178 ceph-mon[67578]: pgmap v463: 129 pgs: 129 active+clean; 137 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.8 MiB/s wr, 421 op/s 2024-03-19T03:35:19.846 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:19 smithi117 ceph-mon[31838]: pgmap v463: 129 pgs: 129 active+clean; 137 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.8 MiB/s wr, 421 op/s 2024-03-19T03:35:21.119 INFO:tasks.workunit.client.0.smithi117.stdout: 32 277812 262 183634 142 150287 582 2024-03-19T03:35:21.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:21 smithi119 ceph-mon[66861]: pgmap v464: 129 pgs: 129 active+clean; 139 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 7.0 MiB/s wr, 437 op/s 2024-03-19T03:35:21.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:21 smithi117 ceph-mon[31838]: pgmap v464: 129 pgs: 129 active+clean; 139 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 7.0 MiB/s wr, 437 op/s 2024-03-19T03:35:21.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:21 smithi178 ceph-mon[67578]: pgmap v464: 129 pgs: 129 active+clean; 139 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 7.0 MiB/s wr, 437 op/s 2024-03-19T03:35:23.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:23 smithi119 ceph-mon[66861]: pgmap v465: 129 pgs: 129 active+clean; 138 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.8 MiB/s wr, 475 op/s 2024-03-19T03:35:23.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:23 smithi117 ceph-mon[31838]: pgmap v465: 129 pgs: 129 active+clean; 138 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.8 MiB/s wr, 475 op/s 2024-03-19T03:35:23.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:23 smithi178 ceph-mon[67578]: pgmap v465: 129 pgs: 129 active+clean; 138 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.8 MiB/s wr, 475 op/s 2024-03-19T03:35:25.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:25 smithi119 ceph-mon[66861]: pgmap v466: 129 pgs: 129 active+clean; 138 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.6 MiB/s wr, 420 op/s 2024-03-19T03:35:25.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:25 smithi117 ceph-mon[31838]: pgmap v466: 129 pgs: 129 active+clean; 138 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.6 MiB/s wr, 420 op/s 2024-03-19T03:35:25.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:25 smithi178 ceph-mon[67578]: pgmap v466: 129 pgs: 129 active+clean; 138 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.6 MiB/s wr, 420 op/s 2024-03-19T03:35:27.798 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:27 smithi117 ceph-mon[31838]: pgmap v467: 129 pgs: 129 active+clean; 142 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 357 op/s 2024-03-19T03:35:27.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:27 smithi178 ceph-mon[67578]: pgmap v467: 129 pgs: 129 active+clean; 142 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 357 op/s 2024-03-19T03:35:27.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:27 smithi119 ceph-mon[66861]: pgmap v467: 129 pgs: 129 active+clean; 142 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 357 op/s 2024-03-19T03:35:29.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:29 smithi117 ceph-mon[31838]: pgmap v468: 129 pgs: 129 active+clean; 142 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 192 op/s 2024-03-19T03:35:29.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:29 smithi178 ceph-mon[67578]: pgmap v468: 129 pgs: 129 active+clean; 142 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 192 op/s 2024-03-19T03:35:29.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:29 smithi119 ceph-mon[66861]: pgmap v468: 129 pgs: 129 active+clean; 142 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 192 op/s 2024-03-19T03:35:30.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:30 smithi178 ceph-mon[67578]: pgmap v469: 129 pgs: 129 active+clean; 144 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 142 op/s 2024-03-19T03:35:30.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:30 smithi119 ceph-mon[66861]: pgmap v469: 129 pgs: 129 active+clean; 144 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 142 op/s 2024-03-19T03:35:31.056 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:30 smithi117 ceph-mon[31838]: pgmap v469: 129 pgs: 129 active+clean; 144 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 142 op/s 2024-03-19T03:35:31.123 INFO:tasks.workunit.client.0.smithi117.stdout: 38 321863 401 212199 300 170732 774 2024-03-19T03:35:32.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:31 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:35:32.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:31 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:35:32.415 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:31 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:35:33.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:32 smithi119 ceph-mon[66861]: pgmap v470: 129 pgs: 129 active+clean; 158 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 227 op/s 2024-03-19T03:35:33.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:32 smithi178 ceph-mon[67578]: pgmap v470: 129 pgs: 129 active+clean; 158 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 227 op/s 2024-03-19T03:35:33.388 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:32 smithi117 ceph-mon[31838]: pgmap v470: 129 pgs: 129 active+clean; 158 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 227 op/s 2024-03-19T03:35:35.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:35 smithi119 ceph-mon[66861]: pgmap v471: 129 pgs: 129 active+clean; 161 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 194 op/s 2024-03-19T03:35:35.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:35 smithi178 ceph-mon[67578]: pgmap v471: 129 pgs: 129 active+clean; 161 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 194 op/s 2024-03-19T03:35:35.971 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:35 smithi117 ceph-mon[31838]: pgmap v471: 129 pgs: 129 active+clean; 161 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 194 op/s 2024-03-19T03:35:37.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:37 smithi119 ceph-mon[66861]: pgmap v472: 129 pgs: 129 active+clean; 178 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 278 op/s 2024-03-19T03:35:37.824 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:37 smithi117 ceph-mon[31838]: pgmap v472: 129 pgs: 129 active+clean; 178 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 278 op/s 2024-03-19T03:35:37.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:37 smithi178 ceph-mon[67578]: pgmap v472: 129 pgs: 129 active+clean; 178 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 278 op/s 2024-03-19T03:35:39.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:39 smithi119 ceph-mon[66861]: pgmap v473: 129 pgs: 129 active+clean; 184 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 4.0 MiB/s wr, 299 op/s 2024-03-19T03:35:39.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:39 smithi119 ceph-mon[66861]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:35:39.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:39 smithi178 ceph-mon[67578]: pgmap v473: 129 pgs: 129 active+clean; 184 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 4.0 MiB/s wr, 299 op/s 2024-03-19T03:35:39.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:39 smithi178 ceph-mon[67578]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:35:39.892 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:39 smithi117 ceph-mon[31838]: pgmap v473: 129 pgs: 129 active+clean; 184 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 4.0 MiB/s wr, 299 op/s 2024-03-19T03:35:39.892 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:39 smithi117 ceph-mon[31838]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:35:41.123 INFO:tasks.workunit.client.0.smithi117.stdout: 48 493994 516 318715 500 232207 549 2024-03-19T03:35:41.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:41 smithi117 ceph-mon[31838]: pgmap v474: 129 pgs: 129 active+clean; 187 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 4.4 MiB/s wr, 324 op/s 2024-03-19T03:35:41.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:35:41.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:35:41.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:35:41.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:41 smithi178 ceph-mon[67578]: pgmap v474: 129 pgs: 129 active+clean; 187 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 4.4 MiB/s wr, 324 op/s 2024-03-19T03:35:41.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:35:41.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:35:41.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:35:41.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:41 smithi119 ceph-mon[66861]: pgmap v474: 129 pgs: 129 active+clean; 187 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 4.4 MiB/s wr, 324 op/s 2024-03-19T03:35:41.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:35:41.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:35:41.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:35:43.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:43 smithi117 ceph-mon[31838]: pgmap v475: 129 pgs: 129 active+clean; 192 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.7 MiB/s wr, 334 op/s 2024-03-19T03:35:43.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:43 smithi178 ceph-mon[67578]: pgmap v475: 129 pgs: 129 active+clean; 192 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.7 MiB/s wr, 334 op/s 2024-03-19T03:35:43.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:43 smithi119 ceph-mon[66861]: pgmap v475: 129 pgs: 129 active+clean; 192 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.7 MiB/s wr, 334 op/s 2024-03-19T03:35:44.821 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:44 smithi117 ceph-mon[31838]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:35:44.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:44 smithi178 ceph-mon[67578]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:35:44.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:44 smithi119 ceph-mon[66861]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:35:45.800 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:45 smithi117 ceph-mon[31838]: pgmap v476: 129 pgs: 129 active+clean; 192 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 236 op/s 2024-03-19T03:35:45.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:45 smithi178 ceph-mon[67578]: pgmap v476: 129 pgs: 129 active+clean; 192 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 236 op/s 2024-03-19T03:35:45.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:45 smithi119 ceph-mon[66861]: pgmap v476: 129 pgs: 129 active+clean; 192 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.4 MiB/s wr, 236 op/s 2024-03-19T03:35:46.797 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:46 smithi117 ceph-mon[31838]: pgmap v477: 129 pgs: 129 active+clean; 193 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.3 MiB/s wr, 232 op/s 2024-03-19T03:35:46.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:46 smithi178 ceph-mon[67578]: pgmap v477: 129 pgs: 129 active+clean; 193 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.3 MiB/s wr, 232 op/s 2024-03-19T03:35:46.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:46 smithi119 ceph-mon[66861]: pgmap v477: 129 pgs: 129 active+clean; 193 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.3 MiB/s wr, 232 op/s 2024-03-19T03:35:49.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:49 smithi119 ceph-mon[66861]: pgmap v478: 129 pgs: 129 active+clean; 193 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 165 op/s 2024-03-19T03:35:49.798 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:49 smithi117 ceph-mon[31838]: pgmap v478: 129 pgs: 129 active+clean; 193 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 165 op/s 2024-03-19T03:35:49.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:49 smithi178 ceph-mon[67578]: pgmap v478: 129 pgs: 129 active+clean; 193 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 165 op/s 2024-03-19T03:35:51.135 INFO:tasks.workunit.client.0.smithi117.stdout: 54 411624 441 256772 272 169897 805 2024-03-19T03:35:51.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:51 smithi119 ceph-mon[66861]: pgmap v479: 129 pgs: 129 active+clean; 197 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 141 op/s 2024-03-19T03:35:51.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:51 smithi117 ceph-mon[31838]: pgmap v479: 129 pgs: 129 active+clean; 197 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 141 op/s 2024-03-19T03:35:51.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:51 smithi178 ceph-mon[67578]: pgmap v479: 129 pgs: 129 active+clean; 197 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 141 op/s 2024-03-19T03:35:53.794 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:53 smithi117 ceph-mon[31838]: pgmap v480: 129 pgs: 129 active+clean; 214 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 200 op/s 2024-03-19T03:35:53.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:53 smithi178 ceph-mon[67578]: pgmap v480: 129 pgs: 129 active+clean; 214 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 200 op/s 2024-03-19T03:35:53.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:53 smithi119 ceph-mon[66861]: pgmap v480: 129 pgs: 129 active+clean; 214 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 200 op/s 2024-03-19T03:35:55.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:55 smithi117 ceph-mon[31838]: pgmap v481: 129 pgs: 129 active+clean; 223 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 196 op/s 2024-03-19T03:35:55.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:55 smithi178 ceph-mon[67578]: pgmap v481: 129 pgs: 129 active+clean; 223 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 196 op/s 2024-03-19T03:35:55.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:55 smithi119 ceph-mon[66861]: pgmap v481: 129 pgs: 129 active+clean; 223 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 196 op/s 2024-03-19T03:35:57.786 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:57 smithi119 ceph-mon[66861]: pgmap v482: 129 pgs: 129 active+clean; 241 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 296 op/s 2024-03-19T03:35:57.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:57 smithi117 ceph-mon[31838]: pgmap v482: 129 pgs: 129 active+clean; 241 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 296 op/s 2024-03-19T03:35:57.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:57 smithi178 ceph-mon[67578]: pgmap v482: 129 pgs: 129 active+clean; 241 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.6 MiB/s wr, 296 op/s 2024-03-19T03:35:59.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:35:59 smithi117 ceph-mon[31838]: pgmap v483: 129 pgs: 129 active+clean; 252 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.5 MiB/s wr, 325 op/s 2024-03-19T03:35:59.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:35:59 smithi178 ceph-mon[67578]: pgmap v483: 129 pgs: 129 active+clean; 252 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.5 MiB/s wr, 325 op/s 2024-03-19T03:35:59.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:35:59 smithi119 ceph-mon[66861]: pgmap v483: 129 pgs: 129 active+clean; 252 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.5 MiB/s wr, 325 op/s 2024-03-19T03:36:01.136 INFO:tasks.workunit.client.0.smithi117.stdout: 57 392911 235 238022 275 176735 1010 2024-03-19T03:36:01.795 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:01 smithi117 ceph-mon[31838]: pgmap v484: 129 pgs: 129 active+clean; 259 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.1 MiB/s wr, 353 op/s 2024-03-19T03:36:01.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:01 smithi178 ceph-mon[67578]: pgmap v484: 129 pgs: 129 active+clean; 259 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.1 MiB/s wr, 353 op/s 2024-03-19T03:36:01.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:01 smithi119 ceph-mon[66861]: pgmap v484: 129 pgs: 129 active+clean; 259 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.1 MiB/s wr, 353 op/s 2024-03-19T03:36:02.799 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:02 smithi117 ceph-mon[31838]: pgmap v485: 129 pgs: 129 active+clean; 262 MiB data, 613 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.1 MiB/s wr, 376 op/s 2024-03-19T03:36:02.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:02 smithi178 ceph-mon[67578]: pgmap v485: 129 pgs: 129 active+clean; 262 MiB data, 613 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.1 MiB/s wr, 376 op/s 2024-03-19T03:36:02.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:02 smithi119 ceph-mon[66861]: pgmap v485: 129 pgs: 129 active+clean; 262 MiB data, 613 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.1 MiB/s wr, 376 op/s 2024-03-19T03:36:05.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:05 smithi119 ceph-mon[66861]: pgmap v486: 129 pgs: 129 active+clean; 265 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.8 MiB/s wr, 299 op/s 2024-03-19T03:36:05.796 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:05 smithi117 ceph-mon[31838]: pgmap v486: 129 pgs: 129 active+clean; 265 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.8 MiB/s wr, 299 op/s 2024-03-19T03:36:05.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:05 smithi178 ceph-mon[67578]: pgmap v486: 129 pgs: 129 active+clean; 265 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.8 MiB/s wr, 299 op/s 2024-03-19T03:36:07.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:07 smithi119 ceph-mon[66861]: pgmap v487: 129 pgs: 129 active+clean; 266 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 291 op/s 2024-03-19T03:36:07.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:07 smithi117 ceph-mon[31838]: pgmap v487: 129 pgs: 129 active+clean; 266 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 291 op/s 2024-03-19T03:36:07.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:07 smithi178 ceph-mon[67578]: pgmap v487: 129 pgs: 129 active+clean; 266 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 291 op/s 2024-03-19T03:36:09.801 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:09 smithi117 ceph-mon[31838]: pgmap v488: 129 pgs: 129 active+clean; 266 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 192 op/s 2024-03-19T03:36:09.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:09 smithi178 ceph-mon[67578]: pgmap v488: 129 pgs: 129 active+clean; 266 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 192 op/s 2024-03-19T03:36:09.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:09 smithi119 ceph-mon[66861]: pgmap v488: 129 pgs: 129 active+clean; 266 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 192 op/s 2024-03-19T03:36:11.151 INFO:tasks.workunit.client.0.smithi117.stdout: 63 508878 310 325617 448 246964 1227 2024-03-19T03:36:11.795 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:11 smithi117 ceph-mon[31838]: pgmap v489: 129 pgs: 129 active+clean; 265 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 1.6 MiB/s wr, 142 op/s 2024-03-19T03:36:11.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:11 smithi178 ceph-mon[67578]: pgmap v489: 129 pgs: 129 active+clean; 265 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 1.6 MiB/s wr, 142 op/s 2024-03-19T03:36:11.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:11 smithi119 ceph-mon[66861]: pgmap v489: 129 pgs: 129 active+clean; 265 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 1.6 MiB/s wr, 142 op/s 2024-03-19T03:36:13.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:13 smithi117 ceph-mon[31838]: pgmap v490: 129 pgs: 129 active+clean; 271 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 113 op/s 2024-03-19T03:36:13.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:13 smithi178 ceph-mon[67578]: pgmap v490: 129 pgs: 129 active+clean; 271 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 113 op/s 2024-03-19T03:36:13.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:13 smithi119 ceph-mon[66861]: pgmap v490: 129 pgs: 129 active+clean; 271 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 113 op/s 2024-03-19T03:36:15.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:15 smithi117 ceph-mon[31838]: pgmap v491: 129 pgs: 129 active+clean; 274 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 84 op/s 2024-03-19T03:36:15.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:15 smithi178 ceph-mon[67578]: pgmap v491: 129 pgs: 129 active+clean; 274 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 84 op/s 2024-03-19T03:36:15.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:15 smithi119 ceph-mon[66861]: pgmap v491: 129 pgs: 129 active+clean; 274 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 84 op/s 2024-03-19T03:36:16.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:16 smithi178 ceph-mon[67578]: pgmap v492: 129 pgs: 129 active+clean; 277 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 111 op/s 2024-03-19T03:36:16.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:16 smithi119 ceph-mon[66861]: pgmap v492: 129 pgs: 129 active+clean; 277 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 111 op/s 2024-03-19T03:36:17.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:16 smithi117 ceph-mon[31838]: pgmap v492: 129 pgs: 129 active+clean; 277 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 111 op/s 2024-03-19T03:36:19.794 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:19 smithi117 ceph-mon[31838]: pgmap v493: 129 pgs: 129 active+clean; 287 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 126 op/s 2024-03-19T03:36:19.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:19 smithi178 ceph-mon[67578]: pgmap v493: 129 pgs: 129 active+clean; 287 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 126 op/s 2024-03-19T03:36:19.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:19 smithi119 ceph-mon[66861]: pgmap v493: 129 pgs: 129 active+clean; 287 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 126 op/s 2024-03-19T03:36:21.198 INFO:tasks.workunit.client.0.smithi117.stdout: 68 481061 339 315589 344 263540 956 2024-03-19T03:36:21.795 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:21 smithi117 ceph-mon[31838]: pgmap v494: 129 pgs: 129 active+clean; 291 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 154 op/s 2024-03-19T03:36:21.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:21 smithi178 ceph-mon[67578]: pgmap v494: 129 pgs: 129 active+clean; 291 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 154 op/s 2024-03-19T03:36:21.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:21 smithi119 ceph-mon[66861]: pgmap v494: 129 pgs: 129 active+clean; 291 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 154 op/s 2024-03-19T03:36:23.809 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:23 smithi117 ceph-mon[31838]: pgmap v495: 129 pgs: 129 active+clean; 300 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.5 MiB/s wr, 249 op/s 2024-03-19T03:36:23.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:23 smithi178 ceph-mon[67578]: pgmap v495: 129 pgs: 129 active+clean; 300 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.5 MiB/s wr, 249 op/s 2024-03-19T03:36:23.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:23 smithi119 ceph-mon[66861]: pgmap v495: 129 pgs: 129 active+clean; 300 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.5 MiB/s wr, 249 op/s 2024-03-19T03:36:25.796 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:25 smithi117 ceph-mon[31838]: pgmap v496: 129 pgs: 129 active+clean; 314 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.2 MiB/s wr, 307 op/s 2024-03-19T03:36:25.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:25 smithi178 ceph-mon[67578]: pgmap v496: 129 pgs: 129 active+clean; 314 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.2 MiB/s wr, 307 op/s 2024-03-19T03:36:25.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:25 smithi119 ceph-mon[66861]: pgmap v496: 129 pgs: 129 active+clean; 314 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.2 MiB/s wr, 307 op/s 2024-03-19T03:36:26.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:26 smithi178 ceph-mon[67578]: pgmap v497: 129 pgs: 129 active+clean; 336 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.9 MiB/s wr, 400 op/s 2024-03-19T03:36:26.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:26 smithi119 ceph-mon[66861]: pgmap v497: 129 pgs: 129 active+clean; 336 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.9 MiB/s wr, 400 op/s 2024-03-19T03:36:27.063 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:26 smithi117 ceph-mon[31838]: pgmap v497: 129 pgs: 129 active+clean; 336 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.9 MiB/s wr, 400 op/s 2024-03-19T03:36:29.798 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:29 smithi117 ceph-mon[31838]: pgmap v498: 129 pgs: 129 active+clean; 347 MiB data, 781 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.6 MiB/s wr, 464 op/s 2024-03-19T03:36:29.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:29 smithi178 ceph-mon[67578]: pgmap v498: 129 pgs: 129 active+clean; 347 MiB data, 781 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.6 MiB/s wr, 464 op/s 2024-03-19T03:36:29.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:29 smithi119 ceph-mon[66861]: pgmap v498: 129 pgs: 129 active+clean; 347 MiB data, 781 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.6 MiB/s wr, 464 op/s 2024-03-19T03:36:31.209 INFO:tasks.workunit.client.0.smithi117.stdout: 72 454073 251 304944 230 263756 733 2024-03-19T03:36:31.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:31 smithi117 ceph-mon[31838]: pgmap v499: 129 pgs: 129 active+clean; 347 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.9 MiB/s wr, 443 op/s 2024-03-19T03:36:31.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:31 smithi178 ceph-mon[67578]: pgmap v499: 129 pgs: 129 active+clean; 347 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.9 MiB/s wr, 443 op/s 2024-03-19T03:36:31.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:31 smithi119 ceph-mon[66861]: pgmap v499: 129 pgs: 129 active+clean; 347 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.9 MiB/s wr, 443 op/s 2024-03-19T03:36:33.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:33 smithi117 ceph-mon[31838]: pgmap v500: 129 pgs: 129 active+clean; 348 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.7 MiB/s wr, 437 op/s 2024-03-19T03:36:33.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:33 smithi178 ceph-mon[67578]: pgmap v500: 129 pgs: 129 active+clean; 348 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.7 MiB/s wr, 437 op/s 2024-03-19T03:36:33.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:33 smithi119 ceph-mon[66861]: pgmap v500: 129 pgs: 129 active+clean; 348 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.7 MiB/s wr, 437 op/s 2024-03-19T03:36:34.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:34 smithi117 ceph-mon[31838]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:36:34.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:34 smithi178 ceph-mon[67578]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:36:34.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:34 smithi119 ceph-mon[66861]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:36:35.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:35 smithi117 ceph-mon[31838]: pgmap v501: 129 pgs: 129 active+clean; 347 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.7 MiB/s wr, 346 op/s 2024-03-19T03:36:35.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:35 smithi178 ceph-mon[67578]: pgmap v501: 129 pgs: 129 active+clean; 347 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.7 MiB/s wr, 346 op/s 2024-03-19T03:36:35.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:35 smithi119 ceph-mon[66861]: pgmap v501: 129 pgs: 129 active+clean; 347 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.7 MiB/s wr, 346 op/s 2024-03-19T03:36:36.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:36 smithi117 ceph-mon[31838]: pgmap v502: 129 pgs: 129 active+clean; 349 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 279 op/s 2024-03-19T03:36:36.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:36 smithi178 ceph-mon[67578]: pgmap v502: 129 pgs: 129 active+clean; 349 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 279 op/s 2024-03-19T03:36:36.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:36 smithi119 ceph-mon[66861]: pgmap v502: 129 pgs: 129 active+clean; 349 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 279 op/s 2024-03-19T03:36:39.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:39 smithi117 ceph-mon[31838]: pgmap v503: 129 pgs: 129 active+clean; 353 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 213 op/s 2024-03-19T03:36:39.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:39 smithi178 ceph-mon[67578]: pgmap v503: 129 pgs: 129 active+clean; 353 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 213 op/s 2024-03-19T03:36:39.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:39 smithi119 ceph-mon[66861]: pgmap v503: 129 pgs: 129 active+clean; 353 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 213 op/s 2024-03-19T03:36:41.302 INFO:tasks.workunit.client.0.smithi117.stdout: 79 449003 456 297621 321 261465 1046 2024-03-19T03:36:41.801 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:41 smithi117 ceph-mon[31838]: pgmap v504: 129 pgs: 129 active+clean; 352 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 127 op/s 2024-03-19T03:36:41.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:41 smithi178 ceph-mon[67578]: pgmap v504: 129 pgs: 129 active+clean; 352 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 127 op/s 2024-03-19T03:36:41.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:41 smithi119 ceph-mon[66861]: pgmap v504: 129 pgs: 129 active+clean; 352 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 127 op/s 2024-03-19T03:36:42.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:42 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:36:42.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:42 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:36:42.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:42 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:36:43.795 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:43 smithi117 ceph-mon[31838]: pgmap v505: 129 pgs: 129 active+clean; 358 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 139 op/s 2024-03-19T03:36:43.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:43 smithi178 ceph-mon[67578]: pgmap v505: 129 pgs: 129 active+clean; 358 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 139 op/s 2024-03-19T03:36:43.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:43 smithi119 ceph-mon[66861]: pgmap v505: 129 pgs: 129 active+clean; 358 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 139 op/s 2024-03-19T03:36:44.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:44 smithi178 ceph-mon[67578]: pgmap v506: 129 pgs: 129 active+clean; 358 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 122 op/s 2024-03-19T03:36:44.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:44 smithi119 ceph-mon[66861]: pgmap v506: 129 pgs: 129 active+clean; 358 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 122 op/s 2024-03-19T03:36:45.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:44 smithi117 ceph-mon[31838]: pgmap v506: 129 pgs: 129 active+clean; 358 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 122 op/s 2024-03-19T03:36:47.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:47 smithi117 ceph-mon[31838]: pgmap v507: 129 pgs: 129 active+clean; 358 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-03-19T03:36:47.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:47 smithi178 ceph-mon[67578]: pgmap v507: 129 pgs: 129 active+clean; 358 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-03-19T03:36:47.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:47 smithi119 ceph-mon[66861]: pgmap v507: 129 pgs: 129 active+clean; 358 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-03-19T03:36:49.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:49 smithi117 ceph-mon[31838]: pgmap v508: 129 pgs: 129 active+clean; 365 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 159 op/s 2024-03-19T03:36:49.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:49 smithi178 ceph-mon[67578]: pgmap v508: 129 pgs: 129 active+clean; 365 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 159 op/s 2024-03-19T03:36:49.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:49 smithi119 ceph-mon[66861]: pgmap v508: 129 pgs: 129 active+clean; 365 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 159 op/s 2024-03-19T03:36:50.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:50 smithi178 ceph-mon[67578]: pgmap v509: 129 pgs: 129 active+clean; 376 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 217 op/s 2024-03-19T03:36:50.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:50 smithi119 ceph-mon[66861]: pgmap v509: 129 pgs: 129 active+clean; 376 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 217 op/s 2024-03-19T03:36:51.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:50 smithi117 ceph-mon[31838]: pgmap v509: 129 pgs: 129 active+clean; 376 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 217 op/s 2024-03-19T03:36:51.300 INFO:tasks.workunit.client.0.smithi117.stdout: 83 440661 294 292210 234 258340 1034 2024-03-19T03:36:53.795 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:53 smithi117 ceph-mon[31838]: pgmap v510: 129 pgs: 129 active+clean; 392 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.9 MiB/s wr, 314 op/s 2024-03-19T03:36:53.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:53 smithi178 ceph-mon[67578]: pgmap v510: 129 pgs: 129 active+clean; 392 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.9 MiB/s wr, 314 op/s 2024-03-19T03:36:53.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:53 smithi119 ceph-mon[66861]: pgmap v510: 129 pgs: 129 active+clean; 392 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.9 MiB/s wr, 314 op/s 2024-03-19T03:36:55.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:55 smithi117 ceph-mon[31838]: pgmap v511: 129 pgs: 129 active+clean; 412 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 398 op/s 2024-03-19T03:36:55.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:36:55.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:36:55.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:36:55.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:55 smithi178 ceph-mon[67578]: pgmap v511: 129 pgs: 129 active+clean; 412 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 398 op/s 2024-03-19T03:36:55.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:36:55.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:36:55.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:36:55.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:55 smithi119 ceph-mon[66861]: pgmap v511: 129 pgs: 129 active+clean; 412 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 398 op/s 2024-03-19T03:36:55.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:36:55.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:36:55.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:36:56.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:56 smithi178 ceph-mon[67578]: pgmap v512: 129 pgs: 129 active+clean; 426 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.4 MiB/s wr, 481 op/s 2024-03-19T03:36:56.846 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:56 smithi117 ceph-mon[31838]: pgmap v512: 129 pgs: 129 active+clean; 426 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.4 MiB/s wr, 481 op/s 2024-03-19T03:36:56.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:56 smithi119 ceph-mon[66861]: pgmap v512: 129 pgs: 129 active+clean; 426 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.4 MiB/s wr, 481 op/s 2024-03-19T03:36:59.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:36:59 smithi117 ceph-mon[31838]: pgmap v513: 129 pgs: 129 active+clean; 431 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.9 MiB/s wr, 524 op/s 2024-03-19T03:36:59.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:36:59 smithi178 ceph-mon[67578]: pgmap v513: 129 pgs: 129 active+clean; 431 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.9 MiB/s wr, 524 op/s 2024-03-19T03:36:59.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:36:59 smithi119 ceph-mon[66861]: pgmap v513: 129 pgs: 129 active+clean; 431 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.9 MiB/s wr, 524 op/s 2024-03-19T03:37:01.309 INFO:tasks.workunit.client.0.smithi117.stdout: 89 405627 255 273850 322 237887 670 2024-03-19T03:37:01.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:01 smithi117 ceph-mon[31838]: pgmap v514: 129 pgs: 129 active+clean; 436 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.8 MiB/s wr, 539 op/s 2024-03-19T03:37:01.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:01 smithi178 ceph-mon[67578]: pgmap v514: 129 pgs: 129 active+clean; 436 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.8 MiB/s wr, 539 op/s 2024-03-19T03:37:01.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:01 smithi119 ceph-mon[66861]: pgmap v514: 129 pgs: 129 active+clean; 436 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.8 MiB/s wr, 539 op/s 2024-03-19T03:37:03.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:03 smithi117 ceph-mon[31838]: pgmap v515: 129 pgs: 129 active+clean; 436 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.0 MiB/s wr, 474 op/s 2024-03-19T03:37:03.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:03 smithi178 ceph-mon[67578]: pgmap v515: 129 pgs: 129 active+clean; 436 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.0 MiB/s wr, 474 op/s 2024-03-19T03:37:03.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:03 smithi119 ceph-mon[66861]: pgmap v515: 129 pgs: 129 active+clean; 436 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.0 MiB/s wr, 474 op/s 2024-03-19T03:37:04.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:04 smithi178 ceph-mon[67578]: pgmap v516: 129 pgs: 129 active+clean; 437 MiB data, 914 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 382 op/s 2024-03-19T03:37:04.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:04 smithi119 ceph-mon[66861]: pgmap v516: 129 pgs: 129 active+clean; 437 MiB data, 914 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 382 op/s 2024-03-19T03:37:05.059 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:04 smithi117 ceph-mon[31838]: pgmap v516: 129 pgs: 129 active+clean; 437 MiB data, 914 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 382 op/s 2024-03-19T03:37:07.793 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:07 smithi117 ceph-mon[31838]: pgmap v517: 129 pgs: 129 active+clean; 440 MiB data, 931 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 287 op/s 2024-03-19T03:37:07.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:07 smithi178 ceph-mon[67578]: pgmap v517: 129 pgs: 129 active+clean; 440 MiB data, 931 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 287 op/s 2024-03-19T03:37:07.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:07 smithi119 ceph-mon[66861]: pgmap v517: 129 pgs: 129 active+clean; 440 MiB data, 931 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 287 op/s 2024-03-19T03:37:09.800 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:09 smithi117 ceph-mon[31838]: pgmap v518: 129 pgs: 129 active+clean; 439 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 202 op/s 2024-03-19T03:37:09.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:09 smithi178 ceph-mon[67578]: pgmap v518: 129 pgs: 129 active+clean; 439 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 202 op/s 2024-03-19T03:37:09.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:09 smithi119 ceph-mon[66861]: pgmap v518: 129 pgs: 129 active+clean; 439 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.8 MiB/s wr, 202 op/s 2024-03-19T03:37:11.309 INFO:tasks.workunit.client.0.smithi117.stdout: 95 494129 338 337857 375 291405 1060 2024-03-19T03:37:11.804 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:11 smithi117 ceph-mon[31838]: pgmap v519: 129 pgs: 129 active+clean; 443 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-03-19T03:37:11.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:11 smithi178 ceph-mon[67578]: pgmap v519: 129 pgs: 129 active+clean; 443 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-03-19T03:37:11.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:11 smithi119 ceph-mon[66861]: pgmap v519: 129 pgs: 129 active+clean; 443 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-03-19T03:37:12.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:12 smithi178 ceph-mon[67578]: pgmap v520: 129 pgs: 129 active+clean; 441 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 136 op/s 2024-03-19T03:37:12.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:12 smithi119 ceph-mon[66861]: pgmap v520: 129 pgs: 129 active+clean; 441 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 136 op/s 2024-03-19T03:37:13.045 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:12 smithi117 ceph-mon[31838]: pgmap v520: 129 pgs: 129 active+clean; 441 MiB data, 910 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 136 op/s 2024-03-19T03:37:15.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:15 smithi117 ceph-mon[31838]: pgmap v521: 129 pgs: 129 active+clean; 441 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 128 op/s 2024-03-19T03:37:15.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:15 smithi178 ceph-mon[67578]: pgmap v521: 129 pgs: 129 active+clean; 441 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 128 op/s 2024-03-19T03:37:15.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:15 smithi119 ceph-mon[66861]: pgmap v521: 129 pgs: 129 active+clean; 441 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 128 op/s 2024-03-19T03:37:17.799 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:17 smithi117 ceph-mon[31838]: pgmap v522: 129 pgs: 129 active+clean; 443 MiB data, 949 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-03-19T03:37:17.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:17 smithi178 ceph-mon[67578]: pgmap v522: 129 pgs: 129 active+clean; 443 MiB data, 949 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-03-19T03:37:17.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:17 smithi119 ceph-mon[66861]: pgmap v522: 129 pgs: 129 active+clean; 443 MiB data, 949 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 120 op/s 2024-03-19T03:37:19.799 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:19 smithi117 ceph-mon[31838]: pgmap v523: 129 pgs: 129 active+clean; 446 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 110 op/s 2024-03-19T03:37:19.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:19 smithi178 ceph-mon[67578]: pgmap v523: 129 pgs: 129 active+clean; 446 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 110 op/s 2024-03-19T03:37:19.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:19 smithi119 ceph-mon[66861]: pgmap v523: 129 pgs: 129 active+clean; 446 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 110 op/s 2024-03-19T03:37:20.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:20 smithi178 ceph-mon[67578]: pgmap v524: 129 pgs: 129 active+clean; 453 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 154 op/s 2024-03-19T03:37:20.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:20 smithi119 ceph-mon[66861]: pgmap v524: 129 pgs: 129 active+clean; 453 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 154 op/s 2024-03-19T03:37:21.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:20 smithi117 ceph-mon[31838]: pgmap v524: 129 pgs: 129 active+clean; 453 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 154 op/s 2024-03-19T03:37:21.311 INFO:tasks.workunit.client.0.smithi117.stdout: 99 470551 385 324658 169 264564 775 2024-03-19T03:37:23.804 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:23 smithi117 ceph-mon[31838]: pgmap v525: 129 pgs: 129 active+clean; 475 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 3.3 MiB/s wr, 257 op/s 2024-03-19T03:37:23.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:23 smithi178 ceph-mon[67578]: pgmap v525: 129 pgs: 129 active+clean; 475 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 3.3 MiB/s wr, 257 op/s 2024-03-19T03:37:23.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:23 smithi119 ceph-mon[66861]: pgmap v525: 129 pgs: 129 active+clean; 475 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 3.3 MiB/s wr, 257 op/s 2024-03-19T03:37:25.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:25 smithi178 ceph-mon[67578]: pgmap v526: 129 pgs: 129 active+clean; 485 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.2 MiB/s wr, 302 op/s 2024-03-19T03:37:25.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:25 smithi119 ceph-mon[66861]: pgmap v526: 129 pgs: 129 active+clean; 485 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.2 MiB/s wr, 302 op/s 2024-03-19T03:37:26.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:25 smithi117 ceph-mon[31838]: pgmap v526: 129 pgs: 129 active+clean; 485 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.2 MiB/s wr, 302 op/s 2024-03-19T03:37:26.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:26 smithi178 ceph-mon[67578]: pgmap v527: 129 pgs: 129 active+clean; 507 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.2 MiB/s wr, 421 op/s 2024-03-19T03:37:26.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:26 smithi119 ceph-mon[66861]: pgmap v527: 129 pgs: 129 active+clean; 507 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.2 MiB/s wr, 421 op/s 2024-03-19T03:37:27.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:26 smithi117 ceph-mon[31838]: pgmap v527: 129 pgs: 129 active+clean; 507 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.2 MiB/s wr, 421 op/s 2024-03-19T03:37:29.802 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:29 smithi117 ceph-mon[31838]: pgmap v528: 129 pgs: 129 active+clean; 513 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.5 MiB/s wr, 461 op/s 2024-03-19T03:37:29.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:29 smithi178 ceph-mon[67578]: pgmap v528: 129 pgs: 129 active+clean; 513 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.5 MiB/s wr, 461 op/s 2024-03-19T03:37:29.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:29 smithi119 ceph-mon[66861]: pgmap v528: 129 pgs: 129 active+clean; 513 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.5 MiB/s wr, 461 op/s 2024-03-19T03:37:31.315 INFO:tasks.workunit.client.0.smithi117.stdout: 103 392014 248 263404 215 216349 657 2024-03-19T03:37:31.798 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:31 smithi117 ceph-mon[31838]: pgmap v529: 129 pgs: 129 active+clean; 519 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 6.9 MiB/s wr, 529 op/s 2024-03-19T03:37:31.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:31 smithi178 ceph-mon[67578]: pgmap v529: 129 pgs: 129 active+clean; 519 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 6.9 MiB/s wr, 529 op/s 2024-03-19T03:37:31.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:31 smithi119 ceph-mon[66861]: pgmap v529: 129 pgs: 129 active+clean; 519 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 6.9 MiB/s wr, 529 op/s 2024-03-19T03:37:33.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:33 smithi117 ceph-mon[31838]: pgmap v530: 129 pgs: 129 active+clean; 523 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.8 MiB/s wr, 513 op/s 2024-03-19T03:37:33.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:33 smithi178 ceph-mon[67578]: pgmap v530: 129 pgs: 129 active+clean; 523 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.8 MiB/s wr, 513 op/s 2024-03-19T03:37:33.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:33 smithi119 ceph-mon[66861]: pgmap v530: 129 pgs: 129 active+clean; 523 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.8 MiB/s wr, 513 op/s 2024-03-19T03:37:35.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:35 smithi119 ceph-mon[66861]: pgmap v531: 129 pgs: 129 active+clean; 523 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.9 MiB/s wr, 388 op/s 2024-03-19T03:37:35.541 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:35 smithi117 ceph-mon[31838]: pgmap v531: 129 pgs: 129 active+clean; 523 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.9 MiB/s wr, 388 op/s 2024-03-19T03:37:35.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:35 smithi178 ceph-mon[67578]: pgmap v531: 129 pgs: 129 active+clean; 523 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.9 MiB/s wr, 388 op/s 2024-03-19T03:37:37.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:37 smithi117 ceph-mon[31838]: pgmap v532: 129 pgs: 129 active+clean; 526 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 MiB/s wr, 354 op/s 2024-03-19T03:37:37.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:37 smithi178 ceph-mon[67578]: pgmap v532: 129 pgs: 129 active+clean; 526 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 MiB/s wr, 354 op/s 2024-03-19T03:37:37.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:37 smithi119 ceph-mon[66861]: pgmap v532: 129 pgs: 129 active+clean; 526 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 MiB/s wr, 354 op/s 2024-03-19T03:37:39.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:39 smithi117 ceph-mon[31838]: pgmap v533: 129 pgs: 129 active+clean; 525 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 224 op/s 2024-03-19T03:37:39.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:39 smithi178 ceph-mon[67578]: pgmap v533: 129 pgs: 129 active+clean; 525 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 224 op/s 2024-03-19T03:37:39.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:39 smithi119 ceph-mon[66861]: pgmap v533: 129 pgs: 129 active+clean; 525 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 224 op/s 2024-03-19T03:37:41.319 INFO:tasks.workunit.client.0.smithi117.stdout: 109 154446 335 104091 290 84344 1021 2024-03-19T03:37:41.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:41 smithi117 ceph-mon[31838]: pgmap v534: 129 pgs: 129 active+clean; 526 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 187 op/s 2024-03-19T03:37:41.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:41 smithi178 ceph-mon[67578]: pgmap v534: 129 pgs: 129 active+clean; 526 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 187 op/s 2024-03-19T03:37:41.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:41 smithi119 ceph-mon[66861]: pgmap v534: 129 pgs: 129 active+clean; 526 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 187 op/s 2024-03-19T03:37:43.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:43 smithi117 ceph-mon[31838]: pgmap v535: 129 pgs: 129 active+clean; 528 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 124 op/s 2024-03-19T03:37:43.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:43 smithi178 ceph-mon[67578]: pgmap v535: 129 pgs: 129 active+clean; 528 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 124 op/s 2024-03-19T03:37:43.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:43 smithi119 ceph-mon[66861]: pgmap v535: 129 pgs: 129 active+clean; 528 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 124 op/s 2024-03-19T03:37:44.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:44 smithi178 ceph-mon[67578]: pgmap v536: 129 pgs: 129 active+clean; 530 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 93 op/s 2024-03-19T03:37:44.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:44 smithi119 ceph-mon[66861]: pgmap v536: 129 pgs: 129 active+clean; 530 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 93 op/s 2024-03-19T03:37:45.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:44 smithi117 ceph-mon[31838]: pgmap v536: 129 pgs: 129 active+clean; 530 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 93 op/s 2024-03-19T03:37:47.828 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:47 smithi117 ceph-mon[31838]: pgmap v537: 129 pgs: 129 active+clean; 534 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 110 op/s 2024-03-19T03:37:47.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:47 smithi178 ceph-mon[67578]: pgmap v537: 129 pgs: 129 active+clean; 534 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 110 op/s 2024-03-19T03:37:47.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:47 smithi119 ceph-mon[66861]: pgmap v537: 129 pgs: 129 active+clean; 534 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 110 op/s 2024-03-19T03:37:49.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:49 smithi117 ceph-mon[31838]: pgmap v538: 129 pgs: 129 active+clean; 536 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 1.4 MiB/s wr, 87 op/s 2024-03-19T03:37:49.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:49 smithi178 ceph-mon[67578]: pgmap v538: 129 pgs: 129 active+clean; 536 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 1.4 MiB/s wr, 87 op/s 2024-03-19T03:37:49.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:49 smithi119 ceph-mon[66861]: pgmap v538: 129 pgs: 129 active+clean; 536 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 1.4 MiB/s wr, 87 op/s 2024-03-19T03:37:50.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:50 smithi119 ceph-mon[66861]: pgmap v539: 129 pgs: 129 active+clean; 536 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 1.4 MiB/s wr, 86 op/s 2024-03-19T03:37:51.049 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:50 smithi117 ceph-mon[31838]: pgmap v539: 129 pgs: 129 active+clean; 536 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 1.4 MiB/s wr, 86 op/s 2024-03-19T03:37:51.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:50 smithi178 ceph-mon[67578]: pgmap v539: 129 pgs: 129 active+clean; 536 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 1.4 MiB/s wr, 86 op/s 2024-03-19T03:37:51.324 INFO:tasks.workunit.client.0.smithi117.stdout: 114 294604 421 197615 325 163166 1151 2024-03-19T03:37:53.794 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:53 smithi117 ceph-mon[31838]: pgmap v540: 129 pgs: 129 active+clean; 548 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 178 op/s 2024-03-19T03:37:53.798 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:53 smithi178 ceph-mon[67578]: pgmap v540: 129 pgs: 129 active+clean; 548 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 178 op/s 2024-03-19T03:37:53.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:53 smithi119 ceph-mon[66861]: pgmap v540: 129 pgs: 129 active+clean; 548 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 178 op/s 2024-03-19T03:37:55.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:55 smithi178 ceph-mon[67578]: pgmap v541: 129 pgs: 129 active+clean; 564 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 254 op/s 2024-03-19T03:37:55.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:55 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:37:55.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:55 smithi119 ceph-mon[66861]: pgmap v541: 129 pgs: 129 active+clean; 564 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 254 op/s 2024-03-19T03:37:55.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:55 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:37:56.046 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:55 smithi117 ceph-mon[31838]: pgmap v541: 129 pgs: 129 active+clean; 564 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 254 op/s 2024-03-19T03:37:56.046 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:55 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:37:57.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:57 smithi178 ceph-mon[67578]: pgmap v542: 129 pgs: 129 active+clean; 570 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 306 op/s 2024-03-19T03:37:57.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:57 smithi119 ceph-mon[66861]: pgmap v542: 129 pgs: 129 active+clean; 570 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 306 op/s 2024-03-19T03:37:58.055 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:57 smithi117 ceph-mon[31838]: pgmap v542: 129 pgs: 129 active+clean; 570 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 306 op/s 2024-03-19T03:37:58.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:37:58 smithi119 ceph-mon[66861]: pgmap v543: 129 pgs: 129 active+clean; 593 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.6 MiB/s wr, 433 op/s 2024-03-19T03:37:59.060 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:37:58 smithi117 ceph-mon[31838]: pgmap v543: 129 pgs: 129 active+clean; 593 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.6 MiB/s wr, 433 op/s 2024-03-19T03:37:59.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:37:58 smithi178 ceph-mon[67578]: pgmap v543: 129 pgs: 129 active+clean; 593 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.6 MiB/s wr, 433 op/s 2024-03-19T03:38:01.397 INFO:tasks.workunit.client.0.smithi117.stdout: 118 413586 177 278223 144 239910 611 2024-03-19T03:38:01.794 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:01 smithi117 ceph-mon[31838]: pgmap v544: 129 pgs: 129 active+clean; 597 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.8 MiB/s wr, 519 op/s 2024-03-19T03:38:01.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:01 smithi178 ceph-mon[67578]: pgmap v544: 129 pgs: 129 active+clean; 597 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.8 MiB/s wr, 519 op/s 2024-03-19T03:38:01.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:01 smithi119 ceph-mon[66861]: pgmap v544: 129 pgs: 129 active+clean; 597 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.8 MiB/s wr, 519 op/s 2024-03-19T03:38:03.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:03 smithi178 ceph-mon[67578]: pgmap v545: 129 pgs: 129 active+clean; 608 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.9 MiB/s wr, 719 op/s 2024-03-19T03:38:03.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:03 smithi119 ceph-mon[66861]: pgmap v545: 129 pgs: 129 active+clean; 608 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.9 MiB/s wr, 719 op/s 2024-03-19T03:38:04.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:03 smithi117 ceph-mon[31838]: pgmap v545: 129 pgs: 129 active+clean; 608 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.9 MiB/s wr, 719 op/s 2024-03-19T03:38:05.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:05 smithi178 ceph-mon[67578]: pgmap v546: 129 pgs: 129 active+clean; 607 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.6 MiB/s wr, 648 op/s 2024-03-19T03:38:05.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:05 smithi119 ceph-mon[66861]: pgmap v546: 129 pgs: 129 active+clean; 607 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.6 MiB/s wr, 648 op/s 2024-03-19T03:38:06.050 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:05 smithi117 ceph-mon[31838]: pgmap v546: 129 pgs: 129 active+clean; 607 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.6 MiB/s wr, 648 op/s 2024-03-19T03:38:07.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:07 smithi178 ceph-mon[67578]: pgmap v547: 129 pgs: 129 active+clean; 611 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.7 MiB/s wr, 616 op/s 2024-03-19T03:38:07.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:07 smithi119 ceph-mon[66861]: pgmap v547: 129 pgs: 129 active+clean; 611 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.7 MiB/s wr, 616 op/s 2024-03-19T03:38:08.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:07 smithi117 ceph-mon[31838]: pgmap v547: 129 pgs: 129 active+clean; 611 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.7 MiB/s wr, 616 op/s 2024-03-19T03:38:08.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:08 smithi119 ceph-mon[66861]: pgmap v548: 129 pgs: 129 active+clean; 612 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 620 op/s 2024-03-19T03:38:09.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:08 smithi117 ceph-mon[31838]: pgmap v548: 129 pgs: 129 active+clean; 612 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 620 op/s 2024-03-19T03:38:09.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:08 smithi178 ceph-mon[67578]: pgmap v548: 129 pgs: 129 active+clean; 612 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 620 op/s 2024-03-19T03:38:11.398 INFO:tasks.workunit.client.0.smithi117.stdout: 125 473027 407 319533 279 263912 992 2024-03-19T03:38:11.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:11 smithi178 ceph-mon[67578]: pgmap v549: 129 pgs: 129 active+clean; 611 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 497 op/s 2024-03-19T03:38:11.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:11 smithi119 ceph-mon[66861]: pgmap v549: 129 pgs: 129 active+clean; 611 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 497 op/s 2024-03-19T03:38:12.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:11 smithi117 ceph-mon[31838]: pgmap v549: 129 pgs: 129 active+clean; 611 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 497 op/s 2024-03-19T03:38:13.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:13 smithi178 ceph-mon[67578]: pgmap v550: 129 pgs: 129 active+clean; 614 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 497 op/s 2024-03-19T03:38:13.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:13 smithi119 ceph-mon[66861]: pgmap v550: 129 pgs: 129 active+clean; 614 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 497 op/s 2024-03-19T03:38:14.044 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:13 smithi117 ceph-mon[31838]: pgmap v550: 129 pgs: 129 active+clean; 614 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 497 op/s 2024-03-19T03:38:14.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:14 smithi119 ceph-mon[66861]: pgmap v551: 129 pgs: 129 active+clean; 614 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 1.0 MiB/s wr, 333 op/s 2024-03-19T03:38:15.047 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:14 smithi117 ceph-mon[31838]: pgmap v551: 129 pgs: 129 active+clean; 614 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 1.0 MiB/s wr, 333 op/s 2024-03-19T03:38:15.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:14 smithi178 ceph-mon[67578]: pgmap v551: 129 pgs: 129 active+clean; 614 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 1.0 MiB/s wr, 333 op/s 2024-03-19T03:38:17.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:17 smithi178 ceph-mon[67578]: pgmap v552: 129 pgs: 129 active+clean; 616 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 343 op/s 2024-03-19T03:38:17.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:17 smithi119 ceph-mon[66861]: pgmap v552: 129 pgs: 129 active+clean; 616 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 343 op/s 2024-03-19T03:38:18.044 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:17 smithi117 ceph-mon[31838]: pgmap v552: 129 pgs: 129 active+clean; 616 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 343 op/s 2024-03-19T03:38:18.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:18 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:38:18.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:18 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:38:18.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:18 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:38:18.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:18 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:38:18.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:18 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:38:18.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:18 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:38:19.044 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:18 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:38:19.044 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:18 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:38:19.044 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:18 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:38:19.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:19 smithi178 ceph-mon[67578]: pgmap v553: 129 pgs: 129 active+clean; 608 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 1.0 MiB/s wr, 354 op/s 2024-03-19T03:38:19.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:19 smithi119 ceph-mon[66861]: pgmap v553: 129 pgs: 129 active+clean; 608 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 1.0 MiB/s wr, 354 op/s 2024-03-19T03:38:20.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:19 smithi117 ceph-mon[31838]: pgmap v553: 129 pgs: 129 active+clean; 608 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 1.0 MiB/s wr, 354 op/s 2024-03-19T03:38:21.408 INFO:tasks.workunit.client.0.smithi117.stdout: 128 456835 313 300958 275 249058 943 2024-03-19T03:38:21.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:21 smithi178 ceph-mon[67578]: pgmap v554: 129 pgs: 129 active+clean; 606 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.5 MiB/s wr, 359 op/s 2024-03-19T03:38:21.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:21 smithi119 ceph-mon[66861]: pgmap v554: 129 pgs: 129 active+clean; 606 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.5 MiB/s wr, 359 op/s 2024-03-19T03:38:22.050 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:21 smithi117 ceph-mon[31838]: pgmap v554: 129 pgs: 129 active+clean; 606 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.5 MiB/s wr, 359 op/s 2024-03-19T03:38:22.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:22 smithi119 ceph-mon[66861]: pgmap v555: 129 pgs: 129 active+clean; 604 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 465 op/s 2024-03-19T03:38:23.050 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:22 smithi117 ceph-mon[31838]: pgmap v555: 129 pgs: 129 active+clean; 604 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 465 op/s 2024-03-19T03:38:23.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:22 smithi178 ceph-mon[67578]: pgmap v555: 129 pgs: 129 active+clean; 604 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 465 op/s 2024-03-19T03:38:25.798 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:25 smithi178 ceph-mon[67578]: pgmap v556: 129 pgs: 129 active+clean; 613 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 479 op/s 2024-03-19T03:38:25.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:25 smithi119 ceph-mon[66861]: pgmap v556: 129 pgs: 129 active+clean; 613 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 479 op/s 2024-03-19T03:38:26.049 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:25 smithi117 ceph-mon[31838]: pgmap v556: 129 pgs: 129 active+clean; 613 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 479 op/s 2024-03-19T03:38:27.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:27 smithi178 ceph-mon[67578]: pgmap v557: 129 pgs: 129 active+clean; 629 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 538 op/s 2024-03-19T03:38:27.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:27 smithi119 ceph-mon[66861]: pgmap v557: 129 pgs: 129 active+clean; 629 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 538 op/s 2024-03-19T03:38:28.057 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:27 smithi117 ceph-mon[31838]: pgmap v557: 129 pgs: 129 active+clean; 629 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 538 op/s 2024-03-19T03:38:29.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:29 smithi178 ceph-mon[67578]: pgmap v558: 129 pgs: 129 active+clean; 641 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.2 MiB/s wr, 593 op/s 2024-03-19T03:38:29.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:29 smithi178 ceph-mon[67578]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:38:29.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:29 smithi119 ceph-mon[66861]: pgmap v558: 129 pgs: 129 active+clean; 641 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.2 MiB/s wr, 593 op/s 2024-03-19T03:38:29.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:29 smithi119 ceph-mon[66861]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:38:30.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:29 smithi117 ceph-mon[31838]: pgmap v558: 129 pgs: 129 active+clean; 641 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.2 MiB/s wr, 593 op/s 2024-03-19T03:38:30.044 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:29 smithi117 ceph-mon[31838]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:38:31.410 INFO:tasks.workunit.client.0.smithi117.stdout: 132 436737 266 295089 202 250850 721 2024-03-19T03:38:31.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:31 smithi178 ceph-mon[67578]: pgmap v559: 129 pgs: 129 active+clean; 645 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.5 MiB/s wr, 598 op/s 2024-03-19T03:38:31.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:31 smithi119 ceph-mon[66861]: pgmap v559: 129 pgs: 129 active+clean; 645 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.5 MiB/s wr, 598 op/s 2024-03-19T03:38:32.063 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:31 smithi117 ceph-mon[31838]: pgmap v559: 129 pgs: 129 active+clean; 645 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.5 MiB/s wr, 598 op/s 2024-03-19T03:38:33.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:33 smithi178 ceph-mon[67578]: pgmap v560: 129 pgs: 129 active+clean; 645 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.1 MiB/s wr, 602 op/s 2024-03-19T03:38:33.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:33 smithi119 ceph-mon[66861]: pgmap v560: 129 pgs: 129 active+clean; 645 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.1 MiB/s wr, 602 op/s 2024-03-19T03:38:34.050 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:33 smithi117 ceph-mon[31838]: pgmap v560: 129 pgs: 129 active+clean; 645 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.1 MiB/s wr, 602 op/s 2024-03-19T03:38:34.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:34 smithi119 ceph-mon[66861]: pgmap v561: 129 pgs: 129 active+clean; 648 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.3 MiB/s wr, 484 op/s 2024-03-19T03:38:35.061 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:34 smithi117 ceph-mon[31838]: pgmap v561: 129 pgs: 129 active+clean; 648 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.3 MiB/s wr, 484 op/s 2024-03-19T03:38:35.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:34 smithi178 ceph-mon[67578]: pgmap v561: 129 pgs: 129 active+clean; 648 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.3 MiB/s wr, 484 op/s 2024-03-19T03:38:37.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:37 smithi178 ceph-mon[67578]: pgmap v562: 129 pgs: 129 active+clean; 647 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.5 MiB/s wr, 395 op/s 2024-03-19T03:38:37.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:37 smithi119 ceph-mon[66861]: pgmap v562: 129 pgs: 129 active+clean; 647 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.5 MiB/s wr, 395 op/s 2024-03-19T03:38:38.049 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:37 smithi117 ceph-mon[31838]: pgmap v562: 129 pgs: 129 active+clean; 647 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.5 MiB/s wr, 395 op/s 2024-03-19T03:38:39.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:39 smithi178 ceph-mon[67578]: pgmap v563: 129 pgs: 129 active+clean; 652 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 309 op/s 2024-03-19T03:38:39.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:39 smithi119 ceph-mon[66861]: pgmap v563: 129 pgs: 129 active+clean; 652 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 309 op/s 2024-03-19T03:38:40.055 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:39 smithi117 ceph-mon[31838]: pgmap v563: 129 pgs: 129 active+clean; 652 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 309 op/s 2024-03-19T03:38:41.419 INFO:tasks.workunit.client.0.smithi117.stdout: 140 492600 555 334586 326 292907 1140 2024-03-19T03:38:41.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:41 smithi178 ceph-mon[67578]: pgmap v564: 129 pgs: 129 active+clean; 651 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 242 op/s 2024-03-19T03:38:41.846 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:41 smithi117 ceph-mon[31838]: pgmap v564: 129 pgs: 129 active+clean; 651 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 242 op/s 2024-03-19T03:38:41.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:41 smithi119 ceph-mon[66861]: pgmap v564: 129 pgs: 129 active+clean; 651 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 242 op/s 2024-03-19T03:38:43.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:43 smithi178 ceph-mon[67578]: pgmap v565: 129 pgs: 129 active+clean; 652 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 209 op/s 2024-03-19T03:38:43.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:43 smithi119 ceph-mon[66861]: pgmap v565: 129 pgs: 129 active+clean; 652 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 209 op/s 2024-03-19T03:38:44.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:43 smithi117 ceph-mon[31838]: pgmap v565: 129 pgs: 129 active+clean; 652 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 209 op/s 2024-03-19T03:38:44.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:44 smithi119 ceph-mon[66861]: pgmap v566: 129 pgs: 129 active+clean; 647 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 177 op/s 2024-03-19T03:38:45.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:44 smithi117 ceph-mon[31838]: pgmap v566: 129 pgs: 129 active+clean; 647 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 177 op/s 2024-03-19T03:38:45.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:44 smithi178 ceph-mon[67578]: pgmap v566: 129 pgs: 129 active+clean; 647 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 177 op/s 2024-03-19T03:38:47.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:47 smithi178 ceph-mon[67578]: pgmap v567: 129 pgs: 129 active+clean; 635 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 212 op/s 2024-03-19T03:38:47.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:47 smithi119 ceph-mon[66861]: pgmap v567: 129 pgs: 129 active+clean; 635 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 212 op/s 2024-03-19T03:38:48.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:47 smithi117 ceph-mon[31838]: pgmap v567: 129 pgs: 129 active+clean; 635 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 212 op/s 2024-03-19T03:38:49.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:49 smithi178 ceph-mon[67578]: pgmap v568: 129 pgs: 129 active+clean; 634 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 246 op/s 2024-03-19T03:38:49.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:49 smithi119 ceph-mon[66861]: pgmap v568: 129 pgs: 129 active+clean; 634 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 246 op/s 2024-03-19T03:38:50.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:49 smithi117 ceph-mon[31838]: pgmap v568: 129 pgs: 129 active+clean; 634 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 246 op/s 2024-03-19T03:38:51.414 INFO:tasks.workunit.client.0.smithi117.stdout: 144 438227 252 298029 244 253427 730 2024-03-19T03:38:51.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:51 smithi178 ceph-mon[67578]: pgmap v569: 129 pgs: 129 active+clean; 635 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 291 op/s 2024-03-19T03:38:51.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:51 smithi119 ceph-mon[66861]: pgmap v569: 129 pgs: 129 active+clean; 635 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 291 op/s 2024-03-19T03:38:52.055 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:51 smithi117 ceph-mon[31838]: pgmap v569: 129 pgs: 129 active+clean; 635 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 291 op/s 2024-03-19T03:38:53.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:53 smithi119 ceph-mon[66861]: pgmap v570: 129 pgs: 129 active+clean; 659 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 448 op/s 2024-03-19T03:38:54.050 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:53 smithi117 ceph-mon[31838]: pgmap v570: 129 pgs: 129 active+clean; 659 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 448 op/s 2024-03-19T03:38:54.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:53 smithi178 ceph-mon[67578]: pgmap v570: 129 pgs: 129 active+clean; 659 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.7 MiB/s wr, 448 op/s 2024-03-19T03:38:54.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:54 smithi119 ceph-mon[66861]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:38:54.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:54 smithi119 ceph-mon[66861]: pgmap v571: 129 pgs: 129 active+clean; 663 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 473 op/s 2024-03-19T03:38:55.045 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:54 smithi117 ceph-mon[31838]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:38:55.045 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:54 smithi117 ceph-mon[31838]: pgmap v571: 129 pgs: 129 active+clean; 663 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 473 op/s 2024-03-19T03:38:55.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:54 smithi178 ceph-mon[67578]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:38:55.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:54 smithi178 ceph-mon[67578]: pgmap v571: 129 pgs: 129 active+clean; 663 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 473 op/s 2024-03-19T03:38:57.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:57 smithi178 ceph-mon[67578]: pgmap v572: 129 pgs: 129 active+clean; 679 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.0 MiB/s wr, 568 op/s 2024-03-19T03:38:57.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:57 smithi119 ceph-mon[66861]: pgmap v572: 129 pgs: 129 active+clean; 679 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.0 MiB/s wr, 568 op/s 2024-03-19T03:38:58.083 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:57 smithi117 ceph-mon[31838]: pgmap v572: 129 pgs: 129 active+clean; 679 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.0 MiB/s wr, 568 op/s 2024-03-19T03:38:59.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:38:59 smithi178 ceph-mon[67578]: pgmap v573: 129 pgs: 129 active+clean; 697 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 6.2 MiB/s wr, 667 op/s 2024-03-19T03:38:59.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:38:59 smithi119 ceph-mon[66861]: pgmap v573: 129 pgs: 129 active+clean; 697 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 6.2 MiB/s wr, 667 op/s 2024-03-19T03:39:00.070 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:38:59 smithi117 ceph-mon[31838]: pgmap v573: 129 pgs: 129 active+clean; 697 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 6.2 MiB/s wr, 667 op/s 2024-03-19T03:39:01.441 INFO:tasks.workunit.client.0.smithi117.stdout: 146 400548 229 269119 186 234434 695 2024-03-19T03:39:01.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:01 smithi178 ceph-mon[67578]: pgmap v574: 129 pgs: 129 active+clean; 710 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.3 MiB/s wr, 700 op/s 2024-03-19T03:39:01.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:01 smithi119 ceph-mon[66861]: pgmap v574: 129 pgs: 129 active+clean; 710 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.3 MiB/s wr, 700 op/s 2024-03-19T03:39:02.048 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:01 smithi117 ceph-mon[31838]: pgmap v574: 129 pgs: 129 active+clean; 710 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.3 MiB/s wr, 700 op/s 2024-03-19T03:39:03.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:03 smithi178 ceph-mon[67578]: pgmap v575: 129 pgs: 129 active+clean; 708 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.0 MiB/s wr, 740 op/s 2024-03-19T03:39:03.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:03 smithi119 ceph-mon[66861]: pgmap v575: 129 pgs: 129 active+clean; 708 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.0 MiB/s wr, 740 op/s 2024-03-19T03:39:04.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:03 smithi117 ceph-mon[31838]: pgmap v575: 129 pgs: 129 active+clean; 708 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.0 MiB/s wr, 740 op/s 2024-03-19T03:39:04.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:04 smithi119 ceph-mon[66861]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:39:04.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:04 smithi119 ceph-mon[66861]: pgmap v576: 129 pgs: 129 active+clean; 707 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.8 MiB/s wr, 583 op/s 2024-03-19T03:39:05.048 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:04 smithi117 ceph-mon[31838]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:39:05.049 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:04 smithi117 ceph-mon[31838]: pgmap v576: 129 pgs: 129 active+clean; 707 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.8 MiB/s wr, 583 op/s 2024-03-19T03:39:05.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:04 smithi178 ceph-mon[67578]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:39:05.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:04 smithi178 ceph-mon[67578]: pgmap v576: 129 pgs: 129 active+clean; 707 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.8 MiB/s wr, 583 op/s 2024-03-19T03:39:07.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:07 smithi178 ceph-mon[67578]: pgmap v577: 129 pgs: 129 active+clean; 708 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 578 op/s 2024-03-19T03:39:07.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:07 smithi119 ceph-mon[66861]: pgmap v577: 129 pgs: 129 active+clean; 708 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 578 op/s 2024-03-19T03:39:08.051 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:07 smithi117 ceph-mon[31838]: pgmap v577: 129 pgs: 129 active+clean; 708 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.6 MiB/s wr, 578 op/s 2024-03-19T03:39:09.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:09 smithi178 ceph-mon[67578]: pgmap v578: 129 pgs: 129 active+clean; 711 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 522 op/s 2024-03-19T03:39:09.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:09 smithi119 ceph-mon[66861]: pgmap v578: 129 pgs: 129 active+clean; 711 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 522 op/s 2024-03-19T03:39:10.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:09 smithi117 ceph-mon[31838]: pgmap v578: 129 pgs: 129 active+clean; 711 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 522 op/s 2024-03-19T03:39:11.443 INFO:tasks.workunit.client.0.smithi117.stdout: 150 460892 349 314961 304 278115 1083 2024-03-19T03:39:11.846 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:11 smithi178 ceph-mon[67578]: pgmap v579: 129 pgs: 129 active+clean; 706 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 415 op/s 2024-03-19T03:39:11.855 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:11 smithi117 ceph-mon[31838]: pgmap v579: 129 pgs: 129 active+clean; 706 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 415 op/s 2024-03-19T03:39:11.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:11 smithi119 ceph-mon[66861]: pgmap v579: 129 pgs: 129 active+clean; 706 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 415 op/s 2024-03-19T03:39:13.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:13 smithi178 ceph-mon[67578]: pgmap v580: 129 pgs: 129 active+clean; 688 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 429 op/s 2024-03-19T03:39:13.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:13 smithi119 ceph-mon[66861]: pgmap v580: 129 pgs: 129 active+clean; 688 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 429 op/s 2024-03-19T03:39:14.061 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:13 smithi117 ceph-mon[31838]: pgmap v580: 129 pgs: 129 active+clean; 688 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 429 op/s 2024-03-19T03:39:14.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:14 smithi119 ceph-mon[66861]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:39:14.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:14 smithi119 ceph-mon[66861]: pgmap v581: 129 pgs: 129 active+clean; 686 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 357 op/s 2024-03-19T03:39:15.044 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:14 smithi117 ceph-mon[31838]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:39:15.044 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:14 smithi117 ceph-mon[31838]: pgmap v581: 129 pgs: 129 active+clean; 686 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 357 op/s 2024-03-19T03:39:15.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:14 smithi178 ceph-mon[67578]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:39:15.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:14 smithi178 ceph-mon[67578]: pgmap v581: 129 pgs: 129 active+clean; 686 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 357 op/s 2024-03-19T03:39:17.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:17 smithi178 ceph-mon[67578]: pgmap v582: 129 pgs: 129 active+clean; 688 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 378 op/s 2024-03-19T03:39:17.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:17 smithi119 ceph-mon[66861]: pgmap v582: 129 pgs: 129 active+clean; 688 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 378 op/s 2024-03-19T03:39:18.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:17 smithi117 ceph-mon[31838]: pgmap v582: 129 pgs: 129 active+clean; 688 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 378 op/s 2024-03-19T03:39:19.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:19 smithi178 ceph-mon[67578]: pgmap v583: 129 pgs: 129 active+clean; 688 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 365 op/s 2024-03-19T03:39:19.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:19 smithi119 ceph-mon[66861]: pgmap v583: 129 pgs: 129 active+clean; 688 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 365 op/s 2024-03-19T03:39:20.044 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:19 smithi117 ceph-mon[31838]: pgmap v583: 129 pgs: 129 active+clean; 688 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 365 op/s 2024-03-19T03:39:20.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:20 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:39:21.044 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:20 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:39:21.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:20 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:39:21.468 INFO:tasks.workunit.client.0.smithi117.stdout: 156 453645 329 305347 314 267695 829 2024-03-19T03:39:21.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:21 smithi119 ceph-mon[66861]: pgmap v584: 129 pgs: 129 active+clean; 687 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 1.1 MiB/s wr, 301 op/s 2024-03-19T03:39:21.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:21 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:39:21.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:21 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:39:21.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:21 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:39:21.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:21 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:39:21.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:21 smithi119 ceph-mon[66861]: Adjusting osd_memory_target on smithi119 to 174.1M 2024-03-19T03:39:21.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:21 smithi119 ceph-mon[66861]: Unable to set osd_memory_target on smithi119 to 182619136: error parsing value: Value '182619136' is below minimum 939524096 2024-03-19T03:39:22.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:21 smithi117 ceph-mon[31838]: pgmap v584: 129 pgs: 129 active+clean; 687 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 1.1 MiB/s wr, 301 op/s 2024-03-19T03:39:22.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:21 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:39:22.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:21 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:39:22.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:21 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:39:22.044 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:21 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:39:22.044 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:21 smithi117 ceph-mon[31838]: Adjusting osd_memory_target on smithi119 to 174.1M 2024-03-19T03:39:22.044 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:21 smithi117 ceph-mon[31838]: Unable to set osd_memory_target on smithi119 to 182619136: error parsing value: Value '182619136' is below minimum 939524096 2024-03-19T03:39:22.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:21 smithi178 ceph-mon[67578]: pgmap v584: 129 pgs: 129 active+clean; 687 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 1.1 MiB/s wr, 301 op/s 2024-03-19T03:39:22.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:21 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:39:22.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:21 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:39:22.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:21 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:39:22.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:21 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:39:22.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:21 smithi178 ceph-mon[67578]: Adjusting osd_memory_target on smithi119 to 174.1M 2024-03-19T03:39:22.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:21 smithi178 ceph-mon[67578]: Unable to set osd_memory_target on smithi119 to 182619136: error parsing value: Value '182619136' is below minimum 939524096 2024-03-19T03:39:22.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:22 smithi119 ceph-mon[66861]: pgmap v585: 129 pgs: 129 active+clean; 707 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 412 op/s 2024-03-19T03:39:23.045 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:22 smithi117 ceph-mon[31838]: pgmap v585: 129 pgs: 129 active+clean; 707 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 412 op/s 2024-03-19T03:39:23.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:22 smithi178 ceph-mon[67578]: pgmap v585: 129 pgs: 129 active+clean; 707 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 412 op/s 2024-03-19T03:39:25.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:25 smithi178 ceph-mon[67578]: pgmap v586: 129 pgs: 129 active+clean; 715 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 3.3 MiB/s wr, 394 op/s 2024-03-19T03:39:25.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:25 smithi119 ceph-mon[66861]: pgmap v586: 129 pgs: 129 active+clean; 715 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 3.3 MiB/s wr, 394 op/s 2024-03-19T03:39:26.052 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:25 smithi117 ceph-mon[31838]: pgmap v586: 129 pgs: 129 active+clean; 715 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 3.3 MiB/s wr, 394 op/s 2024-03-19T03:39:26.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:26 smithi119 ceph-mon[66861]: pgmap v587: 129 pgs: 129 active+clean; 719 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 937 KiB/s rd, 3.5 MiB/s wr, 425 op/s 2024-03-19T03:39:27.046 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:26 smithi117 ceph-mon[31838]: pgmap v587: 129 pgs: 129 active+clean; 719 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 937 KiB/s rd, 3.5 MiB/s wr, 425 op/s 2024-03-19T03:39:27.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:26 smithi178 ceph-mon[67578]: pgmap v587: 129 pgs: 129 active+clean; 719 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 937 KiB/s rd, 3.5 MiB/s wr, 425 op/s 2024-03-19T03:39:29.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:29 smithi119 ceph-mon[66861]: pgmap v588: 129 pgs: 129 active+clean; 733 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.4 MiB/s wr, 517 op/s 2024-03-19T03:39:30.044 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:29 smithi117 ceph-mon[31838]: pgmap v588: 129 pgs: 129 active+clean; 733 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.4 MiB/s wr, 517 op/s 2024-03-19T03:39:30.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:29 smithi178 ceph-mon[67578]: pgmap v588: 129 pgs: 129 active+clean; 733 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.4 MiB/s wr, 517 op/s 2024-03-19T03:39:30.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:30 smithi119 ceph-mon[66861]: pgmap v589: 129 pgs: 129 active+clean; 743 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.2 MiB/s wr, 546 op/s 2024-03-19T03:39:31.045 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:30 smithi117 ceph-mon[31838]: pgmap v589: 129 pgs: 129 active+clean; 743 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.2 MiB/s wr, 546 op/s 2024-03-19T03:39:31.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:30 smithi178 ceph-mon[67578]: pgmap v589: 129 pgs: 129 active+clean; 743 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.2 MiB/s wr, 546 op/s 2024-03-19T03:39:31.609 INFO:tasks.workunit.client.0.smithi117.stdout: 157 404308 156 273267 162 233587 534 2024-03-19T03:39:33.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:33 smithi119 ceph-mon[66861]: pgmap v590: 129 pgs: 129 active+clean; 752 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 6.0 MiB/s wr, 671 op/s 2024-03-19T03:39:34.060 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:33 smithi117 ceph-mon[31838]: pgmap v590: 129 pgs: 129 active+clean; 752 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 6.0 MiB/s wr, 671 op/s 2024-03-19T03:39:34.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:33 smithi178 ceph-mon[67578]: pgmap v590: 129 pgs: 129 active+clean; 752 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 6.0 MiB/s wr, 671 op/s 2024-03-19T03:39:35.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:35 smithi119 ceph-mon[66861]: pgmap v591: 129 pgs: 129 active+clean; 758 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.8 MiB/s wr, 617 op/s 2024-03-19T03:39:36.061 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:35 smithi117 ceph-mon[31838]: pgmap v591: 129 pgs: 129 active+clean; 758 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.8 MiB/s wr, 617 op/s 2024-03-19T03:39:36.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:35 smithi178 ceph-mon[67578]: pgmap v591: 129 pgs: 129 active+clean; 758 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.8 MiB/s wr, 617 op/s 2024-03-19T03:39:36.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:36 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:39:36.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:36 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:39:36.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:36 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:39:36.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:36 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:39:36.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:36 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:39:36.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:36 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:39:36.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:36 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:39:37.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:36 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:39:37.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:36 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:39:37.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:36 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:39:37.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:36 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:39:37.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:36 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:39:37.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:36 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:39:37.044 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:36 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:39:37.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:36 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:39:37.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:36 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:39:37.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:36 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:39:37.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:36 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:39:37.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:36 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:39:37.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:36 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:39:37.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:36 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:39:37.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:37 smithi119 ceph-mon[66861]: Adjusting osd_memory_target on smithi117 to 176.7M 2024-03-19T03:39:37.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:37 smithi119 ceph-mon[66861]: Unable to set osd_memory_target on smithi117 to 185386700: error parsing value: Value '185386700' is below minimum 939524096 2024-03-19T03:39:37.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:37 smithi119 ceph-mon[66861]: pgmap v592: 129 pgs: 129 active+clean; 749 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 571 op/s 2024-03-19T03:39:38.051 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:37 smithi117 ceph-mon[31838]: Adjusting osd_memory_target on smithi117 to 176.7M 2024-03-19T03:39:38.051 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:37 smithi117 ceph-mon[31838]: Unable to set osd_memory_target on smithi117 to 185386700: error parsing value: Value '185386700' is below minimum 939524096 2024-03-19T03:39:38.051 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:37 smithi117 ceph-mon[31838]: pgmap v592: 129 pgs: 129 active+clean; 749 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 571 op/s 2024-03-19T03:39:38.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:37 smithi178 ceph-mon[67578]: Adjusting osd_memory_target on smithi117 to 176.7M 2024-03-19T03:39:38.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:37 smithi178 ceph-mon[67578]: Unable to set osd_memory_target on smithi117 to 185386700: error parsing value: Value '185386700' is below minimum 939524096 2024-03-19T03:39:38.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:37 smithi178 ceph-mon[67578]: pgmap v592: 129 pgs: 129 active+clean; 749 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 571 op/s 2024-03-19T03:39:38.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:38 smithi119 ceph-mon[66861]: pgmap v593: 129 pgs: 129 active+clean; 736 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 593 op/s 2024-03-19T03:39:39.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:38 smithi117 ceph-mon[31838]: pgmap v593: 129 pgs: 129 active+clean; 736 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 593 op/s 2024-03-19T03:39:39.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:38 smithi178 ceph-mon[67578]: pgmap v593: 129 pgs: 129 active+clean; 736 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.0 MiB/s wr, 593 op/s 2024-03-19T03:39:41.509 INFO:tasks.workunit.client.0.smithi117.stdout: 161 458207 360 313715 301 263584 1015 2024-03-19T03:39:41.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:41 smithi119 ceph-mon[66861]: pgmap v594: 129 pgs: 129 active+clean; 735 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 476 op/s 2024-03-19T03:39:42.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:41 smithi117 ceph-mon[31838]: pgmap v594: 129 pgs: 129 active+clean; 735 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 476 op/s 2024-03-19T03:39:42.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:41 smithi178 ceph-mon[67578]: pgmap v594: 129 pgs: 129 active+clean; 735 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 476 op/s 2024-03-19T03:39:42.536 INFO:tasks.workunit.client.0.smithi117.stdout: 2024-03-19T03:39:42.536 INFO:tasks.workunit.client.0.smithi117.stdout:Final score for writes: 161 2024-03-19T03:39:42.536 INFO:tasks.workunit.client.0.smithi117.stdout:Final score for reads : 261227 2024-03-19T03:39:42.536 INFO:tasks.workunit.client.0.smithi117.stdout: 2024-03-19T03:39:42.538 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-19T03:39:42.538 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-19T03:39:43.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:43 smithi119 ceph-mon[66861]: pgmap v595: 129 pgs: 129 active+clean; 733 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1016 KiB/s rd, 2.2 MiB/s wr, 470 op/s 2024-03-19T03:39:44.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:43 smithi117 ceph-mon[31838]: pgmap v595: 129 pgs: 129 active+clean; 733 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1016 KiB/s rd, 2.2 MiB/s wr, 470 op/s 2024-03-19T03:39:44.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:43 smithi178 ceph-mon[67578]: pgmap v595: 129 pgs: 129 active+clean; 733 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1016 KiB/s rd, 2.2 MiB/s wr, 470 op/s 2024-03-19T03:39:45.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:45 smithi119 ceph-mon[66861]: pgmap v596: 129 pgs: 129 active+clean; 736 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 387 op/s 2024-03-19T03:39:46.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:45 smithi117 ceph-mon[31838]: pgmap v596: 129 pgs: 129 active+clean; 736 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 387 op/s 2024-03-19T03:39:46.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:45 smithi178 ceph-mon[67578]: pgmap v596: 129 pgs: 129 active+clean; 736 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 387 op/s 2024-03-19T03:39:46.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:46 smithi119 ceph-mon[66861]: pgmap v597: 129 pgs: 129 active+clean; 736 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 322 op/s 2024-03-19T03:39:47.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:46 smithi117 ceph-mon[31838]: pgmap v597: 129 pgs: 129 active+clean; 736 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 322 op/s 2024-03-19T03:39:47.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:46 smithi178 ceph-mon[67578]: pgmap v597: 129 pgs: 129 active+clean; 736 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 322 op/s 2024-03-19T03:39:49.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:49 smithi119 ceph-mon[66861]: pgmap v598: 129 pgs: 129 active+clean; 735 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 1.6 MiB/s wr, 401 op/s 2024-03-19T03:39:50.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:49 smithi117 ceph-mon[31838]: pgmap v598: 129 pgs: 129 active+clean; 735 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 1.6 MiB/s wr, 401 op/s 2024-03-19T03:39:50.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:49 smithi178 ceph-mon[67578]: pgmap v598: 129 pgs: 129 active+clean; 735 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 1.6 MiB/s wr, 401 op/s 2024-03-19T03:39:50.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:50 smithi119 ceph-mon[66861]: pgmap v599: 129 pgs: 129 active+clean; 735 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 1.7 MiB/s wr, 339 op/s 2024-03-19T03:39:51.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:50 smithi117 ceph-mon[31838]: pgmap v599: 129 pgs: 129 active+clean; 735 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 1.7 MiB/s wr, 339 op/s 2024-03-19T03:39:51.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:50 smithi178 ceph-mon[67578]: pgmap v599: 129 pgs: 129 active+clean; 735 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 1.7 MiB/s wr, 339 op/s 2024-03-19T03:39:52.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:52 smithi119 ceph-mon[66861]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:39:52.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:52 smithi119 ceph-mon[66861]: pgmap v600: 129 pgs: 129 active+clean; 743 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 431 op/s 2024-03-19T03:39:53.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:52 smithi117 ceph-mon[31838]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:39:53.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:52 smithi117 ceph-mon[31838]: pgmap v600: 129 pgs: 129 active+clean; 743 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 431 op/s 2024-03-19T03:39:53.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:52 smithi178 ceph-mon[67578]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:39:53.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:52 smithi178 ceph-mon[67578]: pgmap v600: 129 pgs: 129 active+clean; 743 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 431 op/s 2024-03-19T03:39:55.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:55 smithi119 ceph-mon[66861]: pgmap v601: 129 pgs: 129 active+clean; 752 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 491 op/s 2024-03-19T03:39:56.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:55 smithi117 ceph-mon[31838]: pgmap v601: 129 pgs: 129 active+clean; 752 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 491 op/s 2024-03-19T03:39:56.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:55 smithi178 ceph-mon[67578]: pgmap v601: 129 pgs: 129 active+clean; 752 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 491 op/s 2024-03-19T03:39:56.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:56 smithi119 ceph-mon[66861]: pgmap v602: 129 pgs: 129 active+clean; 755 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 4.1 MiB/s wr, 511 op/s 2024-03-19T03:39:57.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:56 smithi117 ceph-mon[31838]: pgmap v602: 129 pgs: 129 active+clean; 755 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 4.1 MiB/s wr, 511 op/s 2024-03-19T03:39:57.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:56 smithi178 ceph-mon[67578]: pgmap v602: 129 pgs: 129 active+clean; 755 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 4.1 MiB/s wr, 511 op/s 2024-03-19T03:39:59.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:39:59 smithi119 ceph-mon[66861]: pgmap v603: 129 pgs: 129 active+clean; 757 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.6 MiB/s wr, 614 op/s 2024-03-19T03:40:00.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:39:59 smithi117 ceph-mon[31838]: pgmap v603: 129 pgs: 129 active+clean; 757 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.6 MiB/s wr, 614 op/s 2024-03-19T03:40:00.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:39:59 smithi178 ceph-mon[67578]: pgmap v603: 129 pgs: 129 active+clean; 757 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.6 MiB/s wr, 614 op/s 2024-03-19T03:40:00.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:00 smithi119 ceph-mon[66861]: overall HEALTH_OK 2024-03-19T03:40:01.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:00 smithi117 ceph-mon[31838]: overall HEALTH_OK 2024-03-19T03:40:01.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:00 smithi178 ceph-mon[67578]: overall HEALTH_OK 2024-03-19T03:40:01.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:01 smithi119 ceph-mon[66861]: pgmap v604: 129 pgs: 129 active+clean; 756 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 519 op/s 2024-03-19T03:40:02.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:01 smithi117 ceph-mon[31838]: pgmap v604: 129 pgs: 129 active+clean; 756 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 519 op/s 2024-03-19T03:40:02.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:01 smithi178 ceph-mon[67578]: pgmap v604: 129 pgs: 129 active+clean; 756 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 519 op/s 2024-03-19T03:40:02.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:02 smithi119 ceph-mon[66861]: pgmap v605: 129 pgs: 129 active+clean; 733 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 577 op/s 2024-03-19T03:40:03.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:02 smithi117 ceph-mon[31838]: pgmap v605: 129 pgs: 129 active+clean; 733 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 577 op/s 2024-03-19T03:40:03.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:02 smithi178 ceph-mon[67578]: pgmap v605: 129 pgs: 129 active+clean; 733 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 577 op/s 2024-03-19T03:40:05.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:05 smithi119 ceph-mon[66861]: pgmap v606: 129 pgs: 129 active+clean; 713 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 529 op/s 2024-03-19T03:40:06.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:05 smithi117 ceph-mon[31838]: pgmap v606: 129 pgs: 129 active+clean; 713 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 529 op/s 2024-03-19T03:40:06.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:05 smithi178 ceph-mon[67578]: pgmap v606: 129 pgs: 129 active+clean; 713 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 529 op/s 2024-03-19T03:40:06.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:06 smithi119 ceph-mon[66861]: pgmap v607: 129 pgs: 129 active+clean; 708 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 1.6 MiB/s wr, 455 op/s 2024-03-19T03:40:07.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:06 smithi117 ceph-mon[31838]: pgmap v607: 129 pgs: 129 active+clean; 708 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 1.6 MiB/s wr, 455 op/s 2024-03-19T03:40:07.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:06 smithi178 ceph-mon[67578]: pgmap v607: 129 pgs: 129 active+clean; 708 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 1.6 MiB/s wr, 455 op/s 2024-03-19T03:40:09.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:09 smithi119 ceph-mon[66861]: pgmap v608: 129 pgs: 129 active+clean; 700 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 494 op/s 2024-03-19T03:40:10.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:09 smithi117 ceph-mon[31838]: pgmap v608: 129 pgs: 129 active+clean; 700 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 494 op/s 2024-03-19T03:40:10.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:09 smithi178 ceph-mon[67578]: pgmap v608: 129 pgs: 129 active+clean; 700 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 494 op/s 2024-03-19T03:40:10.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:10 smithi119 ceph-mon[66861]: pgmap v609: 129 pgs: 129 active+clean; 698 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 631 KiB/s wr, 367 op/s 2024-03-19T03:40:11.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:10 smithi117 ceph-mon[31838]: pgmap v609: 129 pgs: 129 active+clean; 698 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 631 KiB/s wr, 367 op/s 2024-03-19T03:40:11.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:10 smithi178 ceph-mon[67578]: pgmap v609: 129 pgs: 129 active+clean; 698 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 631 KiB/s wr, 367 op/s 2024-03-19T03:40:11.943 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:11 smithi117 ceph-mon[31838]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:40:11.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:11 smithi119 ceph-mon[66861]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:40:12.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:11 smithi178 ceph-mon[67578]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:40:12.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:12 smithi119 ceph-mon[66861]: pgmap v610: 129 pgs: 129 active+clean; 687 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 724 KiB/s wr, 462 op/s 2024-03-19T03:40:13.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:12 smithi117 ceph-mon[31838]: pgmap v610: 129 pgs: 129 active+clean; 687 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 724 KiB/s wr, 462 op/s 2024-03-19T03:40:13.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:12 smithi178 ceph-mon[67578]: pgmap v610: 129 pgs: 129 active+clean; 687 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 724 KiB/s wr, 462 op/s 2024-03-19T03:40:15.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:15 smithi119 ceph-mon[66861]: pgmap v611: 129 pgs: 129 active+clean; 684 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 471 op/s 2024-03-19T03:40:16.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:15 smithi117 ceph-mon[31838]: pgmap v611: 129 pgs: 129 active+clean; 684 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 471 op/s 2024-03-19T03:40:16.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:15 smithi178 ceph-mon[67578]: pgmap v611: 129 pgs: 129 active+clean; 684 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 471 op/s 2024-03-19T03:40:16.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:16 smithi119 ceph-mon[66861]: pgmap v612: 129 pgs: 129 active+clean; 680 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 969 KiB/s wr, 438 op/s 2024-03-19T03:40:17.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:16 smithi117 ceph-mon[31838]: pgmap v612: 129 pgs: 129 active+clean; 680 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 969 KiB/s wr, 438 op/s 2024-03-19T03:40:17.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:16 smithi178 ceph-mon[67578]: pgmap v612: 129 pgs: 129 active+clean; 680 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 969 KiB/s wr, 438 op/s 2024-03-19T03:40:19.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:19 smithi119 ceph-mon[66861]: pgmap v613: 129 pgs: 129 active+clean; 669 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 1.0 MiB/s wr, 472 op/s 2024-03-19T03:40:20.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:19 smithi117 ceph-mon[31838]: pgmap v613: 129 pgs: 129 active+clean; 669 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 1.0 MiB/s wr, 472 op/s 2024-03-19T03:40:20.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:19 smithi178 ceph-mon[67578]: pgmap v613: 129 pgs: 129 active+clean; 669 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 1.0 MiB/s wr, 472 op/s 2024-03-19T03:40:20.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:20 smithi119 ceph-mon[66861]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:40:20.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:20 smithi119 ceph-mon[66861]: pgmap v614: 129 pgs: 129 active+clean; 670 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 1.0 MiB/s wr, 369 op/s 2024-03-19T03:40:21.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:20 smithi117 ceph-mon[31838]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:40:21.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:20 smithi117 ceph-mon[31838]: pgmap v614: 129 pgs: 129 active+clean; 670 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 1.0 MiB/s wr, 369 op/s 2024-03-19T03:40:21.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:20 smithi178 ceph-mon[67578]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:40:21.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:20 smithi178 ceph-mon[67578]: pgmap v614: 129 pgs: 129 active+clean; 670 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 1.0 MiB/s wr, 369 op/s 2024-03-19T03:40:22.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:22 smithi119 ceph-mon[66861]: pgmap v615: 129 pgs: 129 active+clean; 663 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 438 op/s 2024-03-19T03:40:23.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:22 smithi117 ceph-mon[31838]: pgmap v615: 129 pgs: 129 active+clean; 663 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 438 op/s 2024-03-19T03:40:23.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:22 smithi178 ceph-mon[67578]: pgmap v615: 129 pgs: 129 active+clean; 663 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 438 op/s 2024-03-19T03:40:24.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:24 smithi119 ceph-mon[66861]: pgmap v616: 129 pgs: 129 active+clean; 656 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 398 op/s 2024-03-19T03:40:25.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:24 smithi117 ceph-mon[31838]: pgmap v616: 129 pgs: 129 active+clean; 656 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 398 op/s 2024-03-19T03:40:25.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:24 smithi178 ceph-mon[67578]: pgmap v616: 129 pgs: 129 active+clean; 656 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 398 op/s 2024-03-19T03:40:26.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:26 smithi119 ceph-mon[66861]: pgmap v617: 129 pgs: 129 active+clean; 656 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 365 op/s 2024-03-19T03:40:27.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:26 smithi117 ceph-mon[31838]: pgmap v617: 129 pgs: 129 active+clean; 656 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 365 op/s 2024-03-19T03:40:27.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:26 smithi178 ceph-mon[67578]: pgmap v617: 129 pgs: 129 active+clean; 656 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 365 op/s 2024-03-19T03:40:28.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:28 smithi119 ceph-mon[66861]: pgmap v618: 129 pgs: 129 active+clean; 644 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 445 op/s 2024-03-19T03:40:29.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:28 smithi117 ceph-mon[31838]: pgmap v618: 129 pgs: 129 active+clean; 644 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 445 op/s 2024-03-19T03:40:29.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:28 smithi178 ceph-mon[67578]: pgmap v618: 129 pgs: 129 active+clean; 644 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 445 op/s 2024-03-19T03:40:30.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:30 smithi119 ceph-mon[66861]: pgmap v619: 129 pgs: 129 active+clean; 642 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 965 KiB/s wr, 383 op/s 2024-03-19T03:40:31.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:30 smithi117 ceph-mon[31838]: pgmap v619: 129 pgs: 129 active+clean; 642 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 965 KiB/s wr, 383 op/s 2024-03-19T03:40:31.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:30 smithi178 ceph-mon[67578]: pgmap v619: 129 pgs: 129 active+clean; 642 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 965 KiB/s wr, 383 op/s 2024-03-19T03:40:32.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:32 smithi119 ceph-mon[66861]: pgmap v620: 129 pgs: 129 active+clean; 617 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 948 KiB/s wr, 501 op/s 2024-03-19T03:40:33.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:32 smithi117 ceph-mon[31838]: pgmap v620: 129 pgs: 129 active+clean; 617 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 948 KiB/s wr, 501 op/s 2024-03-19T03:40:33.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:32 smithi178 ceph-mon[67578]: pgmap v620: 129 pgs: 129 active+clean; 617 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 948 KiB/s wr, 501 op/s 2024-03-19T03:40:34.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:34 smithi119 ceph-mon[66861]: pgmap v621: 129 pgs: 129 active+clean; 600 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1022 KiB/s wr, 519 op/s 2024-03-19T03:40:35.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:34 smithi117 ceph-mon[31838]: pgmap v621: 129 pgs: 129 active+clean; 600 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1022 KiB/s wr, 519 op/s 2024-03-19T03:40:35.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:34 smithi178 ceph-mon[67578]: pgmap v621: 129 pgs: 129 active+clean; 600 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1022 KiB/s wr, 519 op/s 2024-03-19T03:40:36.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:36 smithi119 ceph-mon[66861]: pgmap v622: 129 pgs: 129 active+clean; 596 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 954 KiB/s wr, 505 op/s 2024-03-19T03:40:37.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:36 smithi117 ceph-mon[31838]: pgmap v622: 129 pgs: 129 active+clean; 596 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 954 KiB/s wr, 505 op/s 2024-03-19T03:40:37.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:36 smithi178 ceph-mon[67578]: pgmap v622: 129 pgs: 129 active+clean; 596 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 954 KiB/s wr, 505 op/s 2024-03-19T03:40:38.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:37 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:40:38.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:37 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:40:38.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:37 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:40:39.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:38 smithi119 ceph-mon[66861]: pgmap v623: 129 pgs: 129 active+clean; 591 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 861 KiB/s wr, 549 op/s 2024-03-19T03:40:39.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:38 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:40:39.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:38 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:40:39.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:38 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:40:39.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:38 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:40:39.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:38 smithi119 ceph-mon[66861]: Adjusting osd_memory_target on smithi178 to 174.1M 2024-03-19T03:40:39.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:38 smithi119 ceph-mon[66861]: Unable to set osd_memory_target on smithi178 to 182601932: error parsing value: Value '182601932' is below minimum 939524096 2024-03-19T03:40:39.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:38 smithi117 ceph-mon[31838]: pgmap v623: 129 pgs: 129 active+clean; 591 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 861 KiB/s wr, 549 op/s 2024-03-19T03:40:39.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:38 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:40:39.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:38 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:40:39.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:38 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:40:39.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:38 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:40:39.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:38 smithi117 ceph-mon[31838]: Adjusting osd_memory_target on smithi178 to 174.1M 2024-03-19T03:40:39.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:38 smithi117 ceph-mon[31838]: Unable to set osd_memory_target on smithi178 to 182601932: error parsing value: Value '182601932' is below minimum 939524096 2024-03-19T03:40:39.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:38 smithi178 ceph-mon[67578]: pgmap v623: 129 pgs: 129 active+clean; 591 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 861 KiB/s wr, 549 op/s 2024-03-19T03:40:39.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:38 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:40:39.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:38 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:40:39.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:38 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:40:39.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:38 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T03:40:39.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:38 smithi178 ceph-mon[67578]: Adjusting osd_memory_target on smithi178 to 174.1M 2024-03-19T03:40:39.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:38 smithi178 ceph-mon[67578]: Unable to set osd_memory_target on smithi178 to 182601932: error parsing value: Value '182601932' is below minimum 939524096 2024-03-19T03:40:40.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:40:40.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:40:40.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:40:40.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:40:40.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:40:40.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:40:40.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:40:40.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:40:40.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:40:41.291 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:40 smithi117 ceph-mon[31838]: pgmap v624: 129 pgs: 129 active+clean; 589 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 840 KiB/s wr, 441 op/s 2024-03-19T03:40:41.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:40 smithi178 ceph-mon[67578]: pgmap v624: 129 pgs: 129 active+clean; 589 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 840 KiB/s wr, 441 op/s 2024-03-19T03:40:41.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:40 smithi119 ceph-mon[66861]: pgmap v624: 129 pgs: 129 active+clean; 589 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 840 KiB/s wr, 441 op/s 2024-03-19T03:40:42.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:42 smithi119 ceph-mon[66861]: pgmap v625: 129 pgs: 129 active+clean; 582 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 1.1 MiB/s wr, 516 op/s 2024-03-19T03:40:43.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:42 smithi117 ceph-mon[31838]: pgmap v625: 129 pgs: 129 active+clean; 582 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 1.1 MiB/s wr, 516 op/s 2024-03-19T03:40:43.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:42 smithi178 ceph-mon[67578]: pgmap v625: 129 pgs: 129 active+clean; 582 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 1.1 MiB/s wr, 516 op/s 2024-03-19T03:40:44.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:44 smithi119 ceph-mon[66861]: pgmap v626: 129 pgs: 129 active+clean; 574 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 951 KiB/s wr, 461 op/s 2024-03-19T03:40:45.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:44 smithi117 ceph-mon[31838]: pgmap v626: 129 pgs: 129 active+clean; 574 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 951 KiB/s wr, 461 op/s 2024-03-19T03:40:45.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:44 smithi178 ceph-mon[67578]: pgmap v626: 129 pgs: 129 active+clean; 574 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 951 KiB/s wr, 461 op/s 2024-03-19T03:40:46.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:46 smithi119 ceph-mon[66861]: pgmap v627: 129 pgs: 129 active+clean; 571 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 830 KiB/s wr, 408 op/s 2024-03-19T03:40:47.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:46 smithi117 ceph-mon[31838]: pgmap v627: 129 pgs: 129 active+clean; 571 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 830 KiB/s wr, 408 op/s 2024-03-19T03:40:47.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:46 smithi178 ceph-mon[67578]: pgmap v627: 129 pgs: 129 active+clean; 571 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 830 KiB/s wr, 408 op/s 2024-03-19T03:40:48.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:48 smithi119 ceph-mon[66861]: pgmap v628: 129 pgs: 129 active+clean; 568 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 451 op/s 2024-03-19T03:40:49.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:48 smithi117 ceph-mon[31838]: pgmap v628: 129 pgs: 129 active+clean; 568 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 451 op/s 2024-03-19T03:40:49.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:48 smithi178 ceph-mon[67578]: pgmap v628: 129 pgs: 129 active+clean; 568 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 451 op/s 2024-03-19T03:40:50.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:50 smithi119 ceph-mon[66861]: pgmap v629: 129 pgs: 129 active+clean; 567 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 906 KiB/s rd, 1.2 MiB/s wr, 358 op/s 2024-03-19T03:40:51.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:50 smithi117 ceph-mon[31838]: pgmap v629: 129 pgs: 129 active+clean; 567 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 906 KiB/s rd, 1.2 MiB/s wr, 358 op/s 2024-03-19T03:40:51.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:50 smithi178 ceph-mon[67578]: pgmap v629: 129 pgs: 129 active+clean; 567 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 906 KiB/s rd, 1.2 MiB/s wr, 358 op/s 2024-03-19T03:40:52.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:52 smithi119 ceph-mon[66861]: pgmap v630: 129 pgs: 129 active+clean; 571 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 377 op/s 2024-03-19T03:40:53.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:52 smithi117 ceph-mon[31838]: pgmap v630: 129 pgs: 129 active+clean; 571 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 377 op/s 2024-03-19T03:40:53.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:52 smithi178 ceph-mon[67578]: pgmap v630: 129 pgs: 129 active+clean; 571 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 377 op/s 2024-03-19T03:40:54.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:54 smithi119 ceph-mon[66861]: pgmap v631: 129 pgs: 129 active+clean; 555 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 308 op/s 2024-03-19T03:40:55.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:54 smithi117 ceph-mon[31838]: pgmap v631: 129 pgs: 129 active+clean; 555 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 308 op/s 2024-03-19T03:40:55.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:54 smithi178 ceph-mon[67578]: pgmap v631: 129 pgs: 129 active+clean; 555 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 308 op/s 2024-03-19T03:40:56.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:56 smithi119 ceph-mon[66861]: pgmap v632: 129 pgs: 129 active+clean; 549 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 260 op/s 2024-03-19T03:40:57.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:56 smithi117 ceph-mon[31838]: pgmap v632: 129 pgs: 129 active+clean; 549 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 260 op/s 2024-03-19T03:40:57.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:56 smithi178 ceph-mon[67578]: pgmap v632: 129 pgs: 129 active+clean; 549 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 260 op/s 2024-03-19T03:40:59.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:59 smithi119 ceph-mon[66861]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:40:59.734 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:40:59 smithi119 ceph-mon[66861]: pgmap v633: 129 pgs: 129 active+clean; 544 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 322 op/s 2024-03-19T03:40:59.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:59 smithi117 ceph-mon[31838]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:40:59.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:40:59 smithi117 ceph-mon[31838]: pgmap v633: 129 pgs: 129 active+clean; 544 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 322 op/s 2024-03-19T03:40:59.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:59 smithi178 ceph-mon[67578]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:40:59.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:40:59 smithi178 ceph-mon[67578]: pgmap v633: 129 pgs: 129 active+clean; 544 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 322 op/s 2024-03-19T03:41:00.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:00 smithi119 ceph-mon[66861]: pgmap v634: 129 pgs: 129 active+clean; 542 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 243 op/s 2024-03-19T03:41:01.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:00 smithi117 ceph-mon[31838]: pgmap v634: 129 pgs: 129 active+clean; 542 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 243 op/s 2024-03-19T03:41:01.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:00 smithi178 ceph-mon[67578]: pgmap v634: 129 pgs: 129 active+clean; 542 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 243 op/s 2024-03-19T03:41:02.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:02 smithi119 ceph-mon[66861]: pgmap v635: 129 pgs: 129 active+clean; 535 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 385 op/s 2024-03-19T03:41:03.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:02 smithi117 ceph-mon[31838]: pgmap v635: 129 pgs: 129 active+clean; 535 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 385 op/s 2024-03-19T03:41:03.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:02 smithi178 ceph-mon[67578]: pgmap v635: 129 pgs: 129 active+clean; 535 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 385 op/s 2024-03-19T03:41:04.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:04 smithi119 ceph-mon[66861]: pgmap v636: 129 pgs: 129 active+clean; 528 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 483 op/s 2024-03-19T03:41:05.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:04 smithi117 ceph-mon[31838]: pgmap v636: 129 pgs: 129 active+clean; 528 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 483 op/s 2024-03-19T03:41:05.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:04 smithi178 ceph-mon[67578]: pgmap v636: 129 pgs: 129 active+clean; 528 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.4 MiB/s wr, 483 op/s 2024-03-19T03:41:06.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:06 smithi119 ceph-mon[66861]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:41:06.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:06 smithi119 ceph-mon[66861]: pgmap v637: 129 pgs: 129 active+clean; 525 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 516 op/s 2024-03-19T03:41:07.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:06 smithi117 ceph-mon[31838]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:41:07.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:06 smithi117 ceph-mon[31838]: pgmap v637: 129 pgs: 129 active+clean; 525 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 516 op/s 2024-03-19T03:41:07.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:06 smithi178 ceph-mon[67578]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:41:07.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:06 smithi178 ceph-mon[67578]: pgmap v637: 129 pgs: 129 active+clean; 525 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 516 op/s 2024-03-19T03:41:07.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:07 smithi119 ceph-mon[66861]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:41:08.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:07 smithi117 ceph-mon[31838]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:41:08.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:07 smithi178 ceph-mon[67578]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:41:08.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:08 smithi119 ceph-mon[66861]: pgmap v638: 129 pgs: 129 active+clean; 516 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 617 op/s 2024-03-19T03:41:09.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:08 smithi117 ceph-mon[31838]: pgmap v638: 129 pgs: 129 active+clean; 516 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 617 op/s 2024-03-19T03:41:09.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:08 smithi178 ceph-mon[67578]: pgmap v638: 129 pgs: 129 active+clean; 516 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 617 op/s 2024-03-19T03:41:10.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:10 smithi119 ceph-mon[66861]: pgmap v639: 129 pgs: 129 active+clean; 514 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 844 KiB/s wr, 528 op/s 2024-03-19T03:41:11.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:10 smithi117 ceph-mon[31838]: pgmap v639: 129 pgs: 129 active+clean; 514 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 844 KiB/s wr, 528 op/s 2024-03-19T03:41:11.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:10 smithi178 ceph-mon[67578]: pgmap v639: 129 pgs: 129 active+clean; 514 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 844 KiB/s wr, 528 op/s 2024-03-19T03:41:12.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:12 smithi178 ceph-mon[67578]: pgmap v640: 129 pgs: 129 active+clean; 493 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 592 op/s 2024-03-19T03:41:12.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:12 smithi119 ceph-mon[66861]: pgmap v640: 129 pgs: 129 active+clean; 493 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 592 op/s 2024-03-19T03:41:13.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:12 smithi117 ceph-mon[31838]: pgmap v640: 129 pgs: 129 active+clean; 493 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 592 op/s 2024-03-19T03:41:14.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:14 smithi119 ceph-mon[66861]: pgmap v641: 129 pgs: 129 active+clean; 475 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 1.1 MiB/s wr, 500 op/s 2024-03-19T03:41:15.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:14 smithi117 ceph-mon[31838]: pgmap v641: 129 pgs: 129 active+clean; 475 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 1.1 MiB/s wr, 500 op/s 2024-03-19T03:41:15.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:14 smithi178 ceph-mon[67578]: pgmap v641: 129 pgs: 129 active+clean; 475 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 1.1 MiB/s wr, 500 op/s 2024-03-19T03:41:16.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:16 smithi119 ceph-mon[66861]: pgmap v642: 129 pgs: 129 active+clean; 471 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 1021 KiB/s wr, 409 op/s 2024-03-19T03:41:17.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:16 smithi117 ceph-mon[31838]: pgmap v642: 129 pgs: 129 active+clean; 471 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 1021 KiB/s wr, 409 op/s 2024-03-19T03:41:17.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:16 smithi178 ceph-mon[67578]: pgmap v642: 129 pgs: 129 active+clean; 471 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 1021 KiB/s wr, 409 op/s 2024-03-19T03:41:18.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:18 smithi119 ceph-mon[66861]: pgmap v643: 129 pgs: 129 active+clean; 458 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 901 KiB/s wr, 464 op/s 2024-03-19T03:41:19.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:18 smithi117 ceph-mon[31838]: pgmap v643: 129 pgs: 129 active+clean; 458 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 901 KiB/s wr, 464 op/s 2024-03-19T03:41:19.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:18 smithi178 ceph-mon[67578]: pgmap v643: 129 pgs: 129 active+clean; 458 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 901 KiB/s wr, 464 op/s 2024-03-19T03:41:20.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:20 smithi119 ceph-mon[66861]: pgmap v644: 129 pgs: 129 active+clean; 457 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 837 KiB/s wr, 352 op/s 2024-03-19T03:41:21.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:20 smithi117 ceph-mon[31838]: pgmap v644: 129 pgs: 129 active+clean; 457 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 837 KiB/s wr, 352 op/s 2024-03-19T03:41:21.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:20 smithi178 ceph-mon[67578]: pgmap v644: 129 pgs: 129 active+clean; 457 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 837 KiB/s wr, 352 op/s 2024-03-19T03:41:22.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:22 smithi119 ceph-mon[66861]: pgmap v645: 129 pgs: 129 active+clean; 450 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 456 op/s 2024-03-19T03:41:23.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:22 smithi117 ceph-mon[31838]: pgmap v645: 129 pgs: 129 active+clean; 450 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 456 op/s 2024-03-19T03:41:23.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:22 smithi178 ceph-mon[67578]: pgmap v645: 129 pgs: 129 active+clean; 450 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 456 op/s 2024-03-19T03:41:24.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:24 smithi119 ceph-mon[66861]: pgmap v646: 129 pgs: 129 active+clean; 444 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 468 op/s 2024-03-19T03:41:25.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:24 smithi117 ceph-mon[31838]: pgmap v646: 129 pgs: 129 active+clean; 444 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 468 op/s 2024-03-19T03:41:25.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:24 smithi178 ceph-mon[67578]: pgmap v646: 129 pgs: 129 active+clean; 444 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 468 op/s 2024-03-19T03:41:26.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:26 smithi119 ceph-mon[66861]: pgmap v647: 129 pgs: 129 active+clean; 438 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 934 KiB/s wr, 483 op/s 2024-03-19T03:41:27.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:26 smithi117 ceph-mon[31838]: pgmap v647: 129 pgs: 129 active+clean; 438 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 934 KiB/s wr, 483 op/s 2024-03-19T03:41:27.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:26 smithi178 ceph-mon[67578]: pgmap v647: 129 pgs: 129 active+clean; 438 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 934 KiB/s wr, 483 op/s 2024-03-19T03:41:28.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:28 smithi119 ceph-mon[66861]: pgmap v648: 129 pgs: 129 active+clean; 431 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 585 op/s 2024-03-19T03:41:29.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:28 smithi117 ceph-mon[31838]: pgmap v648: 129 pgs: 129 active+clean; 431 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 585 op/s 2024-03-19T03:41:29.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:28 smithi178 ceph-mon[67578]: pgmap v648: 129 pgs: 129 active+clean; 431 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 585 op/s 2024-03-19T03:41:31.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:31 smithi117 ceph-mon[31838]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:41:31.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:31 smithi117 ceph-mon[31838]: pgmap v649: 129 pgs: 129 active+clean; 429 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 491 op/s 2024-03-19T03:41:31.798 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:31 smithi178 ceph-mon[67578]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:41:31.798 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:31 smithi178 ceph-mon[67578]: pgmap v649: 129 pgs: 129 active+clean; 429 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 491 op/s 2024-03-19T03:41:31.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:31 smithi119 ceph-mon[66861]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:41:31.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:31 smithi119 ceph-mon[66861]: pgmap v649: 129 pgs: 129 active+clean; 429 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 491 op/s 2024-03-19T03:41:32.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:32 smithi119 ceph-mon[66861]: pgmap v650: 129 pgs: 129 active+clean; 420 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 585 op/s 2024-03-19T03:41:33.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:32 smithi117 ceph-mon[31838]: pgmap v650: 129 pgs: 129 active+clean; 420 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 585 op/s 2024-03-19T03:41:33.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:32 smithi178 ceph-mon[67578]: pgmap v650: 129 pgs: 129 active+clean; 420 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 585 op/s 2024-03-19T03:41:35.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:35 smithi119 ceph-mon[66861]: pgmap v651: 129 pgs: 129 active+clean; 408 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 912 KiB/s wr, 557 op/s 2024-03-19T03:41:35.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:35 smithi117 ceph-mon[31838]: pgmap v651: 129 pgs: 129 active+clean; 408 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 912 KiB/s wr, 557 op/s 2024-03-19T03:41:35.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:35 smithi178 ceph-mon[67578]: pgmap v651: 129 pgs: 129 active+clean; 408 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 912 KiB/s wr, 557 op/s 2024-03-19T03:41:36.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:36 smithi119 ceph-mon[66861]: pgmap v652: 129 pgs: 129 active+clean; 394 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 505 op/s 2024-03-19T03:41:37.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:36 smithi117 ceph-mon[31838]: pgmap v652: 129 pgs: 129 active+clean; 394 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 505 op/s 2024-03-19T03:41:37.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:36 smithi178 ceph-mon[67578]: pgmap v652: 129 pgs: 129 active+clean; 394 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 505 op/s 2024-03-19T03:41:38.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:38 smithi119 ceph-mon[66861]: pgmap v653: 129 pgs: 129 active+clean; 375 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 538 op/s 2024-03-19T03:41:39.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:38 smithi117 ceph-mon[31838]: pgmap v653: 129 pgs: 129 active+clean; 375 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 538 op/s 2024-03-19T03:41:39.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:38 smithi178 ceph-mon[67578]: pgmap v653: 129 pgs: 129 active+clean; 375 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 538 op/s 2024-03-19T03:41:39.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:39 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:41:40.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:39 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:41:40.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:39 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:41:40.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:40 smithi119 ceph-mon[66861]: pgmap v654: 129 pgs: 129 active+clean; 373 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 761 KiB/s wr, 409 op/s 2024-03-19T03:41:41.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:40 smithi117 ceph-mon[31838]: pgmap v654: 129 pgs: 129 active+clean; 373 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 761 KiB/s wr, 409 op/s 2024-03-19T03:41:41.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:40 smithi178 ceph-mon[67578]: pgmap v654: 129 pgs: 129 active+clean; 373 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 761 KiB/s wr, 409 op/s 2024-03-19T03:41:42.016 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:41:42.016 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:41:42.016 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:41:42.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:41:42.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:41:42.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:41:42.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:41:42.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:41:42.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:41:43.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:42 smithi117 ceph-mon[31838]: pgmap v655: 129 pgs: 129 active+clean; 367 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 478 op/s 2024-03-19T03:41:43.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:42 smithi178 ceph-mon[67578]: pgmap v655: 129 pgs: 129 active+clean; 367 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 478 op/s 2024-03-19T03:41:43.137 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:42 smithi119 ceph-mon[66861]: pgmap v655: 129 pgs: 129 active+clean; 367 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 478 op/s 2024-03-19T03:41:44.689 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:44 smithi117 ceph-mon[31838]: pgmap v656: 129 pgs: 129 active+clean; 360 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 440 op/s 2024-03-19T03:41:45.022 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:44 smithi119 ceph-mon[66861]: pgmap v656: 129 pgs: 129 active+clean; 360 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 440 op/s 2024-03-19T03:41:45.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:44 smithi178 ceph-mon[67578]: pgmap v656: 129 pgs: 129 active+clean; 360 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 440 op/s 2024-03-19T03:41:46.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:46 smithi119 ceph-mon[66861]: pgmap v657: 129 pgs: 129 active+clean; 355 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 392 op/s 2024-03-19T03:41:47.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:46 smithi117 ceph-mon[31838]: pgmap v657: 129 pgs: 129 active+clean; 355 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 392 op/s 2024-03-19T03:41:47.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:46 smithi178 ceph-mon[67578]: pgmap v657: 129 pgs: 129 active+clean; 355 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 392 op/s 2024-03-19T03:41:48.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:48 smithi119 ceph-mon[66861]: pgmap v658: 129 pgs: 129 active+clean; 345 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 738 KiB/s wr, 440 op/s 2024-03-19T03:41:49.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:48 smithi117 ceph-mon[31838]: pgmap v658: 129 pgs: 129 active+clean; 345 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 738 KiB/s wr, 440 op/s 2024-03-19T03:41:49.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:48 smithi178 ceph-mon[67578]: pgmap v658: 129 pgs: 129 active+clean; 345 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 738 KiB/s wr, 440 op/s 2024-03-19T03:41:50.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:50 smithi119 ceph-mon[66861]: pgmap v659: 129 pgs: 129 active+clean; 345 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 846 KiB/s wr, 349 op/s 2024-03-19T03:41:51.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:50 smithi117 ceph-mon[31838]: pgmap v659: 129 pgs: 129 active+clean; 345 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 846 KiB/s wr, 349 op/s 2024-03-19T03:41:51.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:50 smithi178 ceph-mon[67578]: pgmap v659: 129 pgs: 129 active+clean; 345 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 846 KiB/s wr, 349 op/s 2024-03-19T03:41:52.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:52 smithi119 ceph-mon[66861]: pgmap v660: 129 pgs: 129 active+clean; 333 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 435 op/s 2024-03-19T03:41:53.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:52 smithi117 ceph-mon[31838]: pgmap v660: 129 pgs: 129 active+clean; 333 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 435 op/s 2024-03-19T03:41:53.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:52 smithi178 ceph-mon[67578]: pgmap v660: 129 pgs: 129 active+clean; 333 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 435 op/s 2024-03-19T03:41:54.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:54 smithi119 ceph-mon[66861]: pgmap v661: 129 pgs: 129 active+clean; 311 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 993 KiB/s wr, 428 op/s 2024-03-19T03:41:55.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:54 smithi117 ceph-mon[31838]: pgmap v661: 129 pgs: 129 active+clean; 311 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 993 KiB/s wr, 428 op/s 2024-03-19T03:41:55.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:54 smithi178 ceph-mon[67578]: pgmap v661: 129 pgs: 129 active+clean; 311 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 993 KiB/s wr, 428 op/s 2024-03-19T03:41:56.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:56 smithi119 ceph-mon[66861]: pgmap v662: 129 pgs: 129 active+clean; 295 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 780 KiB/s wr, 399 op/s 2024-03-19T03:41:57.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:56 smithi117 ceph-mon[31838]: pgmap v662: 129 pgs: 129 active+clean; 295 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 780 KiB/s wr, 399 op/s 2024-03-19T03:41:57.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:56 smithi178 ceph-mon[67578]: pgmap v662: 129 pgs: 129 active+clean; 295 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 780 KiB/s wr, 399 op/s 2024-03-19T03:41:59.791 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:59 smithi117 ceph-mon[31838]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:41:59.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:41:59 smithi117 ceph-mon[31838]: pgmap v663: 129 pgs: 129 active+clean; 289 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 440 op/s 2024-03-19T03:41:59.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:59 smithi178 ceph-mon[67578]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:41:59.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:41:59 smithi178 ceph-mon[67578]: pgmap v663: 129 pgs: 129 active+clean; 289 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 440 op/s 2024-03-19T03:41:59.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:59 smithi119 ceph-mon[66861]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T03:41:59.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:41:59 smithi119 ceph-mon[66861]: pgmap v663: 129 pgs: 129 active+clean; 289 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 440 op/s 2024-03-19T03:42:00.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:00 smithi119 ceph-mon[66861]: pgmap v664: 129 pgs: 129 active+clean; 286 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 364 op/s 2024-03-19T03:42:01.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:00 smithi117 ceph-mon[31838]: pgmap v664: 129 pgs: 129 active+clean; 286 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 364 op/s 2024-03-19T03:42:01.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:00 smithi178 ceph-mon[67578]: pgmap v664: 129 pgs: 129 active+clean; 286 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 364 op/s 2024-03-19T03:42:02.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:02 smithi119 ceph-mon[66861]: pgmap v665: 129 pgs: 129 active+clean; 279 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 432 op/s 2024-03-19T03:42:03.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:02 smithi117 ceph-mon[31838]: pgmap v665: 129 pgs: 129 active+clean; 279 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 432 op/s 2024-03-19T03:42:03.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:02 smithi178 ceph-mon[67578]: pgmap v665: 129 pgs: 129 active+clean; 279 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 432 op/s 2024-03-19T03:42:04.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:04 smithi119 ceph-mon[66861]: pgmap v666: 129 pgs: 129 active+clean; 272 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 420 op/s 2024-03-19T03:42:05.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:04 smithi117 ceph-mon[31838]: pgmap v666: 129 pgs: 129 active+clean; 272 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 420 op/s 2024-03-19T03:42:05.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:04 smithi178 ceph-mon[67578]: pgmap v666: 129 pgs: 129 active+clean; 272 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 420 op/s 2024-03-19T03:42:06.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:06 smithi119 ceph-mon[66861]: pgmap v667: 129 pgs: 129 active+clean; 266 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 983 KiB/s wr, 391 op/s 2024-03-19T03:42:07.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:06 smithi117 ceph-mon[31838]: pgmap v667: 129 pgs: 129 active+clean; 266 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 983 KiB/s wr, 391 op/s 2024-03-19T03:42:07.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:06 smithi178 ceph-mon[67578]: pgmap v667: 129 pgs: 129 active+clean; 266 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 983 KiB/s wr, 391 op/s 2024-03-19T03:42:08.197 INFO:tasks.workunit:Stopping ['suites/blogbench.sh'] on client.0... 2024-03-19T03:42:08.197 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-19T03:42:08.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:08 smithi119 ceph-mon[66861]: pgmap v668: 129 pgs: 129 active+clean; 257 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 459 op/s 2024-03-19T03:42:09.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:08 smithi117 ceph-mon[31838]: pgmap v668: 129 pgs: 129 active+clean; 257 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 459 op/s 2024-03-19T03:42:09.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:08 smithi178 ceph-mon[67578]: pgmap v668: 129 pgs: 129 active+clean; 257 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 459 op/s 2024-03-19T03:42:09.418 DEBUG:teuthology.parallel:result is None 2024-03-19T03:42:09.418 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-19T03:42:09.512 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-19T03:42:09.512 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-19T03:42:09.523 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:42:09.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.719+0000 7f65d7200640 1 -- 172.21.15.117:0/3671086294 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65d8102880 msgr2=0x7f65d8102d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:09.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.719+0000 7f65d7200640 1 --2- 172.21.15.117:0/3671086294 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65d8102880 0x7f65d8102d00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f65c0009900 tx=0x7f65c002bc30 comp rx=0 tx=0).stop 2024-03-19T03:42:09.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.720+0000 7f65d7200640 1 -- 172.21.15.117:0/3671086294 shutdown_connections 2024-03-19T03:42:09.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.720+0000 7f65d7200640 1 --2- 172.21.15.117:0/3671086294 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65d8103240 0x7f65d8109ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:09.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.720+0000 7f65d7200640 1 --2- 172.21.15.117:0/3671086294 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65d8102880 0x7f65d8102d00 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:09.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.720+0000 7f65d7200640 1 --2- 172.21.15.117:0/3671086294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65d8101690 0x7f65d8101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:09.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.720+0000 7f65d7200640 1 -- 172.21.15.117:0/3671086294 >> 172.21.15.117:0/3671086294 conn(0x7f65d80fcc00 msgr2=0x7f65d80ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:09.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.720+0000 7f65d7200640 1 -- 172.21.15.117:0/3671086294 shutdown_connections 2024-03-19T03:42:09.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.720+0000 7f65d7200640 1 -- 172.21.15.117:0/3671086294 wait complete. 2024-03-19T03:42:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.721+0000 7f65d7200640 1 Processor -- start 2024-03-19T03:42:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.721+0000 7f65d7200640 1 -- start start 2024-03-19T03:42:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.721+0000 7f65d7200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65d8101690 0x7f65d8197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.721+0000 7f65d7200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65d8102880 0x7f65d8197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.721+0000 7f65d7200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65d8103240 0x7f65d819ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.722+0000 7f65d7200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f65d810d340 con 0x7f65d8102880 2024-03-19T03:42:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.722+0000 7f65d7200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f65d810d1c0 con 0x7f65d8103240 2024-03-19T03:42:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.722+0000 7f65d7200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f65d810d4c0 con 0x7f65d8101690 2024-03-19T03:42:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.722+0000 7f65cfe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65d8102880 0x7f65d8197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.722+0000 7f65d6800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65d8103240 0x7f65d819ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.722+0000 7f65cfe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65d8102880 0x7f65d8197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39272/0 (socket says 172.21.15.117:39272) 2024-03-19T03:42:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.722+0000 7f65cfe00640 1 -- 172.21.15.117:0/634836640 learned_addr learned my addr 172.21.15.117:0/634836640 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:42:09.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.722+0000 7f65d5e00640 1 --2- 172.21.15.117:0/634836640 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65d8101690 0x7f65d8197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:09.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.722+0000 7f65d6800640 1 -- 172.21.15.117:0/634836640 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65d8101690 msgr2=0x7f65d8197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:09.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.722+0000 7f65d6800640 1 --2- 172.21.15.117:0/634836640 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65d8101690 0x7f65d8197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:09.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.722+0000 7f65d6800640 1 -- 172.21.15.117:0/634836640 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65d8102880 msgr2=0x7f65d8197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:09.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.722+0000 7f65d6800640 1 --2- 172.21.15.117:0/634836640 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65d8102880 0x7f65d8197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:09.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.722+0000 7f65d6800640 1 -- 172.21.15.117:0/634836640 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65d819f570 con 0x7f65d8103240 2024-03-19T03:42:09.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.722+0000 7f65cfe00640 1 --2- 172.21.15.117:0/634836640 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65d8102880 0x7f65d8197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:42:09.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.723+0000 7f65d6800640 1 --2- 172.21.15.117:0/634836640 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65d8103240 0x7f65d819ef00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f65d000e940 tx=0x7f65d000ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:09.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.723+0000 7f65cea00640 1 -- 172.21.15.117:0/634836640 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d0004d20 con 0x7f65d8103240 2024-03-19T03:42:09.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.723+0000 7f65cea00640 1 -- 172.21.15.117:0/634836640 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65d0005020 con 0x7f65d8103240 2024-03-19T03:42:09.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.723+0000 7f65d7200640 1 -- 172.21.15.117:0/634836640 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65d819f860 con 0x7f65d8103240 2024-03-19T03:42:09.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.723+0000 7f65cea00640 1 -- 172.21.15.117:0/634836640 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d00116b0 con 0x7f65d8103240 2024-03-19T03:42:09.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.723+0000 7f65d7200640 1 -- 172.21.15.117:0/634836640 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65d819fd30 con 0x7f65d8103240 2024-03-19T03:42:09.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.724+0000 7f65d7200640 1 -- 172.21.15.117:0/634836640 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65d8108a00 con 0x7f65d8103240 2024-03-19T03:42:09.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.726+0000 7f65cea00640 1 -- 172.21.15.117:0/634836640 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 20) ==== 137291+0+0 (secure 0 0 0) 0x7f65d00054d0 con 0x7f65d8103240 2024-03-19T03:42:09.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.728+0000 7f65cea00640 1 --2- 172.21.15.117:0/634836640 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f65bc0a2800 0x7f65bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:09.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.728+0000 7f65cea00640 1 -- 172.21.15.117:0/634836640 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f65d00cf2c0 con 0x7f65d8103240 2024-03-19T03:42:09.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.728+0000 7f65d5e00640 1 --2- 172.21.15.117:0/634836640 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f65bc0a2800 0x7f65bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:09.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.729+0000 7f65d5e00640 1 --2- 172.21.15.117:0/634836640 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f65bc0a2800 0x7f65bc0a4cc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f65d81026e0 tx=0x7f65c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:09.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.734+0000 7f65cea00640 1 -- 172.21.15.117:0/634836640 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f65d0016030 con 0x7f65d8103240 2024-03-19T03:42:09.991 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.990+0000 7f65d7200640 1 -- 172.21.15.117:0/634836640 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f65d802d290 con 0x7f65d8103240 2024-03-19T03:42:09.992 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.991+0000 7f65cea00640 1 -- 172.21.15.117:0/634836640 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24989 (secure 0 0 0) 0x7f65d009b8c0 con 0x7f65d8103240 2024-03-19T03:42:09.993 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:42:09.993 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":93,"fsid":"a3a82456-e59f-11ee-95c9-87774f69a715","created":"2024-03-19T03:21:30.427136+0000","modified":"2024-03-19T03:32:40.453890+0000","last_up_change":"2024-03-19T03:29:32.967505+0000","last_in_change":"2024-03-19T03:29:12.848740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:25:18.824270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T03:30:33.137170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T03:30:34.598838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T03:30:38.230023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b32eb0ee-9d4b-4748-87ec-eda02c907415","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6803","nonce":3014934900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6805","nonce":3014934900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6809","nonce":3014934900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6807","nonce":3014934900}]},"public_addr":"172.21.15.117:6803/3014934900","cluster_addr":"172.21.15.117:6805/3014934900","heartbeat_back_addr":"172.21.15.117:6809/3014934900","heartbeat_front_addr":"172.21.15.117:6807/3014934900","state":["exists","up"]},{"osd":1,"uuid":"e07585be-c016-4466-b07c-abbc22a58de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6801","nonce":2425927341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6803","nonce":2425927341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6807","nonce":2425927341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6805","nonce":2425927341}]},"public_addr":"172.21.15.119:6801/2425927341","cluster_addr":"172.21.15.119:6803/2425927341","heartbeat_back_addr":"172.21.15.119:6807/2425927341","heartbeat_front_addr":"172.21.15.119:6805/2425927341","state":["exists","up"]},{"osd":2,"uuid":"ede9edda-3b9f-41ca-ad04-5dcb738ab757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6801","nonce":3733812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6803","nonce":3733812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6807","nonce":3733812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6805","nonce":3733812989}]},"public_addr":"172.21.15.178:6801/3733812989","cluster_addr":"172.21.15.178:6803/3733812989","heartbeat_back_addr":"172.21.15.178:6807/3733812989","heartbeat_front_addr":"172.21.15.178:6805/3733812989","state":["exists","up"]},{"osd":3,"uuid":"f81c41e1-3a69-40db-a90f-58151286b081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6811","nonce":2178310135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6813","nonce":2178310135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6817","nonce":2178310135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6815","nonce":2178310135}]},"public_addr":"172.21.15.117:6811/2178310135","cluster_addr":"172.21.15.117:6813/2178310135","heartbeat_back_addr":"172.21.15.117:6817/2178310135","heartbeat_front_addr":"172.21.15.117:6815/2178310135","state":["exists","up"]},{"osd":4,"uuid":"caf7ce86-caf0-4eaf-bbab-bd2d93e63c33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6809","nonce":3455398086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6811","nonce":3455398086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6815","nonce":3455398086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6813","nonce":3455398086}]},"public_addr":"172.21.15.119:6809/3455398086","cluster_addr":"172.21.15.119:6811/3455398086","heartbeat_back_addr":"172.21.15.119:6815/3455398086","heartbeat_front_addr":"172.21.15.119:6813/3455398086","state":["exists","up"]},{"osd":5,"uuid":"20ba0b8a-2e0f-4ea7-b435-430e44766200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6809","nonce":1259631912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6811","nonce":1259631912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6815","nonce":1259631912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6813","nonce":1259631912}]},"public_addr":"172.21.15.178:6809/1259631912","cluster_addr":"172.21.15.178:6811/1259631912","heartbeat_back_addr":"172.21.15.178:6815/1259631912","heartbeat_front_addr":"172.21.15.178:6813/1259631912","state":["exists","up"]},{"osd":6,"uuid":"75c2ddd7-f837-42e0-ac47-8cfbcf97a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6819","nonce":864146757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6821","nonce":864146757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6825","nonce":864146757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6823","nonce":864146757}]},"public_addr":"172.21.15.117:6819/864146757","cluster_addr":"172.21.15.117:6821/864146757","heartbeat_back_addr":"172.21.15.117:6825/864146757","heartbeat_front_addr":"172.21.15.117:6823/864146757","state":["exists","up"]},{"osd":7,"uuid":"929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6817","nonce":1273365545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6819","nonce":1273365545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6823","nonce":1273365545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6821","nonce":1273365545}]},"public_addr":"172.21.15.119:6817/1273365545","cluster_addr":"172.21.15.119:6819/1273365545","heartbeat_back_addr":"172.21.15.119:6823/1273365545","heartbeat_front_addr":"172.21.15.119:6821/1273365545","state":["exists","up"]},{"osd":8,"uuid":"fd371461-a538-4781-92e1-5eead4ca3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6817","nonce":1059587501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6819","nonce":1059587501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6823","nonce":1059587501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6821","nonce":1059587501}]},"public_addr":"172.21.15.178:6817/1059587501","cluster_addr":"172.21.15.178:6819/1059587501","heartbeat_back_addr":"172.21.15.178:6823/1059587501","heartbeat_front_addr":"172.21.15.178:6821/1059587501","state":["exists","up"]},{"osd":9,"uuid":"0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6827","nonce":3653875600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6829","nonce":3653875600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6833","nonce":3653875600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6831","nonce":3653875600}]},"public_addr":"172.21.15.117:6827/3653875600","cluster_addr":"172.21.15.117:6829/3653875600","heartbeat_back_addr":"172.21.15.117:6833/3653875600","heartbeat_front_addr":"172.21.15.117:6831/3653875600","state":["exists","up"]},{"osd":10,"uuid":"de2049da-d19e-4386-afbf-fd6b5fddbcba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6825","nonce":1050976194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6827","nonce":1050976194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6831","nonce":1050976194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6829","nonce":1050976194}]},"public_addr":"172.21.15.119:6825/1050976194","cluster_addr":"172.21.15.119:6827/1050976194","heartbeat_back_addr":"172.21.15.119:6831/1050976194","heartbeat_front_addr":"172.21.15.119:6829/1050976194","state":["exists","up"]},{"osd":11,"uuid":"3cccc0f9-d273-4b6b-9b8e-c926371472dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6825","nonce":2915080368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6827","nonce":2915080368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6831","nonce":2915080368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6829","nonce":2915080368}]},"public_addr":"172.21.15.178:6825/2915080368","cluster_addr":"172.21.15.178:6827/2915080368","heartbeat_back_addr":"172.21.15.178:6831/2915080368","heartbeat_front_addr":"172.21.15.178:6829/2915080368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:27.876274+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:53.094706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:16.598813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:48.160419+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:14.048925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:38.814304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:12.542328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:38.942747+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:04.488963+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:38.196053+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:05.254728+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:30.874915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7},{"from":5,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.117:0/185684447":"2024-03-20T03:22:38.134578+0000","172.21.15.117:0/361367101":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/2788067603":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3506200675":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6800/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3283471541":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3750920169":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/2780813298":"2024-03-20T03:22:03.905461+0000","172.21.15.117:6800/2788067603":"2024-03-20T03:22:38.134578+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:42:09.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.994+0000 7f65d7200640 1 -- 172.21.15.117:0/634836640 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f65bc0a2800 msgr2=0x7f65bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:09.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.994+0000 7f65d7200640 1 --2- 172.21.15.117:0/634836640 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f65bc0a2800 0x7f65bc0a4cc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f65d81026e0 tx=0x7f65c8005f90 comp rx=0 tx=0).stop 2024-03-19T03:42:09.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.994+0000 7f65d7200640 1 -- 172.21.15.117:0/634836640 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65d8103240 msgr2=0x7f65d819ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:09.995 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.994+0000 7f65d7200640 1 --2- 172.21.15.117:0/634836640 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65d8103240 0x7f65d819ef00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f65d000e940 tx=0x7f65d000ee00 comp rx=0 tx=0).stop 2024-03-19T03:42:09.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.994+0000 7f65d7200640 1 -- 172.21.15.117:0/634836640 shutdown_connections 2024-03-19T03:42:09.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.994+0000 7f65d7200640 1 --2- 172.21.15.117:0/634836640 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f65bc0a2800 0x7f65bc0a4cc0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:09.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.994+0000 7f65d7200640 1 --2- 172.21.15.117:0/634836640 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65d8103240 0x7f65d819ef00 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:09.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.994+0000 7f65d7200640 1 --2- 172.21.15.117:0/634836640 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65d8102880 0x7f65d8197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:09.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.994+0000 7f65d7200640 1 --2- 172.21.15.117:0/634836640 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65d8101690 0x7f65d8197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:09.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.994+0000 7f65d7200640 1 -- 172.21.15.117:0/634836640 >> 172.21.15.117:0/634836640 conn(0x7f65d80fcc00 msgr2=0x7f65d8105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:09.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.994+0000 7f65d7200640 1 -- 172.21.15.117:0/634836640 shutdown_connections 2024-03-19T03:42:09.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:09.994+0000 7f65d7200640 1 -- 172.21.15.117:0/634836640 wait complete. 2024-03-19T03:42:10.014 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T03:25:18.824270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T03:30:33.137170+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T03:30:34.598838+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T03:30:38.230023+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T03:42:10.015 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T03:42:10.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.222+0000 7f8b3b200640 1 -- 172.21.15.117:0/3218810040 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b34101700 msgr2=0x7f8b34101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:10.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.222+0000 7f8b3b200640 1 --2- 172.21.15.117:0/3218810040 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b34101700 0x7f8b34101b20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8b28009860 tx=0x7f8b2802bb70 comp rx=0 tx=0).stop 2024-03-19T03:42:10.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.222+0000 7f8b3b200640 1 -- 172.21.15.117:0/3218810040 shutdown_connections 2024-03-19T03:42:10.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.222+0000 7f8b3b200640 1 --2- 172.21.15.117:0/3218810040 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b341032b0 0x7f8b34109b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:10.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.222+0000 7f8b3b200640 1 --2- 172.21.15.117:0/3218810040 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b341028f0 0x7f8b34102d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:10.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.222+0000 7f8b3b200640 1 --2- 172.21.15.117:0/3218810040 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b34101700 0x7f8b34101b20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:10.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.222+0000 7f8b3b200640 1 -- 172.21.15.117:0/3218810040 >> 172.21.15.117:0/3218810040 conn(0x7f8b340fcc90 msgr2=0x7f8b340ff0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:10.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.222+0000 7f8b3b200640 1 -- 172.21.15.117:0/3218810040 shutdown_connections 2024-03-19T03:42:10.224 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.222+0000 7f8b3b200640 1 -- 172.21.15.117:0/3218810040 wait complete. 2024-03-19T03:42:10.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.223+0000 7f8b3b200640 1 Processor -- start 2024-03-19T03:42:10.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.223+0000 7f8b3b200640 1 -- start start 2024-03-19T03:42:10.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.224+0000 7f8b3b200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b34101700 0x7f8b340755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:10.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.224+0000 7f8b3b200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b341028f0 0x7f8b34073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:10.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.224+0000 7f8b3b200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b341032b0 0x7f8b34074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:10.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.224+0000 7f8b3b200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8b3410d350 con 0x7f8b341028f0 2024-03-19T03:42:10.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.224+0000 7f8b3b200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8b3410d1d0 con 0x7f8b34101700 2024-03-19T03:42:10.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.224+0000 7f8b3b200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f8b3410d4d0 con 0x7f8b341032b0 2024-03-19T03:42:10.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.224+0000 7f8b39e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b34101700 0x7f8b340755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:10.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.224+0000 7f8b39400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b341028f0 0x7f8b34073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:10.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.224+0000 7f8b39e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b34101700 0x7f8b340755d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:56158/0 (socket says 172.21.15.117:56158) 2024-03-19T03:42:10.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.224+0000 7f8b39e00640 1 -- 172.21.15.117:0/2248735895 learned_addr learned my addr 172.21.15.117:0/2248735895 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:42:10.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.224+0000 7f8b3a800640 1 --2- 172.21.15.117:0/2248735895 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b341032b0 0x7f8b34074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:10.227 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.225+0000 7f8b39e00640 1 -- 172.21.15.117:0/2248735895 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b341032b0 msgr2=0x7f8b34074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:10.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.225+0000 7f8b39e00640 1 --2- 172.21.15.117:0/2248735895 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b341032b0 0x7f8b34074150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:10.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.225+0000 7f8b39e00640 1 -- 172.21.15.117:0/2248735895 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b341028f0 msgr2=0x7f8b34073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:10.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.225+0000 7f8b39e00640 1 --2- 172.21.15.117:0/2248735895 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b341028f0 0x7f8b34073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:10.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.225+0000 7f8b39e00640 1 -- 172.21.15.117:0/2248735895 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b340749e0 con 0x7f8b34101700 2024-03-19T03:42:10.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.225+0000 7f8b39e00640 1 --2- 172.21.15.117:0/2248735895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b34101700 0x7f8b340755d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8b280349e0 tx=0x7f8b28032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:10.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.225+0000 7f8b2ea00640 1 -- 172.21.15.117:0/2248735895 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b28041020 con 0x7f8b34101700 2024-03-19T03:42:10.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.225+0000 7f8b3b200640 1 -- 172.21.15.117:0/2248735895 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b341a1a90 con 0x7f8b34101700 2024-03-19T03:42:10.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.225+0000 7f8b2ea00640 1 -- 172.21.15.117:0/2248735895 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b2802c9f0 con 0x7f8b34101700 2024-03-19T03:42:10.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.225+0000 7f8b2ea00640 1 -- 172.21.15.117:0/2248735895 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b28039040 con 0x7f8b34101700 2024-03-19T03:42:10.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.226+0000 7f8b3b200640 1 -- 172.21.15.117:0/2248735895 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b341a2030 con 0x7f8b34101700 2024-03-19T03:42:10.228 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.227+0000 7f8b3b200640 1 -- 172.21.15.117:0/2248735895 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b34108a20 con 0x7f8b34101700 2024-03-19T03:42:10.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.230+0000 7f8b2ea00640 1 -- 172.21.15.117:0/2248735895 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 20) ==== 137291+0+0 (secure 0 0 0) 0x7f8b28044020 con 0x7f8b34101700 2024-03-19T03:42:10.232 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.231+0000 7f8b2ea00640 1 --2- 172.21.15.117:0/2248735895 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8b1c0a2800 0x7f8b1c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:10.232 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.231+0000 7f8b2ea00640 1 -- 172.21.15.117:0/2248735895 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f8b280f09b0 con 0x7f8b34101700 2024-03-19T03:42:10.232 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.231+0000 7f8b39400640 1 --2- 172.21.15.117:0/2248735895 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8b1c0a2800 0x7f8b1c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:10.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.232+0000 7f8b39400640 1 --2- 172.21.15.117:0/2248735895 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8b1c0a2800 0x7f8b1c0a4cc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8b34075340 tx=0x7f8b24009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:10.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.232+0000 7f8b2ea00640 1 -- 172.21.15.117:0/2248735895 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8b2804b020 con 0x7f8b34101700 2024-03-19T03:42:10.291 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:10 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/634836640' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:42:10.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:10 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/634836640' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:42:10.431 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.430+0000 7f8b3b200640 1 -- 172.21.15.117:0/2248735895 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8b3402d380 con 0x7f8b34101700 2024-03-19T03:42:10.432 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.431+0000 7f8b2ea00640 1 -- 172.21.15.117:0/2248735895 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f8b280bd660 con 0x7f8b34101700 2024-03-19T03:42:10.432 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 1 2024-03-19T03:42:10.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.433+0000 7f8b3b200640 1 -- 172.21.15.117:0/2248735895 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8b1c0a2800 msgr2=0x7f8b1c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:10.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.433+0000 7f8b3b200640 1 --2- 172.21.15.117:0/2248735895 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8b1c0a2800 0x7f8b1c0a4cc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8b34075340 tx=0x7f8b24009260 comp rx=0 tx=0).stop 2024-03-19T03:42:10.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.433+0000 7f8b3b200640 1 -- 172.21.15.117:0/2248735895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b34101700 msgr2=0x7f8b340755d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:10.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.433+0000 7f8b3b200640 1 --2- 172.21.15.117:0/2248735895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b34101700 0x7f8b340755d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8b280349e0 tx=0x7f8b28032430 comp rx=0 tx=0).stop 2024-03-19T03:42:10.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.434+0000 7f8b3b200640 1 -- 172.21.15.117:0/2248735895 shutdown_connections 2024-03-19T03:42:10.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.434+0000 7f8b3b200640 1 --2- 172.21.15.117:0/2248735895 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b341032b0 0x7f8b34074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:10.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.434+0000 7f8b3b200640 1 --2- 172.21.15.117:0/2248735895 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b341028f0 0x7f8b34073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:10.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.434+0000 7f8b3b200640 1 --2- 172.21.15.117:0/2248735895 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8b1c0a2800 0x7f8b1c0a4cc0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:10.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.434+0000 7f8b3b200640 1 --2- 172.21.15.117:0/2248735895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b34101700 0x7f8b340755d0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:10.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.434+0000 7f8b3b200640 1 -- 172.21.15.117:0/2248735895 >> 172.21.15.117:0/2248735895 conn(0x7f8b340fcc90 msgr2=0x7f8b34105bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:10.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.434+0000 7f8b3b200640 1 -- 172.21.15.117:0/2248735895 shutdown_connections 2024-03-19T03:42:10.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.434+0000 7f8b3b200640 1 -- 172.21.15.117:0/2248735895 wait complete. 2024-03-19T03:42:10.453 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T03:42:10.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:10 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/634836640' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:42:10.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.650+0000 7f183a000640 1 -- 172.21.15.117:0/1392275785 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1834105290 msgr2=0x7f1834105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:10.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.650+0000 7f183a000640 1 --2- 172.21.15.117:0/1392275785 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1834105290 0x7f1834105710 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f182c009ca0 tx=0x7f182c02c100 comp rx=0 tx=0).stop 2024-03-19T03:42:10.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.651+0000 7f183a000640 1 -- 172.21.15.117:0/1392275785 shutdown_connections 2024-03-19T03:42:10.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.651+0000 7f183a000640 1 --2- 172.21.15.117:0/1392275785 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1834105290 0x7f1834105710 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:10.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.651+0000 7f183a000640 1 --2- 172.21.15.117:0/1392275785 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18341028e0 0x7f1834104d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:10.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.651+0000 7f183a000640 1 --2- 172.21.15.117:0/1392275785 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18340fff70 0x7f18341023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:10.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.651+0000 7f183a000640 1 -- 172.21.15.117:0/1392275785 >> 172.21.15.117:0/1392275785 conn(0x7f18340fbb90 msgr2=0x7f18340fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:10.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.651+0000 7f183a000640 1 -- 172.21.15.117:0/1392275785 shutdown_connections 2024-03-19T03:42:10.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.651+0000 7f183a000640 1 -- 172.21.15.117:0/1392275785 wait complete. 2024-03-19T03:42:10.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.652+0000 7f183a000640 1 Processor -- start 2024-03-19T03:42:10.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.652+0000 7f183a000640 1 -- start start 2024-03-19T03:42:10.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.653+0000 7f183a000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18340fff70 0x7f183419df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:10.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.653+0000 7f183a000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18341028e0 0x7f183419e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:10.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.653+0000 7f183a000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1834105290 0x7f18341a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:10.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.653+0000 7f183a000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1834113950 con 0x7f18340fff70 2024-03-19T03:42:10.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.653+0000 7f183a000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f18341137d0 con 0x7f1834105290 2024-03-19T03:42:10.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.653+0000 7f183a000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f1834113ad0 con 0x7f18341028e0 2024-03-19T03:42:10.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.653+0000 7f1838c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18340fff70 0x7f183419df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:10.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.653+0000 7f182be00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18341028e0 0x7f183419e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:10.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.653+0000 7f182be00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18341028e0 0x7f183419e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:39822/0 (socket says 172.21.15.117:39822) 2024-03-19T03:42:10.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.653+0000 7f182be00640 1 -- 172.21.15.117:0/3908485860 learned_addr learned my addr 172.21.15.117:0/3908485860 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:42:10.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.653+0000 7f1839600640 1 --2- 172.21.15.117:0/3908485860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1834105290 0x7f18341a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:10.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.654+0000 7f182be00640 1 -- 172.21.15.117:0/3908485860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1834105290 msgr2=0x7f18341a54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:10.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.654+0000 7f182be00640 1 --2- 172.21.15.117:0/3908485860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1834105290 0x7f18341a54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:10.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.654+0000 7f182be00640 1 -- 172.21.15.117:0/3908485860 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18340fff70 msgr2=0x7f183419df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:10.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.654+0000 7f182be00640 1 --2- 172.21.15.117:0/3908485860 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18340fff70 0x7f183419df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:10.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.654+0000 7f182be00640 1 -- 172.21.15.117:0/3908485860 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18341a5b60 con 0x7f18341028e0 2024-03-19T03:42:10.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.654+0000 7f1839600640 1 --2- 172.21.15.117:0/3908485860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1834105290 0x7f18341a54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:42:10.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.654+0000 7f182be00640 1 --2- 172.21.15.117:0/3908485860 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18341028e0 0x7f183419e480 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f181c00e9f0 tx=0x7f181c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:10.657 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.654+0000 7f1832000640 1 -- 172.21.15.117:0/3908485860 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c00cd80 con 0x7f18341028e0 2024-03-19T03:42:10.657 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.654+0000 7f1832000640 1 -- 172.21.15.117:0/3908485860 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f181c010910 con 0x7f18341028e0 2024-03-19T03:42:10.657 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.654+0000 7f1832000640 1 -- 172.21.15.117:0/3908485860 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c018990 con 0x7f18341028e0 2024-03-19T03:42:10.657 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.654+0000 7f183a000640 1 -- 172.21.15.117:0/3908485860 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18341a5e50 con 0x7f18341028e0 2024-03-19T03:42:10.657 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.655+0000 7f183a000640 1 -- 172.21.15.117:0/3908485860 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18341a62c0 con 0x7f18341028e0 2024-03-19T03:42:10.657 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.656+0000 7f183a000640 1 -- 172.21.15.117:0/3908485860 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18340fd680 con 0x7f18341028e0 2024-03-19T03:42:10.659 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.658+0000 7f1832000640 1 -- 172.21.15.117:0/3908485860 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 20) ==== 137291+0+0 (secure 0 0 0) 0x7f181c010ac0 con 0x7f18341028e0 2024-03-19T03:42:10.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.660+0000 7f1832000640 1 --2- 172.21.15.117:0/3908485860 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f18180a2880 0x7f18180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:10.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.660+0000 7f1832000640 1 -- 172.21.15.117:0/3908485860 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f181c0cf3d0 con 0x7f18341028e0 2024-03-19T03:42:10.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.660+0000 7f1838c00640 1 --2- 172.21.15.117:0/3908485860 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f18180a2880 0x7f18180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:10.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.661+0000 7f1838c00640 1 --2- 172.21.15.117:0/3908485860 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f18180a2880 0x7f18180a4d40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f18240095f0 tx=0x7f1824005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:10.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.661+0000 7f1832000640 1 -- 172.21.15.117:0/3908485860 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f181c09bad0 con 0x7f18341028e0 2024-03-19T03:42:10.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.895+0000 7f183a000640 1 -- 172.21.15.117:0/3908485860 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f183402d380 con 0x7f18341028e0 2024-03-19T03:42:10.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.896+0000 7f1832000640 1 -- 172.21.15.117:0/3908485860 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f181c09fb70 con 0x7f18341028e0 2024-03-19T03:42:10.897 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 32 2024-03-19T03:42:10.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.899+0000 7f183a000640 1 -- 172.21.15.117:0/3908485860 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f18180a2880 msgr2=0x7f18180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:10.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.899+0000 7f183a000640 1 --2- 172.21.15.117:0/3908485860 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f18180a2880 0x7f18180a4d40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f18240095f0 tx=0x7f1824005f90 comp rx=0 tx=0).stop 2024-03-19T03:42:10.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.899+0000 7f183a000640 1 -- 172.21.15.117:0/3908485860 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18341028e0 msgr2=0x7f183419e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:10.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.899+0000 7f183a000640 1 --2- 172.21.15.117:0/3908485860 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18341028e0 0x7f183419e480 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f181c00e9f0 tx=0x7f181c00eeb0 comp rx=0 tx=0).stop 2024-03-19T03:42:10.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.899+0000 7f183a000640 1 -- 172.21.15.117:0/3908485860 shutdown_connections 2024-03-19T03:42:10.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.899+0000 7f183a000640 1 --2- 172.21.15.117:0/3908485860 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f18180a2880 0x7f18180a4d40 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:10.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.899+0000 7f183a000640 1 --2- 172.21.15.117:0/3908485860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1834105290 0x7f18341a54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:10.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.899+0000 7f183a000640 1 --2- 172.21.15.117:0/3908485860 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18341028e0 0x7f183419e480 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:10.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.899+0000 7f183a000640 1 --2- 172.21.15.117:0/3908485860 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18340fff70 0x7f183419df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:10.900 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.899+0000 7f183a000640 1 -- 172.21.15.117:0/3908485860 >> 172.21.15.117:0/3908485860 conn(0x7f18340fbb90 msgr2=0x7f18341008f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:10.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.899+0000 7f183a000640 1 -- 172.21.15.117:0/3908485860 shutdown_connections 2024-03-19T03:42:10.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:10.899+0000 7f183a000640 1 -- 172.21.15.117:0/3908485860 wait complete. 2024-03-19T03:42:10.925 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T03:42:11.139 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.138+0000 7f2419800640 1 -- 172.21.15.117:0/3245538646 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2414104ab0 msgr2=0x7f2414106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:11.139 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.138+0000 7f2419800640 1 --2- 172.21.15.117:0/3245538646 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2414104ab0 0x7f2414106ea0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2400009900 tx=0x7f240002bc40 comp rx=0 tx=0).stop 2024-03-19T03:42:11.139 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.138+0000 7f2419800640 1 -- 172.21.15.117:0/3245538646 shutdown_connections 2024-03-19T03:42:11.140 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.138+0000 7f2419800640 1 --2- 172.21.15.117:0/3245538646 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24141073e0 0x7f2414109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:11.140 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.138+0000 7f2419800640 1 --2- 172.21.15.117:0/3245538646 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2414104ab0 0x7f2414106ea0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:11.140 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.138+0000 7f2419800640 1 --2- 172.21.15.117:0/3245538646 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2414102180 0x7f2414104570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:11.140 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.138+0000 7f2419800640 1 -- 172.21.15.117:0/3245538646 >> 172.21.15.117:0/3245538646 conn(0x7f24140fbb90 msgr2=0x7f24140fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:11.140 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.138+0000 7f2419800640 1 -- 172.21.15.117:0/3245538646 shutdown_connections 2024-03-19T03:42:11.140 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.138+0000 7f2419800640 1 -- 172.21.15.117:0/3245538646 wait complete. 2024-03-19T03:42:11.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.139+0000 7f2419800640 1 Processor -- start 2024-03-19T03:42:11.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.139+0000 7f2419800640 1 -- start start 2024-03-19T03:42:11.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.140+0000 7f2419800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2414102180 0x7f2414199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:11.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.140+0000 7f2419800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2414104ab0 0x7f241419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:11.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.140+0000 7f2419800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24141073e0 0x7f24141a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:11.142 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.140+0000 7f2419800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f241410d3e0 con 0x7f2414104ab0 2024-03-19T03:42:11.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.140+0000 7f2419800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f241410d260 con 0x7f24141073e0 2024-03-19T03:42:11.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.140+0000 7f2419800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f241410d560 con 0x7f2414102180 2024-03-19T03:42:11.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.140+0000 7f2413400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2414104ab0 0x7f241419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:11.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.140+0000 7f2418e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24141073e0 0x7f24141a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:11.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.140+0000 7f2418e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24141073e0 0x7f24141a1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:56206/0 (socket says 172.21.15.117:56206) 2024-03-19T03:42:11.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.140+0000 7f2418e00640 1 -- 172.21.15.117:0/597682855 learned_addr learned my addr 172.21.15.117:0/597682855 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:42:11.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.140+0000 7f2413e00640 1 --2- 172.21.15.117:0/597682855 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2414102180 0x7f2414199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:11.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.140+0000 7f2418e00640 1 -- 172.21.15.117:0/597682855 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2414102180 msgr2=0x7f2414199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:11.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.140+0000 7f2418e00640 1 --2- 172.21.15.117:0/597682855 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2414102180 0x7f2414199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:11.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.140+0000 7f2418e00640 1 -- 172.21.15.117:0/597682855 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2414104ab0 msgr2=0x7f241419a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:11.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.140+0000 7f2418e00640 1 --2- 172.21.15.117:0/597682855 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2414104ab0 0x7f241419a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:11.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.140+0000 7f2418e00640 1 -- 172.21.15.117:0/597682855 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24141a18f0 con 0x7f24141073e0 2024-03-19T03:42:11.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.140+0000 7f2413400640 1 --2- 172.21.15.117:0/597682855 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2414104ab0 0x7f241419a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:42:11.143 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.141+0000 7f2418e00640 1 --2- 172.21.15.117:0/597682855 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24141073e0 0x7f24141a1250 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f240c00efd0 tx=0x7f240c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:11.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.141+0000 7f2410c00640 1 -- 172.21.15.117:0/597682855 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f240c003b70 con 0x7f24141073e0 2024-03-19T03:42:11.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.141+0000 7f2410c00640 1 -- 172.21.15.117:0/597682855 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f240c003d10 con 0x7f24141073e0 2024-03-19T03:42:11.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.141+0000 7f2419800640 1 -- 172.21.15.117:0/597682855 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24141a1be0 con 0x7f24141073e0 2024-03-19T03:42:11.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.141+0000 7f2410c00640 1 -- 172.21.15.117:0/597682855 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f240c0109f0 con 0x7f24141073e0 2024-03-19T03:42:11.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.141+0000 7f2419800640 1 -- 172.21.15.117:0/597682855 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24141a20b0 con 0x7f24141073e0 2024-03-19T03:42:11.144 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.142+0000 7f2419800640 1 -- 172.21.15.117:0/597682855 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2414108f30 con 0x7f24141073e0 2024-03-19T03:42:11.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.146+0000 7f2410c00640 1 -- 172.21.15.117:0/597682855 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 20) ==== 137291+0+0 (secure 0 0 0) 0x7f240c004020 con 0x7f24141073e0 2024-03-19T03:42:11.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.147+0000 7f2410c00640 1 --2- 172.21.15.117:0/597682855 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f23f80a2800 0x7f23f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:11.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.147+0000 7f2410c00640 1 -- 172.21.15.117:0/597682855 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f240c0cf2b0 con 0x7f24141073e0 2024-03-19T03:42:11.148 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.147+0000 7f2413e00640 1 --2- 172.21.15.117:0/597682855 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f23f80a2800 0x7f23f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:11.149 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.148+0000 7f2413e00640 1 --2- 172.21.15.117:0/597682855 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f23f80a2800 0x7f23f80a4cc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f241419ab80 tx=0x7f2404005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:11.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.151+0000 7f2410c00640 1 -- 172.21.15.117:0/597682855 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f240c09bf00 con 0x7f24141073e0 2024-03-19T03:42:11.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:11 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2248735895' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:42:11.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:11 smithi117 ceph-mon[31838]: pgmap v669: 129 pgs: 129 active+clean; 254 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 856 KiB/s wr, 386 op/s 2024-03-19T03:42:11.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:11 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3908485860' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:42:11.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:11 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2248735895' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:42:11.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:11 smithi178 ceph-mon[67578]: pgmap v669: 129 pgs: 129 active+clean; 254 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 856 KiB/s wr, 386 op/s 2024-03-19T03:42:11.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:11 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3908485860' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:42:11.421 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.420+0000 7f2419800640 1 -- 172.21.15.117:0/597682855 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2414062420 con 0x7f24141073e0 2024-03-19T03:42:11.421 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.420+0000 7f2410c00640 1 -- 172.21.15.117:0/597682855 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f240c09b800 con 0x7f24141073e0 2024-03-19T03:42:11.422 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 64 2024-03-19T03:42:11.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.423+0000 7f2419800640 1 -- 172.21.15.117:0/597682855 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f23f80a2800 msgr2=0x7f23f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:11.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.423+0000 7f2419800640 1 --2- 172.21.15.117:0/597682855 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f23f80a2800 0x7f23f80a4cc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f241419ab80 tx=0x7f2404005f90 comp rx=0 tx=0).stop 2024-03-19T03:42:11.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.423+0000 7f2419800640 1 -- 172.21.15.117:0/597682855 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24141073e0 msgr2=0x7f24141a1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:11.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.423+0000 7f2419800640 1 --2- 172.21.15.117:0/597682855 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24141073e0 0x7f24141a1250 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f240c00efd0 tx=0x7f240c00c6a0 comp rx=0 tx=0).stop 2024-03-19T03:42:11.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.423+0000 7f2419800640 1 -- 172.21.15.117:0/597682855 shutdown_connections 2024-03-19T03:42:11.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.423+0000 7f2419800640 1 --2- 172.21.15.117:0/597682855 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f23f80a2800 0x7f23f80a4cc0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:11.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.423+0000 7f2419800640 1 --2- 172.21.15.117:0/597682855 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24141073e0 0x7f24141a1250 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:11.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.423+0000 7f2419800640 1 --2- 172.21.15.117:0/597682855 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2414104ab0 0x7f241419a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:11.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.423+0000 7f2419800640 1 --2- 172.21.15.117:0/597682855 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2414102180 0x7f2414199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:11.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.423+0000 7f2419800640 1 -- 172.21.15.117:0/597682855 >> 172.21.15.117:0/597682855 conn(0x7f24140fbb90 msgr2=0x7f24141018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:11.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.424+0000 7f2419800640 1 -- 172.21.15.117:0/597682855 shutdown_connections 2024-03-19T03:42:11.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.424+0000 7f2419800640 1 -- 172.21.15.117:0/597682855 wait complete. 2024-03-19T03:42:11.444 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T03:42:11.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:11 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2248735895' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:42:11.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:11 smithi119 ceph-mon[66861]: pgmap v669: 129 pgs: 129 active+clean; 254 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 856 KiB/s wr, 386 op/s 2024-03-19T03:42:11.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:11 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3908485860' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:42:11.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.651+0000 7f99f3800640 1 -- 172.21.15.117:0/1719152405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec102a40 msgr2=0x7f99ec102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:11.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.651+0000 7f99f3800640 1 --2- 172.21.15.117:0/1719152405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec102a40 0x7f99ec102ec0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f99d8009900 tx=0x7f99d802bc40 comp rx=0 tx=0).stop 2024-03-19T03:42:11.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.651+0000 7f99f3800640 1 -- 172.21.15.117:0/1719152405 shutdown_connections 2024-03-19T03:42:11.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.651+0000 7f99f3800640 1 --2- 172.21.15.117:0/1719152405 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f99ec103400 0x7f99ec109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:11.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.651+0000 7f99f3800640 1 --2- 172.21.15.117:0/1719152405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec102a40 0x7f99ec102ec0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:11.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.651+0000 7f99f3800640 1 --2- 172.21.15.117:0/1719152405 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99ec101850 0x7f99ec101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:11.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.651+0000 7f99f3800640 1 -- 172.21.15.117:0/1719152405 >> 172.21.15.117:0/1719152405 conn(0x7f99ec0fcdc0 msgr2=0x7f99ec0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:11.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.651+0000 7f99f3800640 1 -- 172.21.15.117:0/1719152405 shutdown_connections 2024-03-19T03:42:11.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.651+0000 7f99f3800640 1 -- 172.21.15.117:0/1719152405 wait complete. 2024-03-19T03:42:11.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.653+0000 7f99f3800640 1 Processor -- start 2024-03-19T03:42:11.654 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.653+0000 7f99f3800640 1 -- start start 2024-03-19T03:42:11.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.654+0000 7f99f3800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99ec101850 0x7f99ec197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:11.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.654+0000 7f99f3800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec102a40 0x7f99ec1980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:11.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.654+0000 7f99f3800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f99ec103400 0x7f99ec19f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:11.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.654+0000 7f99f3800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f99ec10d4b0 con 0x7f99ec101850 2024-03-19T03:42:11.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.654+0000 7f99f3800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f99ec10d330 con 0x7f99ec102a40 2024-03-19T03:42:11.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.654+0000 7f99f3800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f99ec10d630 con 0x7f99ec103400 2024-03-19T03:42:11.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.654+0000 7f99f0e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99ec101850 0x7f99ec197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:11.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.654+0000 7f99ebe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec102a40 0x7f99ec1980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:11.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.654+0000 7f99ebe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec102a40 0x7f99ec1980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:56216/0 (socket says 172.21.15.117:56216) 2024-03-19T03:42:11.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.654+0000 7f99f0e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99ec101850 0x7f99ec197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39346/0 (socket says 172.21.15.117:39346) 2024-03-19T03:42:11.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.654+0000 7f99ebe00640 1 -- 172.21.15.117:0/2896438590 learned_addr learned my addr 172.21.15.117:0/2896438590 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:42:11.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.654+0000 7f99f1800640 1 --2- 172.21.15.117:0/2896438590 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f99ec103400 0x7f99ec19f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:11.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.654+0000 7f99ebe00640 1 -- 172.21.15.117:0/2896438590 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f99ec103400 msgr2=0x7f99ec19f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:11.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.654+0000 7f99ebe00640 1 --2- 172.21.15.117:0/2896438590 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f99ec103400 0x7f99ec19f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:11.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.654+0000 7f99ebe00640 1 -- 172.21.15.117:0/2896438590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99ec101850 msgr2=0x7f99ec197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:11.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.654+0000 7f99ebe00640 1 --2- 172.21.15.117:0/2896438590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99ec101850 0x7f99ec197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:11.657 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.654+0000 7f99ebe00640 1 -- 172.21.15.117:0/2896438590 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99ec19f7a0 con 0x7f99ec102a40 2024-03-19T03:42:11.657 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.655+0000 7f99ebe00640 1 --2- 172.21.15.117:0/2896438590 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec102a40 0x7f99ec1980c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f99d8009230 tx=0x7f99d8005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:11.657 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.655+0000 7f99e9600640 1 -- 172.21.15.117:0/2896438590 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d8041070 con 0x7f99ec102a40 2024-03-19T03:42:11.657 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.655+0000 7f99e9600640 1 -- 172.21.15.117:0/2896438590 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99d80054e0 con 0x7f99ec102a40 2024-03-19T03:42:11.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.655+0000 7f99e9600640 1 -- 172.21.15.117:0/2896438590 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d8034440 con 0x7f99ec102a40 2024-03-19T03:42:11.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.655+0000 7f99f3800640 1 -- 172.21.15.117:0/2896438590 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99ec19fa30 con 0x7f99ec102a40 2024-03-19T03:42:11.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.655+0000 7f99f3800640 1 -- 172.21.15.117:0/2896438590 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99ec19fed0 con 0x7f99ec102a40 2024-03-19T03:42:11.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.656+0000 7f99f3800640 1 -- 172.21.15.117:0/2896438590 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99ec108bc0 con 0x7f99ec102a40 2024-03-19T03:42:11.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.660+0000 7f99e9600640 1 -- 172.21.15.117:0/2896438590 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 20) ==== 137291+0+0 (secure 0 0 0) 0x7f99d8044070 con 0x7f99ec102a40 2024-03-19T03:42:11.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.661+0000 7f99e9600640 1 --2- 172.21.15.117:0/2896438590 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f99d40a2880 0x7f99d40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:11.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.662+0000 7f99e9600640 1 -- 172.21.15.117:0/2896438590 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f99d80f0b00 con 0x7f99ec102a40 2024-03-19T03:42:11.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.662+0000 7f99f0e00640 1 --2- 172.21.15.117:0/2896438590 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f99d40a2880 0x7f99d40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:11.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.662+0000 7f99f0e00640 1 --2- 172.21.15.117:0/2896438590 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f99d40a2880 0x7f99d40a4d40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f99e0005fd0 tx=0x7f99e000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:11.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.668+0000 7f99e9600640 1 -- 172.21.15.117:0/2896438590 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f99d80bd100 con 0x7f99ec102a40 2024-03-19T03:42:11.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.926+0000 7f99f3800640 1 -- 172.21.15.117:0/2896438590 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f99ec0624c0 con 0x7f99ec102a40 2024-03-19T03:42:11.928 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.927+0000 7f99e9600640 1 -- 172.21.15.117:0/2896438590 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f99d80c11a0 con 0x7f99ec102a40 2024-03-19T03:42:11.928 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 32 2024-03-19T03:42:11.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.930+0000 7f99f3800640 1 -- 172.21.15.117:0/2896438590 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f99d40a2880 msgr2=0x7f99d40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:11.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.930+0000 7f99f3800640 1 --2- 172.21.15.117:0/2896438590 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f99d40a2880 0x7f99d40a4d40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f99e0005fd0 tx=0x7f99e000b410 comp rx=0 tx=0).stop 2024-03-19T03:42:11.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.930+0000 7f99f3800640 1 -- 172.21.15.117:0/2896438590 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec102a40 msgr2=0x7f99ec1980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:11.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.930+0000 7f99f3800640 1 --2- 172.21.15.117:0/2896438590 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec102a40 0x7f99ec1980c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f99d8009230 tx=0x7f99d8005dd0 comp rx=0 tx=0).stop 2024-03-19T03:42:11.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.930+0000 7f99f3800640 1 -- 172.21.15.117:0/2896438590 shutdown_connections 2024-03-19T03:42:11.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.930+0000 7f99f3800640 1 --2- 172.21.15.117:0/2896438590 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f99ec103400 0x7f99ec19f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:11.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.930+0000 7f99f3800640 1 --2- 172.21.15.117:0/2896438590 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f99d40a2880 0x7f99d40a4d40 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:11.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.930+0000 7f99f3800640 1 --2- 172.21.15.117:0/2896438590 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec102a40 0x7f99ec1980c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:11.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.930+0000 7f99f3800640 1 --2- 172.21.15.117:0/2896438590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99ec101850 0x7f99ec197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:11.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.930+0000 7f99f3800640 1 -- 172.21.15.117:0/2896438590 >> 172.21.15.117:0/2896438590 conn(0x7f99ec0fcdc0 msgr2=0x7f99ec105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:11.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.930+0000 7f99f3800640 1 -- 172.21.15.117:0/2896438590 shutdown_connections 2024-03-19T03:42:11.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:11.930+0000 7f99f3800640 1 -- 172.21.15.117:0/2896438590 wait complete. 2024-03-19T03:42:11.953 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-19T03:42:12.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.144+0000 7f0a29000640 1 -- 172.21.15.117:0/2770901971 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a24105610 msgr2=0x7f0a2410bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:12.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.144+0000 7f0a29000640 1 --2- 172.21.15.117:0/2770901971 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a24105610 0x7f0a2410bea0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c009ca0 tx=0x7f0a1c02c100 comp rx=0 tx=0).stop 2024-03-19T03:42:12.146 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.144+0000 7f0a29000640 1 -- 172.21.15.117:0/2770901971 shutdown_connections 2024-03-19T03:42:12.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.144+0000 7f0a29000640 1 --2- 172.21.15.117:0/2770901971 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a24105610 0x7f0a2410bea0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:12.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.144+0000 7f0a29000640 1 --2- 172.21.15.117:0/2770901971 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a24104c50 0x7f0a241050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:12.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.144+0000 7f0a29000640 1 --2- 172.21.15.117:0/2770901971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a24103a60 0x7f0a24103e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:12.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.144+0000 7f0a29000640 1 -- 172.21.15.117:0/2770901971 >> 172.21.15.117:0/2770901971 conn(0x7f0a240ff010 msgr2=0x7f0a24101430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:12.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.145+0000 7f0a29000640 1 -- 172.21.15.117:0/2770901971 shutdown_connections 2024-03-19T03:42:12.147 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.145+0000 7f0a29000640 1 -- 172.21.15.117:0/2770901971 wait complete. 2024-03-19T03:42:12.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.146+0000 7f0a29000640 1 Processor -- start 2024-03-19T03:42:12.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.146+0000 7f0a29000640 1 -- start start 2024-03-19T03:42:12.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.146+0000 7f0a29000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a24103a60 0x7f0a24199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:12.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.146+0000 7f0a29000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a24104c50 0x7f0a2419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:12.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.146+0000 7f0a29000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a24105610 0x7f0a241a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:12.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.146+0000 7f0a29000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2410f6c0 con 0x7f0a24104c50 2024-03-19T03:42:12.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.146+0000 7f0a29000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2410f540 con 0x7f0a24105610 2024-03-19T03:42:12.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.146+0000 7f0a29000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2410f840 con 0x7f0a24103a60 2024-03-19T03:42:12.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.146+0000 7f0a21e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a24103a60 0x7f0a24199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:12.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.146+0000 7f0a22800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a24105610 0x7f0a241a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:12.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.146+0000 7f0a21e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a24103a60 0x7f0a24199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:39870/0 (socket says 172.21.15.117:39870) 2024-03-19T03:42:12.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.146+0000 7f0a21e00640 1 -- 172.21.15.117:0/2125650409 learned_addr learned my addr 172.21.15.117:0/2125650409 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:42:12.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.147+0000 7f0a21400640 1 --2- 172.21.15.117:0/2125650409 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a24104c50 0x7f0a2419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:12.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.147+0000 7f0a22800640 1 -- 172.21.15.117:0/2125650409 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a24103a60 msgr2=0x7f0a24199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:12.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.147+0000 7f0a22800640 1 --2- 172.21.15.117:0/2125650409 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a24103a60 0x7f0a24199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:12.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.147+0000 7f0a22800640 1 -- 172.21.15.117:0/2125650409 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a24104c50 msgr2=0x7f0a2419a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:12.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.147+0000 7f0a22800640 1 --2- 172.21.15.117:0/2125650409 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a24104c50 0x7f0a2419a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:12.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.147+0000 7f0a22800640 1 -- 172.21.15.117:0/2125650409 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a241a1940 con 0x7f0a24105610 2024-03-19T03:42:12.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.147+0000 7f0a21400640 1 --2- 172.21.15.117:0/2125650409 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a24104c50 0x7f0a2419a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:42:12.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.147+0000 7f0a21e00640 1 --2- 172.21.15.117:0/2125650409 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a24103a60 0x7f0a24199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:42:12.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.147+0000 7f0a22800640 1 --2- 172.21.15.117:0/2125650409 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a24105610 0x7f0a241a12a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c037040 tx=0x7f0a1c035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:12.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.147+0000 7f0a1aa00640 1 -- 172.21.15.117:0/2125650409 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1c039040 con 0x7f0a24105610 2024-03-19T03:42:12.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.147+0000 7f0a1aa00640 1 -- 172.21.15.117:0/2125650409 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a1c044070 con 0x7f0a24105610 2024-03-19T03:42:12.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.147+0000 7f0a29000640 1 -- 172.21.15.117:0/2125650409 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a241a1bd0 con 0x7f0a24105610 2024-03-19T03:42:12.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.147+0000 7f0a1aa00640 1 -- 172.21.15.117:0/2125650409 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1c004450 con 0x7f0a24105610 2024-03-19T03:42:12.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.148+0000 7f0a29000640 1 -- 172.21.15.117:0/2125650409 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a241a20e0 con 0x7f0a24105610 2024-03-19T03:42:12.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.149+0000 7f0a29000640 1 -- 172.21.15.117:0/2125650409 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a2410ad80 con 0x7f0a24105610 2024-03-19T03:42:12.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.150+0000 7f0a1aa00640 1 -- 172.21.15.117:0/2125650409 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 20) ==== 137291+0+0 (secure 0 0 0) 0x7f0a1c0045f0 con 0x7f0a24105610 2024-03-19T03:42:12.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.151+0000 7f0a1aa00640 1 --2- 172.21.15.117:0/2125650409 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0a080a2880 0x7f0a080a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:12.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.152+0000 7f0a1aa00640 1 -- 172.21.15.117:0/2125650409 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f0a1c0f13e0 con 0x7f0a24105610 2024-03-19T03:42:12.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.152+0000 7f0a21e00640 1 --2- 172.21.15.117:0/2125650409 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0a080a2880 0x7f0a080a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:12.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.152+0000 7f0a21e00640 1 --2- 172.21.15.117:0/2125650409 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0a080a2880 0x7f0a080a4d40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0a2419b200 tx=0x7f0a14005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:12.156 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.156+0000 7f0a1aa00640 1 -- 172.21.15.117:0/2125650409 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0a1c0be130 con 0x7f0a24105610 2024-03-19T03:42:12.291 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:12 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/597682855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:42:12.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:12 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2896438590' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:42:12.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:12 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/597682855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:42:12.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:12 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2896438590' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:42:12.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:12 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/597682855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:42:12.484 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:12 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2896438590' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:42:12.486 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.485+0000 7f0a29000640 1 -- 172.21.15.117:0/2125650409 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f0a2402d1f0 con 0x7f0a24105610 2024-03-19T03:42:12.498 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.497+0000 7f0a1aa00640 1 -- 172.21.15.117:0/2125650409 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v20) ==== 81+0+739838 (secure 0 0 0) 0x7f0a1c0098b0 con 0x7f0a24105610 2024-03-19T03:42:12.510 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:42:12.510 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-19T03:42:12.510 INFO:teuthology.orchestra.run.smithi117.stdout: "epoch": 20, 2024-03-19T03:42:12.510 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.510 INFO:teuthology.orchestra.run.smithi117.stdout: "active_gid": 14152, 2024-03-19T03:42:12.510 INFO:teuthology.orchestra.run.smithi117.stdout: "active_name": "x", 2024-03-19T03:42:12.511 INFO:teuthology.orchestra.run.smithi117.stdout: "active_addrs": { 2024-03-19T03:42:12.511 INFO:teuthology.orchestra.run.smithi117.stdout: "addrvec": [ 2024-03-19T03:42:12.511 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.511 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "v2", 2024-03-19T03:42:12.511 INFO:teuthology.orchestra.run.smithi117.stdout: "addr": "172.21.15.117:6800", 2024-03-19T03:42:12.511 INFO:teuthology.orchestra.run.smithi117.stdout: "nonce": 3144614841 2024-03-19T03:42:12.511 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.511 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.511 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "v1", 2024-03-19T03:42:12.511 INFO:teuthology.orchestra.run.smithi117.stdout: "addr": "172.21.15.117:6801", 2024-03-19T03:42:12.511 INFO:teuthology.orchestra.run.smithi117.stdout: "nonce": 3144614841 2024-03-19T03:42:12.511 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.511 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-03-19T03:42:12.512 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.512 INFO:teuthology.orchestra.run.smithi117.stdout: "active_addr": "172.21.15.117:6801/3144614841", 2024-03-19T03:42:12.512 INFO:teuthology.orchestra.run.smithi117.stdout: "active_change": "2024-03-19T03:22:38.134762+0000", 2024-03-19T03:42:12.512 INFO:teuthology.orchestra.run.smithi117.stdout: "active_mgr_features": 4540701547738038271, 2024-03-19T03:42:12.512 INFO:teuthology.orchestra.run.smithi117.stdout: "available": true, 2024-03-19T03:42:12.512 INFO:teuthology.orchestra.run.smithi117.stdout: "standbys": [ 2024-03-19T03:42:12.512 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.512 INFO:teuthology.orchestra.run.smithi117.stdout: "gid": 24109, 2024-03-19T03:42:12.512 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "z", 2024-03-19T03:42:12.512 INFO:teuthology.orchestra.run.smithi117.stdout: "mgr_features": 4540701547738038271, 2024-03-19T03:42:12.512 INFO:teuthology.orchestra.run.smithi117.stdout: "available_modules": [ 2024-03-19T03:42:12.512 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.512 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "alerts", 2024-03-19T03:42:12.512 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.513 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.513 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.513 INFO:teuthology.orchestra.run.smithi117.stdout: "interval": { 2024-03-19T03:42:12.513 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "interval", 2024-03-19T03:42:12.513 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.513 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.513 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.513 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "60", 2024-03-19T03:42:12.513 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.513 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.513 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.513 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "How frequently to reexamine health status", 2024-03-19T03:42:12.513 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.513 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.513 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.514 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.514 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.514 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.514 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.514 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.514 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.514 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.514 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.514 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.514 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.514 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.514 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.514 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.515 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.515 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.515 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.515 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.515 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.515 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.515 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.515 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.515 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.515 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.515 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.515 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.515 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.515 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.516 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.516 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.516 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.516 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.516 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.516 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.516 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.516 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.516 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.516 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.516 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.516 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.516 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.516 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.517 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.517 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.517 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.517 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.517 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.517 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.517 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.517 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.517 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.517 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.517 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.517 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.517 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.517 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.517 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.517 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.518 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.518 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.518 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.518 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.518 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.518 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.518 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.518 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.518 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.518 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.518 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.518 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.518 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.518 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.518 INFO:teuthology.orchestra.run.smithi117.stdout: "smtp_destination": { 2024-03-19T03:42:12.519 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "smtp_destination", 2024-03-19T03:42:12.519 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.519 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.519 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.519 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.519 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.519 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.519 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.519 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Email address to send alerts to", 2024-03-19T03:42:12.519 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.519 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.519 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.519 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.519 INFO:teuthology.orchestra.run.smithi117.stdout: "smtp_from_name": { 2024-03-19T03:42:12.519 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "smtp_from_name", 2024-03-19T03:42:12.520 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.520 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.520 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.520 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "Ceph", 2024-03-19T03:42:12.520 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.520 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.520 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.520 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Email From: name", 2024-03-19T03:42:12.520 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.520 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.520 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.520 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.520 INFO:teuthology.orchestra.run.smithi117.stdout: "smtp_host": { 2024-03-19T03:42:12.520 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "smtp_host", 2024-03-19T03:42:12.520 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.520 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.521 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.521 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.521 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.521 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.521 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.521 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "SMTP server", 2024-03-19T03:42:12.521 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.521 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.521 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.521 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.521 INFO:teuthology.orchestra.run.smithi117.stdout: "smtp_password": { 2024-03-19T03:42:12.521 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "smtp_password", 2024-03-19T03:42:12.521 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.521 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.522 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.522 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.522 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.522 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.522 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.522 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Password to authenticate with", 2024-03-19T03:42:12.522 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.522 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.522 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.522 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.522 INFO:teuthology.orchestra.run.smithi117.stdout: "smtp_port": { 2024-03-19T03:42:12.522 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "smtp_port", 2024-03-19T03:42:12.522 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.522 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.522 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.523 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "465", 2024-03-19T03:42:12.523 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.523 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.523 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.523 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "SMTP port", 2024-03-19T03:42:12.523 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.523 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.523 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.523 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.523 INFO:teuthology.orchestra.run.smithi117.stdout: "smtp_sender": { 2024-03-19T03:42:12.523 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "smtp_sender", 2024-03-19T03:42:12.523 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.523 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.523 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.523 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.524 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.524 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.524 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.524 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "SMTP envelope sender", 2024-03-19T03:42:12.524 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.524 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.524 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.524 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.524 INFO:teuthology.orchestra.run.smithi117.stdout: "smtp_ssl": { 2024-03-19T03:42:12.524 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "smtp_ssl", 2024-03-19T03:42:12.524 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.524 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.524 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.524 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.525 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.525 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.525 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.525 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-19T03:42:12.525 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.525 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.525 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.525 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.525 INFO:teuthology.orchestra.run.smithi117.stdout: "smtp_user": { 2024-03-19T03:42:12.525 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "smtp_user", 2024-03-19T03:42:12.525 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.525 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.525 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.525 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.526 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.526 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.526 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.526 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "User to authenticate as", 2024-03-19T03:42:12.526 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.526 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.526 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.526 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.526 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.526 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.526 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.526 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "balancer", 2024-03-19T03:42:12.526 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.527 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.527 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.527 INFO:teuthology.orchestra.run.smithi117.stdout: "active": { 2024-03-19T03:42:12.527 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "active", 2024-03-19T03:42:12.527 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.527 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.527 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.527 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.527 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.527 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.527 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.527 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "automatically balance PGs across cluster", 2024-03-19T03:42:12.527 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.527 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.527 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.527 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.528 INFO:teuthology.orchestra.run.smithi117.stdout: "begin_time": { 2024-03-19T03:42:12.528 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "begin_time", 2024-03-19T03:42:12.528 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.528 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.528 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.528 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "0000", 2024-03-19T03:42:12.528 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.528 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.528 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.528 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "beginning time of day to automatically balance", 2024-03-19T03:42:12.528 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-19T03:42:12.528 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.528 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.528 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.528 INFO:teuthology.orchestra.run.smithi117.stdout: "begin_weekday": { 2024-03-19T03:42:12.528 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "begin_weekday", 2024-03-19T03:42:12.529 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "uint", 2024-03-19T03:42:12.529 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.529 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.529 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "0", 2024-03-19T03:42:12.529 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "0", 2024-03-19T03:42:12.529 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "6", 2024-03-19T03:42:12.529 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.529 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-19T03:42:12.529 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-19T03:42:12.529 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.529 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.529 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.529 INFO:teuthology.orchestra.run.smithi117.stdout: "crush_compat_max_iterations": { 2024-03-19T03:42:12.529 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "crush_compat_max_iterations", 2024-03-19T03:42:12.530 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "uint", 2024-03-19T03:42:12.530 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.530 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.530 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "25", 2024-03-19T03:42:12.530 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "1", 2024-03-19T03:42:12.530 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "250", 2024-03-19T03:42:12.530 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.530 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-19T03:42:12.530 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.530 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.530 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.530 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.530 INFO:teuthology.orchestra.run.smithi117.stdout: "crush_compat_metrics": { 2024-03-19T03:42:12.531 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "crush_compat_metrics", 2024-03-19T03:42:12.531 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.531 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.531 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.531 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "pgs,objects,bytes", 2024-03-19T03:42:12.531 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.531 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.531 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.531 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-19T03:42:12.531 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-19T03:42:12.531 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.531 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.531 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.531 INFO:teuthology.orchestra.run.smithi117.stdout: "crush_compat_step": { 2024-03-19T03:42:12.531 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "crush_compat_step", 2024-03-19T03:42:12.532 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "float", 2024-03-19T03:42:12.532 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.532 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.532 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "0.5", 2024-03-19T03:42:12.532 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "0.001", 2024-03-19T03:42:12.532 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "0.999", 2024-03-19T03:42:12.532 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.532 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "aggressiveness of optimization", 2024-03-19T03:42:12.532 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-19T03:42:12.532 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.532 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.532 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.532 INFO:teuthology.orchestra.run.smithi117.stdout: "end_time": { 2024-03-19T03:42:12.532 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "end_time", 2024-03-19T03:42:12.533 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.533 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.533 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.533 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "2359", 2024-03-19T03:42:12.533 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.533 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.533 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.533 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "ending time of day to automatically balance", 2024-03-19T03:42:12.533 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-19T03:42:12.533 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.533 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.533 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.533 INFO:teuthology.orchestra.run.smithi117.stdout: "end_weekday": { 2024-03-19T03:42:12.533 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "end_weekday", 2024-03-19T03:42:12.533 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "uint", 2024-03-19T03:42:12.534 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.534 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.534 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "0", 2024-03-19T03:42:12.534 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "0", 2024-03-19T03:42:12.534 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "6", 2024-03-19T03:42:12.534 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.534 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-19T03:42:12.534 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-19T03:42:12.534 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.534 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.534 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.534 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.534 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.534 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.535 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.535 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.535 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.535 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.535 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.535 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.535 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.535 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.535 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.535 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.535 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.535 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.535 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.535 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.535 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.536 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.536 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.536 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.536 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.536 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.536 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.536 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.536 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.536 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.536 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.536 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.536 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.536 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.536 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.537 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.537 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.537 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.537 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.537 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.537 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.537 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.537 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.537 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.537 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.537 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.537 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.537 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.537 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.537 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.538 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.538 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.538 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.538 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.538 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.538 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.538 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.538 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.538 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.538 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.538 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.538 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.538 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.538 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.538 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.539 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.539 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.539 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.539 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.539 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.539 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.539 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.539 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.539 INFO:teuthology.orchestra.run.smithi117.stdout: "min_score": { 2024-03-19T03:42:12.539 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "min_score", 2024-03-19T03:42:12.539 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "float", 2024-03-19T03:42:12.539 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.539 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.539 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "0", 2024-03-19T03:42:12.539 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.540 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.540 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.540 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-19T03:42:12.540 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.540 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.540 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.540 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.540 INFO:teuthology.orchestra.run.smithi117.stdout: "mode": { 2024-03-19T03:42:12.540 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "mode", 2024-03-19T03:42:12.540 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.540 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.540 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.540 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "upmap", 2024-03-19T03:42:12.540 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.540 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.541 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.541 INFO:teuthology.orchestra.run.smithi117.stdout: "crush-compat", 2024-03-19T03:42:12.541 INFO:teuthology.orchestra.run.smithi117.stdout: "none", 2024-03-19T03:42:12.541 INFO:teuthology.orchestra.run.smithi117.stdout: "read", 2024-03-19T03:42:12.541 INFO:teuthology.orchestra.run.smithi117.stdout: "upmap", 2024-03-19T03:42:12.541 INFO:teuthology.orchestra.run.smithi117.stdout: "upmap-read" 2024-03-19T03:42:12.541 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.541 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Balancer mode", 2024-03-19T03:42:12.541 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.541 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.541 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.541 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.541 INFO:teuthology.orchestra.run.smithi117.stdout: "pool_ids": { 2024-03-19T03:42:12.541 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "pool_ids", 2024-03-19T03:42:12.541 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.541 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.541 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.542 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.542 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.542 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.542 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.542 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-19T03:42:12.542 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.542 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.542 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.542 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.542 INFO:teuthology.orchestra.run.smithi117.stdout: "sleep_interval": { 2024-03-19T03:42:12.542 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "sleep_interval", 2024-03-19T03:42:12.542 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.542 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.542 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.543 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "60", 2024-03-19T03:42:12.543 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.543 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.543 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.543 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-19T03:42:12.543 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.543 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.543 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.543 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.543 INFO:teuthology.orchestra.run.smithi117.stdout: "upmap_max_deviation": { 2024-03-19T03:42:12.543 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "upmap_max_deviation", 2024-03-19T03:42:12.543 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.543 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.543 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.543 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "5", 2024-03-19T03:42:12.544 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "1", 2024-03-19T03:42:12.544 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.544 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.544 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-19T03:42:12.544 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-19T03:42:12.544 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.544 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.544 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.544 INFO:teuthology.orchestra.run.smithi117.stdout: "upmap_max_optimizations": { 2024-03-19T03:42:12.544 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "upmap_max_optimizations", 2024-03-19T03:42:12.544 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "uint", 2024-03-19T03:42:12.544 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.544 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.544 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10", 2024-03-19T03:42:12.544 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.545 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.545 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.545 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-19T03:42:12.545 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.545 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.545 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.545 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.545 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.545 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.545 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.545 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "cephadm", 2024-03-19T03:42:12.545 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.545 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.545 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.545 INFO:teuthology.orchestra.run.smithi117.stdout: "agent_down_multiplier": { 2024-03-19T03:42:12.546 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "agent_down_multiplier", 2024-03-19T03:42:12.546 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "float", 2024-03-19T03:42:12.546 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.546 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.546 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "3.0", 2024-03-19T03:42:12.546 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.546 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.546 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.546 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-19T03:42:12.546 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.546 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.546 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.546 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.546 INFO:teuthology.orchestra.run.smithi117.stdout: "agent_refresh_rate": { 2024-03-19T03:42:12.546 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "agent_refresh_rate", 2024-03-19T03:42:12.547 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.547 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.547 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.547 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "20", 2024-03-19T03:42:12.547 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.547 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.547 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.547 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-19T03:42:12.547 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.547 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.547 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.547 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.547 INFO:teuthology.orchestra.run.smithi117.stdout: "agent_starting_port": { 2024-03-19T03:42:12.548 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "agent_starting_port", 2024-03-19T03:42:12.548 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.548 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.548 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.548 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "4721", 2024-03-19T03:42:12.548 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.548 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.548 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.548 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-19T03:42:12.548 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.548 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.548 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.548 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.548 INFO:teuthology.orchestra.run.smithi117.stdout: "allow_ptrace": { 2024-03-19T03:42:12.548 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "allow_ptrace", 2024-03-19T03:42:12.549 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.549 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.549 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.549 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.549 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.549 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.549 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.549 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-19T03:42:12.549 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-19T03:42:12.549 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.549 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.549 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.549 INFO:teuthology.orchestra.run.smithi117.stdout: "autotune_interval": { 2024-03-19T03:42:12.549 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "autotune_interval", 2024-03-19T03:42:12.550 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.550 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.550 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.550 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "600", 2024-03-19T03:42:12.550 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.550 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.550 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.550 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-19T03:42:12.550 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.550 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.550 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.550 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.550 INFO:teuthology.orchestra.run.smithi117.stdout: "autotune_memory_target_ratio": { 2024-03-19T03:42:12.551 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "autotune_memory_target_ratio", 2024-03-19T03:42:12.551 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "float", 2024-03-19T03:42:12.551 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.551 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.551 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "0.7", 2024-03-19T03:42:12.551 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.551 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.551 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.551 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-19T03:42:12.551 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.551 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.551 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.551 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.551 INFO:teuthology.orchestra.run.smithi117.stdout: "cephadm_log_destination": { 2024-03-19T03:42:12.551 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "cephadm_log_destination", 2024-03-19T03:42:12.551 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.552 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.552 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.552 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.552 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.552 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.552 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.552 INFO:teuthology.orchestra.run.smithi117.stdout: "file", 2024-03-19T03:42:12.552 INFO:teuthology.orchestra.run.smithi117.stdout: "file,syslog", 2024-03-19T03:42:12.552 INFO:teuthology.orchestra.run.smithi117.stdout: "syslog" 2024-03-19T03:42:12.552 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.552 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-19T03:42:12.552 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.552 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.553 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.553 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.553 INFO:teuthology.orchestra.run.smithi117.stdout: "cgroups_split": { 2024-03-19T03:42:12.553 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "cgroups_split", 2024-03-19T03:42:12.553 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.553 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.553 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.553 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.553 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.553 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.553 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.553 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-19T03:42:12.553 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.553 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.553 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.553 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.554 INFO:teuthology.orchestra.run.smithi117.stdout: "config_checks_enabled": { 2024-03-19T03:42:12.554 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "config_checks_enabled", 2024-03-19T03:42:12.554 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.554 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.554 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.554 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.554 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.554 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.554 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.554 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-19T03:42:12.554 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.554 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.554 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.554 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.555 INFO:teuthology.orchestra.run.smithi117.stdout: "config_dashboard": { 2024-03-19T03:42:12.555 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "config_dashboard", 2024-03-19T03:42:12.555 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.555 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.555 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.555 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.555 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.555 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.555 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.555 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-19T03:42:12.555 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.555 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.555 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.555 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.555 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_alertmanager": { 2024-03-19T03:42:12.555 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_alertmanager", 2024-03-19T03:42:12.556 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.556 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.556 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.556 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-19T03:42:12.556 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.556 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.556 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.556 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Prometheus container image", 2024-03-19T03:42:12.556 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.556 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.556 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.556 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.556 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_base": { 2024-03-19T03:42:12.556 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_base", 2024-03-19T03:42:12.556 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.557 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.557 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.557 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-19T03:42:12.557 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.557 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.557 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.557 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Container image name, without the tag", 2024-03-19T03:42:12.557 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.557 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.557 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.557 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.557 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_elasticsearch": { 2024-03-19T03:42:12.557 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_elasticsearch", 2024-03-19T03:42:12.558 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.558 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.558 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.558 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-19T03:42:12.558 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.558 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.558 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.558 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "elasticsearch container image", 2024-03-19T03:42:12.558 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.558 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.558 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.558 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.558 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_grafana": { 2024-03-19T03:42:12.558 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_grafana", 2024-03-19T03:42:12.559 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.559 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.559 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.559 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-19T03:42:12.559 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.559 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.559 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.559 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Prometheus container image", 2024-03-19T03:42:12.559 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.559 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.559 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.559 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.559 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_haproxy": { 2024-03-19T03:42:12.559 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_haproxy", 2024-03-19T03:42:12.559 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.560 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.560 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.560 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-19T03:42:12.560 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.560 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.560 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.560 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "HAproxy container image", 2024-03-19T03:42:12.560 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.560 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.560 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.560 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.560 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_jaeger_agent": { 2024-03-19T03:42:12.560 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_jaeger_agent", 2024-03-19T03:42:12.560 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.560 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.561 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.561 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-19T03:42:12.561 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.561 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.561 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.561 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Jaeger agent container image", 2024-03-19T03:42:12.561 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.561 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.561 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.561 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.561 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_jaeger_collector": { 2024-03-19T03:42:12.561 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_jaeger_collector", 2024-03-19T03:42:12.561 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.561 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.561 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.562 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-19T03:42:12.562 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.562 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.562 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.562 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Jaeger collector container image", 2024-03-19T03:42:12.562 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.562 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.562 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.562 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.562 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_jaeger_query": { 2024-03-19T03:42:12.562 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_jaeger_query", 2024-03-19T03:42:12.562 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.562 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.562 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.563 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-19T03:42:12.563 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.563 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.563 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.563 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Jaeger query container image", 2024-03-19T03:42:12.563 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.563 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.563 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.563 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.563 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_keepalived": { 2024-03-19T03:42:12.563 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_keepalived", 2024-03-19T03:42:12.563 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.563 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.563 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.563 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-19T03:42:12.563 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.564 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.564 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.564 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Keepalived container image", 2024-03-19T03:42:12.564 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.564 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.564 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.564 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.564 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_loki": { 2024-03-19T03:42:12.564 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_loki", 2024-03-19T03:42:12.564 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.564 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.564 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.564 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-19T03:42:12.564 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.564 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.564 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.565 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Loki container image", 2024-03-19T03:42:12.565 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.565 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.565 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.565 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.565 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_node_exporter": { 2024-03-19T03:42:12.565 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_node_exporter", 2024-03-19T03:42:12.565 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.565 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.565 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.565 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-19T03:42:12.565 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.565 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.565 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.565 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Prometheus container image", 2024-03-19T03:42:12.565 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.566 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.566 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.566 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.566 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_nvmeof": { 2024-03-19T03:42:12.566 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_nvmeof", 2024-03-19T03:42:12.566 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.566 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.566 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.566 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-19T03:42:12.566 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.566 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.566 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.566 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Nvme-of container image", 2024-03-19T03:42:12.566 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.567 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.567 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.567 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.567 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_prometheus": { 2024-03-19T03:42:12.567 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_prometheus", 2024-03-19T03:42:12.567 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.567 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.567 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.567 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-19T03:42:12.567 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.567 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.567 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.567 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Prometheus container image", 2024-03-19T03:42:12.567 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.568 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.568 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.568 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.568 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_promtail": { 2024-03-19T03:42:12.568 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_promtail", 2024-03-19T03:42:12.568 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.568 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.568 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.568 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-19T03:42:12.568 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.568 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.568 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.568 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Promtail container image", 2024-03-19T03:42:12.568 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.569 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.569 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.569 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.569 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_snmp_gateway": { 2024-03-19T03:42:12.569 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_snmp_gateway", 2024-03-19T03:42:12.569 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.569 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.569 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.569 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-19T03:42:12.569 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.569 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.569 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.569 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "SNMP Gateway container image", 2024-03-19T03:42:12.569 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.570 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.570 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.570 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.570 INFO:teuthology.orchestra.run.smithi117.stdout: "container_init": { 2024-03-19T03:42:12.570 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_init", 2024-03-19T03:42:12.570 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.570 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.570 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.570 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.570 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.570 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.570 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.570 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Run podman/docker with `--init`", 2024-03-19T03:42:12.570 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.570 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.571 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.571 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.571 INFO:teuthology.orchestra.run.smithi117.stdout: "daemon_cache_timeout": { 2024-03-19T03:42:12.571 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "daemon_cache_timeout", 2024-03-19T03:42:12.571 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.571 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.571 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.571 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "600", 2024-03-19T03:42:12.571 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.571 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.571 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.571 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-19T03:42:12.571 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.572 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.572 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.572 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.572 INFO:teuthology.orchestra.run.smithi117.stdout: "default_cephadm_command_timeout": { 2024-03-19T03:42:12.572 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "default_cephadm_command_timeout", 2024-03-19T03:42:12.572 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.572 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.572 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.572 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "900", 2024-03-19T03:42:12.572 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.572 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.572 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.572 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-19T03:42:12.572 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.572 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.572 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.572 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.573 INFO:teuthology.orchestra.run.smithi117.stdout: "default_registry": { 2024-03-19T03:42:12.573 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "default_registry", 2024-03-19T03:42:12.573 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.573 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.573 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.573 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "docker.io", 2024-03-19T03:42:12.573 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.573 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.573 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.573 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-19T03:42:12.573 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.573 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.573 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.573 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.573 INFO:teuthology.orchestra.run.smithi117.stdout: "device_cache_timeout": { 2024-03-19T03:42:12.573 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "device_cache_timeout", 2024-03-19T03:42:12.574 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.574 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.574 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.574 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "1800", 2024-03-19T03:42:12.574 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.574 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.574 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.574 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "seconds to cache device inventory", 2024-03-19T03:42:12.574 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.574 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.574 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.574 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.574 INFO:teuthology.orchestra.run.smithi117.stdout: "device_enhanced_scan": { 2024-03-19T03:42:12.574 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "device_enhanced_scan", 2024-03-19T03:42:12.574 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.575 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.575 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.575 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.575 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.575 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.575 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.575 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-19T03:42:12.575 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.575 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.575 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.575 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.575 INFO:teuthology.orchestra.run.smithi117.stdout: "facts_cache_timeout": { 2024-03-19T03:42:12.575 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "facts_cache_timeout", 2024-03-19T03:42:12.575 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.575 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.575 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.576 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "60", 2024-03-19T03:42:12.576 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.576 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.576 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.576 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "seconds to cache host facts data", 2024-03-19T03:42:12.576 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.576 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.576 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.576 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.576 INFO:teuthology.orchestra.run.smithi117.stdout: "host_check_interval": { 2024-03-19T03:42:12.576 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "host_check_interval", 2024-03-19T03:42:12.576 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.576 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.576 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.576 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "600", 2024-03-19T03:42:12.576 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.577 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.577 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.577 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "how frequently to perform a host check", 2024-03-19T03:42:12.577 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.577 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.577 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.577 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.577 INFO:teuthology.orchestra.run.smithi117.stdout: "hw_monitoring": { 2024-03-19T03:42:12.577 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "hw_monitoring", 2024-03-19T03:42:12.577 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.577 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.577 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.577 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.578 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.578 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.578 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.578 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-19T03:42:12.578 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.578 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.578 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.578 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.578 INFO:teuthology.orchestra.run.smithi117.stdout: "inventory_list_all": { 2024-03-19T03:42:12.578 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "inventory_list_all", 2024-03-19T03:42:12.578 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.578 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.578 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.578 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.578 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.579 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.579 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.579 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-19T03:42:12.579 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.579 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.579 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.579 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.579 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.579 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.579 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.579 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.579 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.579 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.579 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.579 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.579 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.580 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.580 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.580 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.580 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.580 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.580 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.580 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.580 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.580 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.580 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.580 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.580 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.580 INFO:teuthology.orchestra.run.smithi117.stdout: "log_refresh_metadata": { 2024-03-19T03:42:12.580 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_refresh_metadata", 2024-03-19T03:42:12.581 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.581 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.581 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.581 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.581 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.581 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.581 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.581 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-19T03:42:12.581 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.581 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.581 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.581 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.581 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.581 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.581 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.581 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.581 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.582 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.582 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.582 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.582 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.582 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-19T03:42:12.582 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.582 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.582 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.582 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.582 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.582 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.582 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.582 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.582 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.583 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.583 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.583 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.583 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.583 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.583 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.583 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.583 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.583 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.583 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.583 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.583 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.583 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.584 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.584 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.584 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.584 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.584 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.584 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.584 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.584 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.584 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.584 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.584 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.584 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.584 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.584 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.585 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.585 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.585 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.585 INFO:teuthology.orchestra.run.smithi117.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-19T03:42:12.585 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-19T03:42:12.585 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.585 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.585 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.585 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.585 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.585 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.585 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.585 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-19T03:42:12.585 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.585 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.586 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.586 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.586 INFO:teuthology.orchestra.run.smithi117.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-19T03:42:12.586 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-19T03:42:12.586 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.586 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.586 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.586 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "*", 2024-03-19T03:42:12.586 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.586 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.586 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.586 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-19T03:42:12.587 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.587 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.587 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.587 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.587 INFO:teuthology.orchestra.run.smithi117.stdout: "max_count_per_host": { 2024-03-19T03:42:12.587 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "max_count_per_host", 2024-03-19T03:42:12.587 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.587 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.587 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.587 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10", 2024-03-19T03:42:12.587 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.587 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.587 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.588 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "max number of daemons per service per host", 2024-03-19T03:42:12.588 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.588 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.588 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.588 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.588 INFO:teuthology.orchestra.run.smithi117.stdout: "max_osd_draining_count": { 2024-03-19T03:42:12.588 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "max_osd_draining_count", 2024-03-19T03:42:12.588 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.588 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.588 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.588 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10", 2024-03-19T03:42:12.588 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.588 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.588 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.589 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-19T03:42:12.589 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.589 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.589 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.589 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.589 INFO:teuthology.orchestra.run.smithi117.stdout: "migration_current": { 2024-03-19T03:42:12.589 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "migration_current", 2024-03-19T03:42:12.589 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.589 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.589 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.589 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.589 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.589 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.589 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.589 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "internal - do not modify", 2024-03-19T03:42:12.590 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.590 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.590 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.590 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.590 INFO:teuthology.orchestra.run.smithi117.stdout: "mode": { 2024-03-19T03:42:12.590 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "mode", 2024-03-19T03:42:12.590 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.590 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.590 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.590 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "root", 2024-03-19T03:42:12.590 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.590 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.590 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.591 INFO:teuthology.orchestra.run.smithi117.stdout: "cephadm-package", 2024-03-19T03:42:12.591 INFO:teuthology.orchestra.run.smithi117.stdout: "root" 2024-03-19T03:42:12.591 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.591 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "mode for remote execution of cephadm", 2024-03-19T03:42:12.591 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.591 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.591 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.591 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.591 INFO:teuthology.orchestra.run.smithi117.stdout: "oob_default_addr": { 2024-03-19T03:42:12.591 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "oob_default_addr", 2024-03-19T03:42:12.591 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.591 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.591 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.591 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "169.254.1.1", 2024-03-19T03:42:12.591 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.592 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.592 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.592 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-19T03:42:12.592 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.592 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.592 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.592 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.592 INFO:teuthology.orchestra.run.smithi117.stdout: "prometheus_alerts_path": { 2024-03-19T03:42:12.592 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "prometheus_alerts_path", 2024-03-19T03:42:12.592 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.592 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.592 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.592 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-19T03:42:12.593 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.593 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.593 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.593 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-19T03:42:12.593 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.593 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.593 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.593 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.593 INFO:teuthology.orchestra.run.smithi117.stdout: "registry_insecure": { 2024-03-19T03:42:12.593 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "registry_insecure", 2024-03-19T03:42:12.593 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.593 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.593 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.593 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.593 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.593 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.594 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.594 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-19T03:42:12.594 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.594 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.594 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.594 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.594 INFO:teuthology.orchestra.run.smithi117.stdout: "registry_password": { 2024-03-19T03:42:12.594 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "registry_password", 2024-03-19T03:42:12.594 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.594 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.594 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.594 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.594 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.595 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.595 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.595 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-19T03:42:12.595 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.595 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.595 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.595 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.595 INFO:teuthology.orchestra.run.smithi117.stdout: "registry_url": { 2024-03-19T03:42:12.595 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "registry_url", 2024-03-19T03:42:12.595 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.595 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.595 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.595 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.595 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.595 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.596 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.596 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-19T03:42:12.596 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.596 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.596 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.596 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.596 INFO:teuthology.orchestra.run.smithi117.stdout: "registry_username": { 2024-03-19T03:42:12.596 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "registry_username", 2024-03-19T03:42:12.596 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.596 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.596 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.596 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.596 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.596 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.596 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.596 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-19T03:42:12.597 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.597 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.597 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.597 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.597 INFO:teuthology.orchestra.run.smithi117.stdout: "secure_monitoring_stack": { 2024-03-19T03:42:12.597 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "secure_monitoring_stack", 2024-03-19T03:42:12.597 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.597 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.597 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.597 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.597 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.597 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.597 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.597 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-19T03:42:12.597 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.598 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.598 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.598 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.598 INFO:teuthology.orchestra.run.smithi117.stdout: "service_discovery_port": { 2024-03-19T03:42:12.598 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "service_discovery_port", 2024-03-19T03:42:12.598 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.598 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.598 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.598 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "8765", 2024-03-19T03:42:12.598 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.598 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.598 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.598 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "cephadm service discovery port", 2024-03-19T03:42:12.598 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.599 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.599 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.599 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.599 INFO:teuthology.orchestra.run.smithi117.stdout: "ssh_config_file": { 2024-03-19T03:42:12.599 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ssh_config_file", 2024-03-19T03:42:12.599 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.599 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.599 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.599 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.599 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.599 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.599 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.599 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-19T03:42:12.599 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.600 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.600 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.600 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.600 INFO:teuthology.orchestra.run.smithi117.stdout: "use_agent": { 2024-03-19T03:42:12.600 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "use_agent", 2024-03-19T03:42:12.600 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.600 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.600 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.600 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.600 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.600 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.600 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.600 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-19T03:42:12.600 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.600 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.601 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.601 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.601 INFO:teuthology.orchestra.run.smithi117.stdout: "use_repo_digest": { 2024-03-19T03:42:12.601 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "use_repo_digest", 2024-03-19T03:42:12.601 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.601 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.601 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.601 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.601 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.601 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.601 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.601 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-19T03:42:12.601 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.601 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.601 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.602 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.602 INFO:teuthology.orchestra.run.smithi117.stdout: "warn_on_failed_host_check": { 2024-03-19T03:42:12.602 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "warn_on_failed_host_check", 2024-03-19T03:42:12.602 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.602 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.602 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.602 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.602 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.602 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.602 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.602 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "raise a health warning if the host check fails", 2024-03-19T03:42:12.602 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.602 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.602 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.602 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.603 INFO:teuthology.orchestra.run.smithi117.stdout: "warn_on_stray_daemons": { 2024-03-19T03:42:12.603 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "warn_on_stray_daemons", 2024-03-19T03:42:12.603 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.603 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.603 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.603 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.603 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.603 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.603 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.603 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-19T03:42:12.603 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.603 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.603 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.603 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.604 INFO:teuthology.orchestra.run.smithi117.stdout: "warn_on_stray_hosts": { 2024-03-19T03:42:12.604 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "warn_on_stray_hosts", 2024-03-19T03:42:12.604 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.604 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.604 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.604 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.604 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.604 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.604 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.604 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-19T03:42:12.604 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.604 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.604 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.604 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.605 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.605 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.605 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.605 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "crash", 2024-03-19T03:42:12.605 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.605 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.605 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.605 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.605 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.605 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.605 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.605 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.605 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.605 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.605 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.606 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.606 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.606 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.606 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.606 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.606 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.606 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.606 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.606 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.606 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.606 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.606 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.606 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.606 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.606 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.607 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.607 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.607 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.607 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.607 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.607 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.607 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.607 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.607 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.607 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.607 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.607 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.609 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.609 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.609 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.609 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.609 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.609 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.609 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.609 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.609 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.609 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.609 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.609 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.609 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.610 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.610 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.610 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.610 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.610 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.610 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.610 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.610 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.610 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.610 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.610 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.610 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.610 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.610 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.611 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.611 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.611 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.611 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.611 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.611 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.611 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.611 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.611 INFO:teuthology.orchestra.run.smithi117.stdout: "retain_interval": { 2024-03-19T03:42:12.611 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "retain_interval", 2024-03-19T03:42:12.611 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.611 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.611 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.611 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "31536000", 2024-03-19T03:42:12.612 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.612 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.612 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.612 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-19T03:42:12.612 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.612 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.612 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.612 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.612 INFO:teuthology.orchestra.run.smithi117.stdout: "warn_recent_interval": { 2024-03-19T03:42:12.612 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "warn_recent_interval", 2024-03-19T03:42:12.612 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.612 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.613 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.613 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "1209600", 2024-03-19T03:42:12.613 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.613 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.613 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.613 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-19T03:42:12.613 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.613 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.613 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.613 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.613 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.613 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.613 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.613 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "dashboard", 2024-03-19T03:42:12.613 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.614 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.614 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.614 INFO:teuthology.orchestra.run.smithi117.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-19T03:42:12.614 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-19T03:42:12.614 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.614 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.614 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.614 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10", 2024-03-19T03:42:12.614 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.614 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.614 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.614 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.614 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.614 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.615 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.615 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.615 INFO:teuthology.orchestra.run.smithi117.stdout: "ALERTMANAGER_API_HOST": { 2024-03-19T03:42:12.615 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-19T03:42:12.615 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.615 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.615 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.615 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.615 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.615 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.615 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.615 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.615 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.616 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.616 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.616 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.616 INFO:teuthology.orchestra.run.smithi117.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-19T03:42:12.616 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-19T03:42:12.616 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.616 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.616 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.616 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.616 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.616 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.616 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.616 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.616 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.616 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.617 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.617 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.617 INFO:teuthology.orchestra.run.smithi117.stdout: "AUDIT_API_ENABLED": { 2024-03-19T03:42:12.617 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "AUDIT_API_ENABLED", 2024-03-19T03:42:12.617 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.617 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.617 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.617 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.617 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.617 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.617 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.617 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.617 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.617 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.617 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.618 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.618 INFO:teuthology.orchestra.run.smithi117.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-19T03:42:12.618 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-19T03:42:12.618 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.618 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.618 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.618 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.618 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.618 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.618 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.618 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.618 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.618 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.618 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.618 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.619 INFO:teuthology.orchestra.run.smithi117.stdout: "ENABLE_BROWSABLE_API": { 2024-03-19T03:42:12.619 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-19T03:42:12.619 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.619 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.619 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.619 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.619 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.619 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.619 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.619 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.619 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.619 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.619 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.619 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.619 INFO:teuthology.orchestra.run.smithi117.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-19T03:42:12.619 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-19T03:42:12.620 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.620 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.620 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.620 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.620 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.620 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.620 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.620 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.620 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.620 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.620 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.620 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.620 INFO:teuthology.orchestra.run.smithi117.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-19T03:42:12.620 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-19T03:42:12.620 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.620 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.621 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.621 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.621 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.621 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.621 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.621 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.621 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.621 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.621 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.621 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.621 INFO:teuthology.orchestra.run.smithi117.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-19T03:42:12.621 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-19T03:42:12.621 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.621 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.622 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.622 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.622 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.622 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.622 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.622 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.622 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.622 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.622 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.622 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.622 INFO:teuthology.orchestra.run.smithi117.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-19T03:42:12.622 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-19T03:42:12.622 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.622 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.623 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.623 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.623 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.623 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.623 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.623 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.623 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.623 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.623 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.623 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.623 INFO:teuthology.orchestra.run.smithi117.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-19T03:42:12.623 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-19T03:42:12.623 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.623 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.623 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.623 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.623 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.624 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.624 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.624 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.624 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.624 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.624 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.624 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.624 INFO:teuthology.orchestra.run.smithi117.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-19T03:42:12.624 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-19T03:42:12.624 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.624 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.624 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.624 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.624 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.624 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.625 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.625 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.625 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.625 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.625 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.625 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.625 INFO:teuthology.orchestra.run.smithi117.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-19T03:42:12.625 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-19T03:42:12.625 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.625 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.625 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.625 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.625 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.625 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.625 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.625 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.625 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.626 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.626 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.626 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.626 INFO:teuthology.orchestra.run.smithi117.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-19T03:42:12.626 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-19T03:42:12.626 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.626 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.626 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.626 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.626 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.626 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.626 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.626 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.626 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.627 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.627 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.627 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.627 INFO:teuthology.orchestra.run.smithi117.stdout: "GRAFANA_API_PASSWORD": { 2024-03-19T03:42:12.627 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-19T03:42:12.627 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.627 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.627 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.627 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "admin", 2024-03-19T03:42:12.627 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.627 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.627 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.627 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.627 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.627 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.628 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.628 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.628 INFO:teuthology.orchestra.run.smithi117.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-19T03:42:12.628 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-19T03:42:12.628 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.628 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.628 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.628 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.628 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.628 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.628 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.628 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.628 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.628 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.629 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.629 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.629 INFO:teuthology.orchestra.run.smithi117.stdout: "GRAFANA_API_URL": { 2024-03-19T03:42:12.629 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "GRAFANA_API_URL", 2024-03-19T03:42:12.629 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.629 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.629 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.629 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.629 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.629 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.629 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.629 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.629 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.629 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.630 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.630 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.630 INFO:teuthology.orchestra.run.smithi117.stdout: "GRAFANA_API_USERNAME": { 2024-03-19T03:42:12.630 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-19T03:42:12.630 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.630 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.630 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.630 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "admin", 2024-03-19T03:42:12.630 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.630 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.630 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.630 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.630 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.631 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.631 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.631 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.631 INFO:teuthology.orchestra.run.smithi117.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-19T03:42:12.631 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-19T03:42:12.631 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.631 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.631 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.631 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.631 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.631 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.631 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.631 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.631 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.631 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.631 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.631 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.632 INFO:teuthology.orchestra.run.smithi117.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-19T03:42:12.632 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-19T03:42:12.632 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.632 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.632 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.632 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.632 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.632 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.632 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.632 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.632 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.632 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.632 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.632 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.632 INFO:teuthology.orchestra.run.smithi117.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-19T03:42:12.632 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-19T03:42:12.633 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.633 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.633 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.633 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.633 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.633 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.633 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.633 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.633 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.633 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.633 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.633 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.633 INFO:teuthology.orchestra.run.smithi117.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-19T03:42:12.633 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-19T03:42:12.634 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.634 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.634 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.634 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.634 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.634 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.634 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.634 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.634 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.634 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.634 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.634 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.634 INFO:teuthology.orchestra.run.smithi117.stdout: "MULTICLUSTER_CONFIG": { 2024-03-19T03:42:12.634 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-19T03:42:12.635 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.635 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.635 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.635 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "{}", 2024-03-19T03:42:12.635 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.635 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.635 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.635 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.635 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.635 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.635 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.635 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.635 INFO:teuthology.orchestra.run.smithi117.stdout: "PROMETHEUS_API_HOST": { 2024-03-19T03:42:12.635 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-19T03:42:12.635 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.636 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.636 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.636 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.636 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.636 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.636 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.636 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.636 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.636 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.636 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.636 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.636 INFO:teuthology.orchestra.run.smithi117.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-19T03:42:12.636 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-19T03:42:12.636 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.637 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.637 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.637 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.637 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.637 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.637 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.637 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.637 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.637 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.637 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.637 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.637 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-19T03:42:12.637 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-19T03:42:12.637 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.638 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.638 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.638 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.638 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.638 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.638 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.638 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.638 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.638 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.638 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.638 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.638 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-19T03:42:12.638 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-19T03:42:12.639 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.639 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.639 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.639 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.639 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.639 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.639 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.639 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.639 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.639 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.639 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.639 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.639 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-19T03:42:12.639 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-19T03:42:12.640 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.640 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.640 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.640 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.640 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.640 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.640 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.640 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.640 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.640 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.640 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.640 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.640 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-19T03:42:12.641 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-19T03:42:12.641 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.641 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.641 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.641 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.641 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.641 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.641 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.641 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.641 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.641 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.641 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.641 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.642 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-19T03:42:12.642 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-19T03:42:12.642 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.642 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.642 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.642 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.642 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.642 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.642 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.642 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.642 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.642 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.642 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.642 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.643 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-19T03:42:12.643 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-19T03:42:12.643 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.643 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.643 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.643 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.643 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.643 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.643 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.643 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.643 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.643 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.643 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.643 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.643 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-19T03:42:12.644 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-19T03:42:12.644 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.644 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.644 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.644 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.644 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.644 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.644 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.644 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.644 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.644 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.644 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.644 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.645 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_ENABLED": { 2024-03-19T03:42:12.645 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-19T03:42:12.645 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.645 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.645 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.645 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.645 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.645 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.645 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.645 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.645 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.645 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.645 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.645 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.646 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-19T03:42:12.646 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-19T03:42:12.646 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.646 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.646 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.646 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-19T03:42:12.646 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.646 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.646 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.646 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.646 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.646 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.646 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.647 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.647 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-19T03:42:12.647 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-19T03:42:12.647 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.647 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.647 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.647 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10", 2024-03-19T03:42:12.647 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.647 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.647 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.647 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.647 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.648 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.648 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.648 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.648 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-19T03:42:12.648 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-19T03:42:12.648 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.648 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.648 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.648 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "8", 2024-03-19T03:42:12.648 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.648 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.648 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.648 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.648 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.648 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.649 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.649 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.649 INFO:teuthology.orchestra.run.smithi117.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-19T03:42:12.649 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-19T03:42:12.649 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.649 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.649 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.649 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "45", 2024-03-19T03:42:12.649 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.649 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.649 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.649 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.649 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.649 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.649 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.650 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.650 INFO:teuthology.orchestra.run.smithi117.stdout: "RGW_API_ACCESS_KEY": { 2024-03-19T03:42:12.650 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-19T03:42:12.650 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.650 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.650 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.650 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.650 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.650 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.650 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.650 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.650 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.650 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.651 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.651 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.651 INFO:teuthology.orchestra.run.smithi117.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-19T03:42:12.651 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-19T03:42:12.651 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.651 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.651 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.651 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "admin", 2024-03-19T03:42:12.651 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.651 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.651 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.651 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.651 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.652 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.652 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.652 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.652 INFO:teuthology.orchestra.run.smithi117.stdout: "RGW_API_SECRET_KEY": { 2024-03-19T03:42:12.652 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-19T03:42:12.652 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.652 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.652 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.652 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.652 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.652 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.652 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.652 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.652 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.653 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.653 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.653 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.653 INFO:teuthology.orchestra.run.smithi117.stdout: "RGW_API_SSL_VERIFY": { 2024-03-19T03:42:12.653 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-19T03:42:12.653 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.653 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.653 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.653 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.653 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.653 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.653 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.653 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.653 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.654 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.654 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.654 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.654 INFO:teuthology.orchestra.run.smithi117.stdout: "UNSAFE_TLS_v1_2": { 2024-03-19T03:42:12.654 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-19T03:42:12.654 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.654 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.654 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.654 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.654 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.654 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.654 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.654 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.654 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.655 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.655 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.655 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.655 INFO:teuthology.orchestra.run.smithi117.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-19T03:42:12.655 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-19T03:42:12.655 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.655 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.655 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.655 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "0", 2024-03-19T03:42:12.655 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.655 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.655 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.655 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.655 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.656 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.656 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.656 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.656 INFO:teuthology.orchestra.run.smithi117.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-19T03:42:12.656 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-19T03:42:12.656 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.656 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.656 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.656 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10", 2024-03-19T03:42:12.656 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.656 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.656 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.656 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.656 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.657 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.657 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.657 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.657 INFO:teuthology.orchestra.run.smithi117.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-19T03:42:12.657 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-19T03:42:12.657 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.657 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.657 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.657 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "5", 2024-03-19T03:42:12.657 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.657 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.657 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.657 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.657 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.658 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.658 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.658 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.658 INFO:teuthology.orchestra.run.smithi117.stdout: "cross_origin_url": { 2024-03-19T03:42:12.658 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "cross_origin_url", 2024-03-19T03:42:12.658 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.658 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.658 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.658 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.658 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.658 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.658 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.658 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.658 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.659 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.659 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.659 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.659 INFO:teuthology.orchestra.run.smithi117.stdout: "crt_file": { 2024-03-19T03:42:12.659 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "crt_file", 2024-03-19T03:42:12.659 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.659 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.659 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.659 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.659 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.659 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.659 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.659 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.659 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.660 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.660 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.660 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.660 INFO:teuthology.orchestra.run.smithi117.stdout: "debug": { 2024-03-19T03:42:12.660 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "debug", 2024-03-19T03:42:12.660 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.660 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.660 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.660 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.660 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.660 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.660 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.660 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Enable/disable debug options", 2024-03-19T03:42:12.661 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.661 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.661 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.661 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.661 INFO:teuthology.orchestra.run.smithi117.stdout: "jwt_token_ttl": { 2024-03-19T03:42:12.661 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "jwt_token_ttl", 2024-03-19T03:42:12.661 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.661 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.661 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.661 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "28800", 2024-03-19T03:42:12.661 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.661 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.661 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.661 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.662 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.662 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.662 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.662 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.662 INFO:teuthology.orchestra.run.smithi117.stdout: "key_file": { 2024-03-19T03:42:12.662 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "key_file", 2024-03-19T03:42:12.662 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.662 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.662 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.662 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.662 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.662 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.662 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.662 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.663 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.663 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.663 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.663 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.663 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.663 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.663 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.663 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.663 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.663 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.663 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.663 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.663 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.663 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.663 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.664 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.664 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.664 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.664 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.664 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.664 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.664 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.664 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.664 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.664 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.664 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.664 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.664 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.664 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.664 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.665 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.665 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.665 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.665 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.665 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.665 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.665 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.665 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.665 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.665 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.665 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.665 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.665 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.665 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.665 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.666 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.666 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.666 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.666 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.666 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.666 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.666 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.666 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.666 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.666 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.666 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.666 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.666 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.666 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.667 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.667 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.667 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.667 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.667 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.667 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.667 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.667 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.667 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.667 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.667 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.667 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.667 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.667 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.667 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.667 INFO:teuthology.orchestra.run.smithi117.stdout: "motd": { 2024-03-19T03:42:12.668 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "motd", 2024-03-19T03:42:12.668 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.668 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.668 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.668 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.668 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.668 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.668 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.668 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "The message of the day", 2024-03-19T03:42:12.668 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.668 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.668 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.668 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.668 INFO:teuthology.orchestra.run.smithi117.stdout: "redirect_resolve_ip_addr": { 2024-03-19T03:42:12.668 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "redirect_resolve_ip_addr", 2024-03-19T03:42:12.669 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.669 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.669 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.669 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.669 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.669 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.669 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.669 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.669 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.669 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.669 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.669 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.669 INFO:teuthology.orchestra.run.smithi117.stdout: "server_addr": { 2024-03-19T03:42:12.669 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "server_addr", 2024-03-19T03:42:12.669 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.670 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.670 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.670 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "::", 2024-03-19T03:42:12.670 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.670 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.670 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.670 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.670 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.670 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.670 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.670 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.670 INFO:teuthology.orchestra.run.smithi117.stdout: "server_port": { 2024-03-19T03:42:12.670 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "server_port", 2024-03-19T03:42:12.670 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.671 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.671 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.671 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "8080", 2024-03-19T03:42:12.671 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.671 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.671 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.671 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.671 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.671 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.671 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.671 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.671 INFO:teuthology.orchestra.run.smithi117.stdout: "ssl": { 2024-03-19T03:42:12.671 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ssl", 2024-03-19T03:42:12.671 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.671 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.672 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.672 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.672 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.672 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.672 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.672 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.672 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.672 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.672 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.672 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.672 INFO:teuthology.orchestra.run.smithi117.stdout: "ssl_server_port": { 2024-03-19T03:42:12.672 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ssl_server_port", 2024-03-19T03:42:12.672 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.672 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.672 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.673 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "8443", 2024-03-19T03:42:12.673 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.673 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.673 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.673 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.673 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.673 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.673 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.673 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.673 INFO:teuthology.orchestra.run.smithi117.stdout: "standby_behaviour": { 2024-03-19T03:42:12.673 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "standby_behaviour", 2024-03-19T03:42:12.673 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.673 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.674 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.674 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "redirect", 2024-03-19T03:42:12.674 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.674 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.674 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.674 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.674 INFO:teuthology.orchestra.run.smithi117.stdout: "redirect" 2024-03-19T03:42:12.674 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.674 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.674 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.674 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.674 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.674 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.674 INFO:teuthology.orchestra.run.smithi117.stdout: "standby_error_status_code": { 2024-03-19T03:42:12.675 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "standby_error_status_code", 2024-03-19T03:42:12.675 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.675 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.675 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.675 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "500", 2024-03-19T03:42:12.675 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "400", 2024-03-19T03:42:12.675 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "599", 2024-03-19T03:42:12.675 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.675 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.675 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.675 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.675 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.675 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.675 INFO:teuthology.orchestra.run.smithi117.stdout: "url_prefix": { 2024-03-19T03:42:12.676 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "url_prefix", 2024-03-19T03:42:12.676 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.676 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.676 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.676 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.676 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.676 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.676 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.676 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.676 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.676 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.676 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.676 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.676 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.676 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.677 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.677 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "devicehealth", 2024-03-19T03:42:12.677 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.677 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.677 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.677 INFO:teuthology.orchestra.run.smithi117.stdout: "enable_monitoring": { 2024-03-19T03:42:12.677 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "enable_monitoring", 2024-03-19T03:42:12.677 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.677 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.677 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.677 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.677 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.677 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.678 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.678 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "monitor device health metrics", 2024-03-19T03:42:12.678 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.678 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.678 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.678 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.678 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.678 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.678 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.678 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.678 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.678 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.678 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.679 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.679 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.679 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.679 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.679 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.679 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.679 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.679 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.679 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.679 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.679 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.679 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.679 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.679 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.679 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.680 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.680 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.680 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.680 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.680 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.680 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.680 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.680 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.680 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.680 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.680 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.680 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.680 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.680 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.681 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.681 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.681 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.681 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.681 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.681 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.681 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.681 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.681 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.681 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.681 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.681 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.681 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.681 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.681 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.682 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.682 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.682 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.682 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.682 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.682 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.682 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.682 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.682 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.682 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.682 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.682 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.682 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.682 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.683 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.683 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.683 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.683 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.683 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.683 INFO:teuthology.orchestra.run.smithi117.stdout: "mark_out_threshold": { 2024-03-19T03:42:12.683 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "mark_out_threshold", 2024-03-19T03:42:12.683 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.683 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.683 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.683 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "2419200", 2024-03-19T03:42:12.683 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.683 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.683 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.683 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-19T03:42:12.683 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.684 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.684 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.684 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.684 INFO:teuthology.orchestra.run.smithi117.stdout: "pool_name": { 2024-03-19T03:42:12.684 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "pool_name", 2024-03-19T03:42:12.684 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.684 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.684 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.684 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "device_health_metrics", 2024-03-19T03:42:12.684 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.684 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.684 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.684 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-19T03:42:12.684 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.684 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.684 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.685 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.685 INFO:teuthology.orchestra.run.smithi117.stdout: "retention_period": { 2024-03-19T03:42:12.685 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "retention_period", 2024-03-19T03:42:12.685 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.685 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.685 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.685 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "15552000", 2024-03-19T03:42:12.685 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.685 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.685 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.685 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "how long to retain device health metrics", 2024-03-19T03:42:12.685 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.685 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.685 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.685 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.686 INFO:teuthology.orchestra.run.smithi117.stdout: "scrape_frequency": { 2024-03-19T03:42:12.686 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "scrape_frequency", 2024-03-19T03:42:12.686 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.686 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.686 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.686 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "86400", 2024-03-19T03:42:12.686 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.686 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.686 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.686 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-19T03:42:12.686 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.686 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.686 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.687 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.687 INFO:teuthology.orchestra.run.smithi117.stdout: "self_heal": { 2024-03-19T03:42:12.687 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "self_heal", 2024-03-19T03:42:12.687 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.687 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.687 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.687 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.687 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.687 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.687 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.687 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-19T03:42:12.687 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.687 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.687 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.688 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.688 INFO:teuthology.orchestra.run.smithi117.stdout: "sleep_interval": { 2024-03-19T03:42:12.688 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "sleep_interval", 2024-03-19T03:42:12.688 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.688 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.688 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.688 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "600", 2024-03-19T03:42:12.688 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.688 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.688 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.688 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "how frequently to wake up and check device health", 2024-03-19T03:42:12.688 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.688 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.688 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.689 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.689 INFO:teuthology.orchestra.run.smithi117.stdout: "warn_threshold": { 2024-03-19T03:42:12.689 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "warn_threshold", 2024-03-19T03:42:12.689 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.689 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.689 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.689 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "7257600", 2024-03-19T03:42:12.689 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.689 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.689 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.689 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-19T03:42:12.689 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.689 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.689 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.689 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.690 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.690 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.690 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.690 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "diskprediction_local", 2024-03-19T03:42:12.690 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.690 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.690 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.690 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.690 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.690 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.690 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.690 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.691 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.691 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.691 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.691 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.691 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.691 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.691 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.691 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.691 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.691 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.691 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.691 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.691 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.691 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.691 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.692 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.692 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.692 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.692 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.692 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.692 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.692 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.692 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.692 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.692 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.692 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.692 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.692 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.692 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.692 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.693 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.693 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.693 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.693 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.693 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.693 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.693 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.693 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.693 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.693 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.693 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.693 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.693 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.693 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.693 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.694 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.694 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.694 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.694 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.694 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.694 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.694 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.694 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.694 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.694 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.694 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.694 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.694 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.694 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.695 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.695 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.695 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.695 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.695 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.695 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.695 INFO:teuthology.orchestra.run.smithi117.stdout: "predict_interval": { 2024-03-19T03:42:12.695 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "predict_interval", 2024-03-19T03:42:12.695 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.695 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.695 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.695 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "86400", 2024-03-19T03:42:12.695 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.695 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.695 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.696 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.696 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.696 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.696 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.696 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.696 INFO:teuthology.orchestra.run.smithi117.stdout: "predictor_model": { 2024-03-19T03:42:12.696 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "predictor_model", 2024-03-19T03:42:12.696 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.696 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.696 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.696 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "prophetstor", 2024-03-19T03:42:12.696 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.696 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.696 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.696 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.697 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.697 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.697 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.697 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.697 INFO:teuthology.orchestra.run.smithi117.stdout: "sleep_interval": { 2024-03-19T03:42:12.697 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "sleep_interval", 2024-03-19T03:42:12.697 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.697 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.697 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.697 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "600", 2024-03-19T03:42:12.697 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.697 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.697 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.697 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.697 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.698 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.698 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.698 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.698 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.698 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.698 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.698 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "influx", 2024-03-19T03:42:12.698 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": false, 2024-03-19T03:42:12.698 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "influxdb python module not found", 2024-03-19T03:42:12.698 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.698 INFO:teuthology.orchestra.run.smithi117.stdout: "batch_size": { 2024-03-19T03:42:12.698 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "batch_size", 2024-03-19T03:42:12.698 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.699 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.699 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.699 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "5000", 2024-03-19T03:42:12.699 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.699 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.699 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.699 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-19T03:42:12.699 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.699 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.699 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.699 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.699 INFO:teuthology.orchestra.run.smithi117.stdout: "database": { 2024-03-19T03:42:12.699 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "database", 2024-03-19T03:42:12.699 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.699 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.700 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.700 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "ceph", 2024-03-19T03:42:12.700 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.700 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.700 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.700 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-19T03:42:12.700 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.700 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.700 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.700 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.700 INFO:teuthology.orchestra.run.smithi117.stdout: "hostname": { 2024-03-19T03:42:12.700 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "hostname", 2024-03-19T03:42:12.700 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.700 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.701 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.701 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.701 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.701 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.701 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.701 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "InfluxDB server hostname", 2024-03-19T03:42:12.701 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.701 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.701 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.701 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.701 INFO:teuthology.orchestra.run.smithi117.stdout: "interval": { 2024-03-19T03:42:12.701 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "interval", 2024-03-19T03:42:12.701 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.701 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.701 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.702 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "30", 2024-03-19T03:42:12.702 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "5", 2024-03-19T03:42:12.702 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.702 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.702 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-19T03:42:12.702 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.702 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.702 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.702 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.702 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.702 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.702 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.702 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.702 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.703 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.703 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.703 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.703 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.703 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.703 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.703 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.703 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.703 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.703 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.703 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.703 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.703 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.703 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.704 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.704 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.704 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.704 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.704 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.704 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.704 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.704 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.704 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.704 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.704 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.704 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.704 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.704 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.705 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.705 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.705 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.705 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.705 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.705 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.705 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.705 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.705 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.705 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.705 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.705 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.705 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.706 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.706 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.706 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.706 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.706 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.706 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.706 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.706 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.706 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.706 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.706 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.706 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.706 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.706 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.707 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.707 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.707 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.707 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.707 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.707 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.707 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.707 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.707 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.707 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.707 INFO:teuthology.orchestra.run.smithi117.stdout: "password": { 2024-03-19T03:42:12.707 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "password", 2024-03-19T03:42:12.707 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.707 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.708 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.708 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.708 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.708 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.708 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.708 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "password of InfluxDB server user", 2024-03-19T03:42:12.708 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.708 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.708 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.708 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.708 INFO:teuthology.orchestra.run.smithi117.stdout: "port": { 2024-03-19T03:42:12.708 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "port", 2024-03-19T03:42:12.708 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.708 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.708 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.709 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "8086", 2024-03-19T03:42:12.709 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.709 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.709 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.709 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "InfluxDB server port", 2024-03-19T03:42:12.709 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.709 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.709 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.709 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.709 INFO:teuthology.orchestra.run.smithi117.stdout: "ssl": { 2024-03-19T03:42:12.709 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ssl", 2024-03-19T03:42:12.709 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.709 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.709 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.709 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "false", 2024-03-19T03:42:12.710 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.710 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.710 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.710 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-19T03:42:12.710 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.710 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.710 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.710 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.710 INFO:teuthology.orchestra.run.smithi117.stdout: "threads": { 2024-03-19T03:42:12.710 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "threads", 2024-03-19T03:42:12.710 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.710 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.710 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.711 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "5", 2024-03-19T03:42:12.711 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "1", 2024-03-19T03:42:12.711 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "32", 2024-03-19T03:42:12.711 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.711 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-19T03:42:12.711 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.711 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.711 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.711 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.711 INFO:teuthology.orchestra.run.smithi117.stdout: "username": { 2024-03-19T03:42:12.711 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "username", 2024-03-19T03:42:12.711 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.711 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.711 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.711 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.712 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.712 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.712 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.712 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "username of InfluxDB server user", 2024-03-19T03:42:12.712 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.712 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.712 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.712 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.712 INFO:teuthology.orchestra.run.smithi117.stdout: "verify_ssl": { 2024-03-19T03:42:12.712 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "verify_ssl", 2024-03-19T03:42:12.712 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.712 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.712 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.713 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "true", 2024-03-19T03:42:12.713 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.713 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.713 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.713 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-19T03:42:12.713 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.713 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.713 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.713 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.713 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.713 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.713 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.713 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "insights", 2024-03-19T03:42:12.714 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.714 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.714 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.714 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.714 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.714 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.714 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.714 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.714 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.714 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.714 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.714 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.714 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.714 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.715 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.715 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.715 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.715 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.715 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.715 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.715 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.715 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.715 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.715 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.715 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.715 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.715 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.715 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.716 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.716 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.716 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.716 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.716 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.716 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.716 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.716 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.716 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.716 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.716 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.716 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.716 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.716 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.717 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.717 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.717 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.717 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.717 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.717 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.717 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.717 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.717 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.717 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.717 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.717 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.717 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.718 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.718 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.718 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.718 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.718 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.718 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.718 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.718 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.718 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.718 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.718 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.718 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.718 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.719 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.719 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.719 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.719 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.719 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.719 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.719 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.719 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.719 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "iostat", 2024-03-19T03:42:12.719 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.719 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.719 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.719 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.719 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.719 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.720 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.720 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.720 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.720 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.720 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.720 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.720 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.720 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.720 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.720 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.720 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.720 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.720 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.720 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.721 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.721 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.721 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.721 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.721 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.721 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.721 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.721 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.721 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.721 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.721 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.721 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.721 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.721 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.722 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.722 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.722 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.722 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.722 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.722 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.722 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.722 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.722 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.722 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.722 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.722 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.722 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.722 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.723 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.723 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.723 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.723 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.723 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.723 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.723 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.723 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.723 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.723 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.723 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.723 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.723 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.723 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.723 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.724 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.724 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.724 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.724 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.724 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.724 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.724 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.724 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.724 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.724 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.724 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.724 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.724 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.724 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "k8sevents", 2024-03-19T03:42:12.725 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.725 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.725 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.725 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_event_retention_days": { 2024-03-19T03:42:12.725 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ceph_event_retention_days", 2024-03-19T03:42:12.725 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.725 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.725 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.725 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "7", 2024-03-19T03:42:12.725 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.725 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.725 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.725 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-19T03:42:12.725 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.725 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.726 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.726 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.726 INFO:teuthology.orchestra.run.smithi117.stdout: "config_check_secs": { 2024-03-19T03:42:12.726 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "config_check_secs", 2024-03-19T03:42:12.726 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.726 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.726 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.726 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10", 2024-03-19T03:42:12.726 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "10", 2024-03-19T03:42:12.726 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.726 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.726 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-19T03:42:12.726 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.727 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.727 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.727 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.727 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.727 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.727 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.727 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.727 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.727 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.727 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.727 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.727 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.727 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.727 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.728 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.728 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.728 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.728 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.728 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.728 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.728 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.728 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.728 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.728 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.728 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.728 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.728 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.729 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.729 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.729 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.729 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.729 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.729 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.729 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.729 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.729 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.729 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.729 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.729 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.730 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.730 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.730 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.730 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.730 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.730 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.730 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.730 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.730 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.730 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.730 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.730 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.730 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.730 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.731 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.731 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.731 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.731 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.731 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.731 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.731 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.731 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.731 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.731 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.731 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.731 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.731 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.731 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.731 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.732 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.732 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.732 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.732 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.732 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.732 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.732 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.732 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.732 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "localpool", 2024-03-19T03:42:12.732 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.732 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.732 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.732 INFO:teuthology.orchestra.run.smithi117.stdout: "failure_domain": { 2024-03-19T03:42:12.732 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "failure_domain", 2024-03-19T03:42:12.733 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.733 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.733 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.733 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "host", 2024-03-19T03:42:12.733 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.733 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.733 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.733 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "failure domain for any created local pool", 2024-03-19T03:42:12.733 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-19T03:42:12.733 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.733 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.733 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.733 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.734 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.734 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.734 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.734 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.734 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.734 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.734 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.734 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.734 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.734 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.734 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.734 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.734 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.734 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.735 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.735 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.735 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.735 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.735 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.735 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.735 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.735 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.735 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.735 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.735 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.735 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.735 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.735 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.735 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.736 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.736 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.736 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.736 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.736 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.736 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.736 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.736 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.736 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.736 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.736 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.736 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.736 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.736 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.737 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.737 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.737 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.737 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.737 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.737 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.737 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.737 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.737 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.737 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.737 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.737 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.737 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.737 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.737 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.737 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.738 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.738 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.738 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.738 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.738 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.738 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.738 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.738 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.738 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.738 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.738 INFO:teuthology.orchestra.run.smithi117.stdout: "min_size": { 2024-03-19T03:42:12.738 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "min_size", 2024-03-19T03:42:12.738 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.738 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.739 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.739 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.739 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.739 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.739 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.739 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "default min_size for any created local pool", 2024-03-19T03:42:12.739 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-19T03:42:12.739 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.739 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.739 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.739 INFO:teuthology.orchestra.run.smithi117.stdout: "num_rep": { 2024-03-19T03:42:12.739 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "num_rep", 2024-03-19T03:42:12.739 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.739 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.739 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.740 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "3", 2024-03-19T03:42:12.740 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.740 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.740 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.740 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "default replica count for any created local pool", 2024-03-19T03:42:12.740 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.740 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.740 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.740 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.740 INFO:teuthology.orchestra.run.smithi117.stdout: "pg_num": { 2024-03-19T03:42:12.740 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "pg_num", 2024-03-19T03:42:12.740 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.740 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.741 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.741 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "128", 2024-03-19T03:42:12.741 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.741 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.741 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.741 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "default pg_num for any created local pool", 2024-03-19T03:42:12.741 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.741 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.741 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.741 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.741 INFO:teuthology.orchestra.run.smithi117.stdout: "prefix": { 2024-03-19T03:42:12.741 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "prefix", 2024-03-19T03:42:12.741 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.741 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.741 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.742 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.742 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.742 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.742 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.742 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "name prefix for any created local pool", 2024-03-19T03:42:12.742 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.742 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.742 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.742 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.742 INFO:teuthology.orchestra.run.smithi117.stdout: "subtree": { 2024-03-19T03:42:12.742 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "subtree", 2024-03-19T03:42:12.742 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.742 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.742 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.743 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "rack", 2024-03-19T03:42:12.743 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.743 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.743 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.743 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-19T03:42:12.743 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-19T03:42:12.743 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.743 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.743 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.743 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.743 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.743 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.743 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "mds_autoscaler", 2024-03-19T03:42:12.743 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.744 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.744 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.744 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.744 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.744 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.744 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.744 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.744 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.744 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.744 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.744 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.744 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.744 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.744 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.744 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.745 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.745 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.745 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.745 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.745 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.745 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.745 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.745 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.745 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.745 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.745 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.745 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.745 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.745 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.746 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.746 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.746 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.746 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.746 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.746 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.746 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.746 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.746 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.746 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.746 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.746 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.746 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.747 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.747 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.747 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.747 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.747 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.747 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.747 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.747 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.747 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.747 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.747 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.747 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.747 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.747 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.747 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.748 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.748 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.748 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.748 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.748 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.748 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.748 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.748 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.748 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.748 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.748 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.748 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.748 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.748 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.748 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.749 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.749 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.749 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.749 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "mirroring", 2024-03-19T03:42:12.749 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.749 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.749 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.749 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.749 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.749 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.749 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.749 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.749 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.749 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.749 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.749 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.750 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.750 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.750 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.750 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.750 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.750 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.750 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.750 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.750 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.750 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.750 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.750 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.751 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.751 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.751 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.751 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.751 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.751 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.751 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.751 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.751 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.751 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.751 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.751 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.751 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.751 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.751 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.752 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.752 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.752 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.752 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.752 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.752 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.752 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.752 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.752 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.752 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.752 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.752 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.752 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.752 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.752 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.753 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.753 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.753 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.753 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.753 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.753 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.753 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.753 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.753 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.753 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.753 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.753 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.753 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.753 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.754 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.754 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.754 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.754 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.754 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.754 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.754 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.754 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.754 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "nfs", 2024-03-19T03:42:12.754 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.754 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.754 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.754 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.754 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.755 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.755 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.755 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.755 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.755 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.755 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.755 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.755 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.755 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.755 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.755 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.755 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.755 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.755 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.756 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.756 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.756 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.756 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.756 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.756 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.756 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.756 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.756 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.756 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.756 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.756 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.756 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.756 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.757 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.757 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.757 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.757 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.757 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.757 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.757 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.757 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.757 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.757 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.757 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.757 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.757 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.757 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.757 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.758 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.758 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.758 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.758 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.758 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.758 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.758 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.758 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.758 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.758 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.758 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.758 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.758 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.759 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.759 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.759 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.759 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.759 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.759 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.759 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.759 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.759 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.759 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.759 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.759 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.759 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.759 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.759 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.760 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "orchestrator", 2024-03-19T03:42:12.760 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.760 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.760 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.760 INFO:teuthology.orchestra.run.smithi117.stdout: "fail_fs": { 2024-03-19T03:42:12.760 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "fail_fs", 2024-03-19T03:42:12.760 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.760 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.760 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.760 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.760 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.760 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.760 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.760 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-19T03:42:12.760 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.761 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.761 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.761 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.761 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.761 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.761 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.761 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.761 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.761 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.761 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.761 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.761 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.761 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.761 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.761 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.762 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.762 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.762 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.762 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.762 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.762 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.762 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.762 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.762 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.762 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.762 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.762 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.762 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.762 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.763 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.763 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.763 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.763 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.763 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.763 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.763 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.763 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.763 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.763 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.763 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.763 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.763 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.763 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.764 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.764 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.764 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.764 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.764 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.764 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.764 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.764 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.764 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.764 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.764 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.764 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.764 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.764 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.764 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.765 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.765 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.765 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.765 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.765 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.765 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.765 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.765 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.765 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.765 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.765 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.765 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.765 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.765 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.765 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.766 INFO:teuthology.orchestra.run.smithi117.stdout: "orchestrator": { 2024-03-19T03:42:12.766 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "orchestrator", 2024-03-19T03:42:12.766 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.766 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.766 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.766 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.766 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.766 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.766 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.766 INFO:teuthology.orchestra.run.smithi117.stdout: "cephadm", 2024-03-19T03:42:12.766 INFO:teuthology.orchestra.run.smithi117.stdout: "rook", 2024-03-19T03:42:12.766 INFO:teuthology.orchestra.run.smithi117.stdout: "test_orchestrator" 2024-03-19T03:42:12.766 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.766 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Orchestrator backend", 2024-03-19T03:42:12.767 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.767 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.767 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.767 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.767 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.767 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.767 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.767 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "osd_perf_query", 2024-03-19T03:42:12.767 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.767 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.767 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.767 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.767 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.767 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.768 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.768 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.768 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.768 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.768 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.768 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.768 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.768 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.768 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.768 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.768 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.768 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.768 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.768 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.768 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.769 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.769 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.769 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.769 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.769 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.769 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.769 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.769 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.769 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.769 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.769 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.769 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.769 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.769 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.770 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.770 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.770 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.770 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.770 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.770 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.770 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.770 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.770 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.770 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.770 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.770 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.770 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.770 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.771 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.771 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.771 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.771 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.771 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.771 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.771 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.771 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.771 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.771 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.771 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.771 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.771 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.771 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.772 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.772 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.772 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.772 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.772 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.772 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.772 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.772 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.772 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.772 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.772 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.772 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.772 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.772 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "osd_support", 2024-03-19T03:42:12.773 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.773 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.773 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.773 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.773 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.773 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.773 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.773 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.773 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.773 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.773 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.773 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.773 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.773 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.773 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.774 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.774 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.774 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.774 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.774 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.774 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.774 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.774 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.774 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.774 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.774 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.774 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.774 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.775 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.775 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.775 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.775 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.775 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.775 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.775 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.775 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.775 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.775 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.775 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.775 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.775 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.775 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.775 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.775 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.776 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.776 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.776 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.776 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.776 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.776 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.776 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.776 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.776 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.776 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.776 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.776 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.776 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.776 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.776 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.777 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.777 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.777 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.777 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.777 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.777 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.777 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.777 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.777 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.777 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.777 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.777 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.777 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.777 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.777 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.777 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.777 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.778 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "pg_autoscaler", 2024-03-19T03:42:12.778 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.778 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.778 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.778 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.778 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.778 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.778 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.778 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.778 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.778 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.778 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.778 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.778 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.779 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.779 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.779 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.779 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.779 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.779 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.779 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.779 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.779 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.779 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.779 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.779 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.779 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.779 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.779 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.780 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.780 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.780 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.780 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.780 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.780 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.780 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.780 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.780 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.780 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.780 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.780 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.780 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.780 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.780 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.781 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.781 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.781 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.781 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.781 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.781 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.781 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.781 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.781 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.781 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.781 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.781 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.782 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.782 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.782 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.782 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.782 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.782 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.782 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.782 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.782 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.782 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.782 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.782 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.782 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.782 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.783 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.783 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.783 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.783 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.783 INFO:teuthology.orchestra.run.smithi117.stdout: "sleep_interval": { 2024-03-19T03:42:12.783 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "sleep_interval", 2024-03-19T03:42:12.783 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.783 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.783 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.783 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "60", 2024-03-19T03:42:12.783 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.783 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.783 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.783 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.783 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.784 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.784 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.784 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.784 INFO:teuthology.orchestra.run.smithi117.stdout: "threshold": { 2024-03-19T03:42:12.784 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "threshold", 2024-03-19T03:42:12.784 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "float", 2024-03-19T03:42:12.784 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.784 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.784 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "3.0", 2024-03-19T03:42:12.784 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "1.0", 2024-03-19T03:42:12.784 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.784 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.784 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "scaling threshold", 2024-03-19T03:42:12.784 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-19T03:42:12.785 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.785 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.785 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.785 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.785 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.785 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.785 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "progress", 2024-03-19T03:42:12.785 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.785 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.785 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.785 INFO:teuthology.orchestra.run.smithi117.stdout: "allow_pg_recovery_event": { 2024-03-19T03:42:12.785 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "allow_pg_recovery_event", 2024-03-19T03:42:12.785 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.785 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.785 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.786 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.786 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.786 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.786 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.786 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-19T03:42:12.786 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.786 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.786 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.786 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.786 INFO:teuthology.orchestra.run.smithi117.stdout: "enabled": { 2024-03-19T03:42:12.786 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "enabled", 2024-03-19T03:42:12.786 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.786 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.787 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.787 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.787 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.787 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.787 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.787 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.787 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.787 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.787 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.787 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.787 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.787 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.787 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.787 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.787 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.787 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.788 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.788 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.788 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.788 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.788 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.788 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.788 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.788 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.788 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.788 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.788 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.788 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.788 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.788 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.789 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.789 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.789 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.789 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.789 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.789 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.789 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.789 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.789 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.789 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.789 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.789 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.789 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.789 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.789 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.790 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.790 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.790 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.790 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.790 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.790 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.790 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.790 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.790 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.790 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.790 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.790 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.790 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.791 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.791 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.791 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.791 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.791 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.791 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.791 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.791 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.791 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.791 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.791 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.791 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.791 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.791 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.792 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.792 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.792 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.792 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.792 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.792 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.792 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.792 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.792 INFO:teuthology.orchestra.run.smithi117.stdout: "max_completed_events": { 2024-03-19T03:42:12.792 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "max_completed_events", 2024-03-19T03:42:12.792 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.792 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.792 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.792 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "50", 2024-03-19T03:42:12.793 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.793 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.793 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.793 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "number of past completed events to remember", 2024-03-19T03:42:12.793 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.793 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.793 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.793 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.793 INFO:teuthology.orchestra.run.smithi117.stdout: "sleep_interval": { 2024-03-19T03:42:12.793 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "sleep_interval", 2024-03-19T03:42:12.793 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.793 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.793 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.793 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "5", 2024-03-19T03:42:12.794 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.794 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.794 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.794 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "how long the module is going to sleep", 2024-03-19T03:42:12.794 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.794 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.794 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.794 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.794 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.794 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.794 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.794 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "prometheus", 2024-03-19T03:42:12.795 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.795 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.795 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.795 INFO:teuthology.orchestra.run.smithi117.stdout: "cache": { 2024-03-19T03:42:12.795 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "cache", 2024-03-19T03:42:12.795 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.795 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.795 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.795 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.795 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.795 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.795 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.795 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.795 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.796 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.796 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.796 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.796 INFO:teuthology.orchestra.run.smithi117.stdout: "exclude_perf_counters": { 2024-03-19T03:42:12.796 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "exclude_perf_counters", 2024-03-19T03:42:12.796 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.796 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.796 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.796 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.796 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.796 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.796 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.796 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-19T03:42:12.797 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-19T03:42:12.797 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.797 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.797 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.797 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.797 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.797 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.797 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.797 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.797 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.797 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.797 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.797 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.798 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.798 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.798 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.798 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.798 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.798 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.798 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.798 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.798 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.798 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.798 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.798 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.798 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.798 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.799 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.799 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.799 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.799 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.799 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.799 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.799 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.799 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.799 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.799 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.799 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.799 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.799 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.799 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.800 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.800 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.800 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.800 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.800 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.800 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.800 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.800 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.800 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.800 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.800 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.800 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.800 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.800 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.800 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.801 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.801 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.801 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.801 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.801 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.801 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.801 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.801 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.801 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.801 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.801 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.801 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.801 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.801 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.802 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.802 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.802 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.802 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.802 INFO:teuthology.orchestra.run.smithi117.stdout: "rbd_stats_pools": { 2024-03-19T03:42:12.802 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rbd_stats_pools", 2024-03-19T03:42:12.802 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.802 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.802 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.802 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.802 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.802 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.802 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.802 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.803 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.803 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.803 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.803 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.803 INFO:teuthology.orchestra.run.smithi117.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-19T03:42:12.803 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-19T03:42:12.803 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.803 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.803 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.803 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "300", 2024-03-19T03:42:12.803 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.803 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.803 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.803 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.804 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.804 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.804 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.804 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.804 INFO:teuthology.orchestra.run.smithi117.stdout: "scrape_interval": { 2024-03-19T03:42:12.804 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "scrape_interval", 2024-03-19T03:42:12.804 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "float", 2024-03-19T03:42:12.804 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.804 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.804 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "15.0", 2024-03-19T03:42:12.804 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.804 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.804 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.804 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.804 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.805 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.805 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.805 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.805 INFO:teuthology.orchestra.run.smithi117.stdout: "server_addr": { 2024-03-19T03:42:12.805 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "server_addr", 2024-03-19T03:42:12.805 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.805 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.805 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.805 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "::", 2024-03-19T03:42:12.805 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.805 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.805 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.805 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-19T03:42:12.805 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.805 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.806 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.806 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.806 INFO:teuthology.orchestra.run.smithi117.stdout: "server_port": { 2024-03-19T03:42:12.806 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "server_port", 2024-03-19T03:42:12.806 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.806 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.806 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.806 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "9283", 2024-03-19T03:42:12.806 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.806 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.806 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.806 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-19T03:42:12.806 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.807 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.807 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.807 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.807 INFO:teuthology.orchestra.run.smithi117.stdout: "stale_cache_strategy": { 2024-03-19T03:42:12.807 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "stale_cache_strategy", 2024-03-19T03:42:12.807 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.807 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.807 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.807 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "log", 2024-03-19T03:42:12.807 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.807 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.807 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.807 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.808 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.808 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.808 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.808 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.808 INFO:teuthology.orchestra.run.smithi117.stdout: "standby_behaviour": { 2024-03-19T03:42:12.808 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "standby_behaviour", 2024-03-19T03:42:12.808 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.808 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.808 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.808 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "default", 2024-03-19T03:42:12.808 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.808 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.808 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.808 INFO:teuthology.orchestra.run.smithi117.stdout: "default", 2024-03-19T03:42:12.808 INFO:teuthology.orchestra.run.smithi117.stdout: "error" 2024-03-19T03:42:12.809 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.809 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.809 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.809 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.809 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.809 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.809 INFO:teuthology.orchestra.run.smithi117.stdout: "standby_error_status_code": { 2024-03-19T03:42:12.809 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "standby_error_status_code", 2024-03-19T03:42:12.809 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.809 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.809 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.809 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "500", 2024-03-19T03:42:12.809 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "400", 2024-03-19T03:42:12.809 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "599", 2024-03-19T03:42:12.810 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.810 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.810 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.810 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.810 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.810 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.810 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.810 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.810 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.810 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rbd_support", 2024-03-19T03:42:12.810 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.810 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.810 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.810 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.811 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.811 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.811 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.811 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.811 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.811 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.811 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.811 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.811 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.811 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.811 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.811 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.811 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.811 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.812 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.812 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.812 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.812 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.812 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.812 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.812 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.812 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.812 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.812 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.812 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.812 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.812 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.812 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.812 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.813 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.813 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.813 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.813 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.813 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.813 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.813 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.813 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.813 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.813 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.813 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.813 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.813 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.813 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.814 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.814 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.814 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.814 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.814 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.814 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.814 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.814 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.814 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.814 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.814 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.814 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.814 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.814 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.815 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.815 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.815 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.815 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.815 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.815 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.815 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.815 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.815 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.815 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.815 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.815 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.815 INFO:teuthology.orchestra.run.smithi117.stdout: "max_concurrent_snap_create": { 2024-03-19T03:42:12.815 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "max_concurrent_snap_create", 2024-03-19T03:42:12.815 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.816 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.816 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.816 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10", 2024-03-19T03:42:12.816 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.816 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.816 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.816 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.816 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.816 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.816 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.816 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.816 INFO:teuthology.orchestra.run.smithi117.stdout: "mirror_snapshot_schedule": { 2024-03-19T03:42:12.816 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "mirror_snapshot_schedule", 2024-03-19T03:42:12.816 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.816 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.816 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.817 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.817 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.817 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.817 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.817 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.817 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.817 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.817 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.817 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.817 INFO:teuthology.orchestra.run.smithi117.stdout: "trash_purge_schedule": { 2024-03-19T03:42:12.817 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "trash_purge_schedule", 2024-03-19T03:42:12.817 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.817 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.817 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.817 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.818 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.818 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.818 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.818 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.818 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.818 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.818 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.818 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.818 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.818 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.818 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.818 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "restful", 2024-03-19T03:42:12.818 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.818 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.818 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.819 INFO:teuthology.orchestra.run.smithi117.stdout: "enable_auth": { 2024-03-19T03:42:12.819 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "enable_auth", 2024-03-19T03:42:12.819 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.819 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.819 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.819 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.819 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.819 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.819 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.819 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.819 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.819 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.819 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.819 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.819 INFO:teuthology.orchestra.run.smithi117.stdout: "key_file": { 2024-03-19T03:42:12.820 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "key_file", 2024-03-19T03:42:12.820 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.820 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.820 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.820 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.820 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.820 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.820 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.820 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.820 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.820 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.820 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.820 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.820 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.820 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.821 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.821 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.821 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.821 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.821 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.821 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.821 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.821 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.821 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.822 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.822 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.822 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.822 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.822 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.822 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.822 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.822 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.822 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.822 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.822 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.822 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.822 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.823 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.823 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.823 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.823 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.823 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.823 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.823 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.823 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.823 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.823 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.823 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.823 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.823 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.823 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.823 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.824 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.824 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.824 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.824 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.824 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.824 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.824 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.824 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.824 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.824 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.824 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.824 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.824 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.824 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.824 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.825 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.825 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.825 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.825 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.825 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.825 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.825 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.825 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.825 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.825 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.825 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.825 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.825 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.826 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.826 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.826 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.826 INFO:teuthology.orchestra.run.smithi117.stdout: "server_addr": { 2024-03-19T03:42:12.826 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "server_addr", 2024-03-19T03:42:12.826 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.826 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.826 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.826 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.826 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.826 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.826 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.826 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.826 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.827 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.827 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.827 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.827 INFO:teuthology.orchestra.run.smithi117.stdout: "server_port": { 2024-03-19T03:42:12.827 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "server_port", 2024-03-19T03:42:12.827 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.827 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.827 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.827 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.827 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.827 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.827 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.827 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.827 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.828 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.828 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.828 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.828 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.828 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.828 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.828 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rgw", 2024-03-19T03:42:12.828 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.828 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.828 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.828 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.828 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.828 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.828 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.828 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.829 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.829 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.829 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.829 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.829 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.829 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.829 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.829 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.829 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.829 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.829 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.829 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.829 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.829 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.830 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.830 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.830 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.830 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.830 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.830 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.830 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.830 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.830 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.830 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.830 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.830 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.830 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.830 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.831 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.831 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.831 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.831 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.831 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.831 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.831 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.831 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.831 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.831 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.831 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.831 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.831 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.831 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.831 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.832 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.832 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.832 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.832 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.832 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.832 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.832 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.832 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.832 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.832 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.832 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.832 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.832 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.832 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.833 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.833 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.833 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.833 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.833 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.833 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.833 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.833 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.833 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.833 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.833 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.833 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rook", 2024-03-19T03:42:12.833 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.834 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.834 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.834 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.834 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.834 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.834 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.834 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.834 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.834 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.834 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.834 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.834 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.834 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.835 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.835 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.835 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.835 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.835 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.835 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.835 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.835 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.835 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.835 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.835 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.835 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.835 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.836 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.836 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.836 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.836 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.836 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.836 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.836 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.836 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.836 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.836 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.836 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.836 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.836 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.836 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.837 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.837 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.837 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.837 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.837 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.837 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.837 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.837 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.837 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.837 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.837 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.837 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.837 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.837 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.838 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.838 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.838 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.838 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.838 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.838 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.838 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.838 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.838 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.838 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.838 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.838 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.838 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.839 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.839 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.839 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.839 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.839 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.839 INFO:teuthology.orchestra.run.smithi117.stdout: "storage_class": { 2024-03-19T03:42:12.839 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "storage_class", 2024-03-19T03:42:12.839 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.839 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.839 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.839 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "local", 2024-03-19T03:42:12.839 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.839 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.839 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.840 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-19T03:42:12.840 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.840 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.840 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.840 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.840 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.840 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.840 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.840 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "selftest", 2024-03-19T03:42:12.840 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.840 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.840 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.840 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.840 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.841 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.841 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.841 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.841 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.841 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.841 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.841 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.841 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.841 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.841 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.841 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.841 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.841 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.841 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.841 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.841 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.842 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.842 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.842 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.842 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.842 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.842 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.842 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.842 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.842 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.842 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.842 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.842 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.842 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.843 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.843 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.843 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.843 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.843 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.843 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.843 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.843 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.843 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.843 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.843 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.843 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.843 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.843 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.844 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.844 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.844 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.844 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.844 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.844 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.844 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.844 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.844 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.844 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.844 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.844 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.844 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.844 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.845 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.845 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.845 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.845 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.845 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.845 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.845 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.845 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.845 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.845 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.845 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.845 INFO:teuthology.orchestra.run.smithi117.stdout: "roption1": { 2024-03-19T03:42:12.845 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "roption1", 2024-03-19T03:42:12.845 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.845 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.846 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.846 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.846 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.846 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.846 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.846 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.846 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.846 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.846 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.846 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.846 INFO:teuthology.orchestra.run.smithi117.stdout: "roption2": { 2024-03-19T03:42:12.846 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "roption2", 2024-03-19T03:42:12.846 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.846 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.846 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.847 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "xyz", 2024-03-19T03:42:12.847 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.847 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.847 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.847 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.847 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.847 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.847 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.847 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.847 INFO:teuthology.orchestra.run.smithi117.stdout: "rwoption1": { 2024-03-19T03:42:12.847 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rwoption1", 2024-03-19T03:42:12.847 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.847 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.847 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.847 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.848 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.848 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.848 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.848 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.848 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.848 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.848 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.848 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.848 INFO:teuthology.orchestra.run.smithi117.stdout: "rwoption2": { 2024-03-19T03:42:12.848 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rwoption2", 2024-03-19T03:42:12.848 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.848 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.848 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.848 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.848 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.849 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.849 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.849 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.849 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.849 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.849 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.849 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.849 INFO:teuthology.orchestra.run.smithi117.stdout: "rwoption3": { 2024-03-19T03:42:12.849 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rwoption3", 2024-03-19T03:42:12.849 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "float", 2024-03-19T03:42:12.849 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.849 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.849 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.849 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.850 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.850 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.850 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.850 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.850 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.850 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.850 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.850 INFO:teuthology.orchestra.run.smithi117.stdout: "rwoption4": { 2024-03-19T03:42:12.850 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rwoption4", 2024-03-19T03:42:12.850 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.850 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.850 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.850 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.850 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.851 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.851 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.851 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.851 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.851 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.851 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.851 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.851 INFO:teuthology.orchestra.run.smithi117.stdout: "rwoption5": { 2024-03-19T03:42:12.851 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rwoption5", 2024-03-19T03:42:12.851 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.851 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.851 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.851 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.851 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.851 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.852 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.852 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.852 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.852 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.852 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.852 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.852 INFO:teuthology.orchestra.run.smithi117.stdout: "rwoption6": { 2024-03-19T03:42:12.852 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rwoption6", 2024-03-19T03:42:12.852 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.852 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.852 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.852 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.852 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.852 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.852 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.853 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.853 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.853 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.853 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.853 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.853 INFO:teuthology.orchestra.run.smithi117.stdout: "rwoption7": { 2024-03-19T03:42:12.853 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rwoption7", 2024-03-19T03:42:12.853 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.853 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.853 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.853 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.853 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "1", 2024-03-19T03:42:12.853 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "42", 2024-03-19T03:42:12.853 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.853 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.854 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.854 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.854 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.854 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.854 INFO:teuthology.orchestra.run.smithi117.stdout: "testkey": { 2024-03-19T03:42:12.854 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "testkey", 2024-03-19T03:42:12.854 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.854 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.854 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.854 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.854 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.854 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.854 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.855 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.855 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.855 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.855 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.855 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.855 INFO:teuthology.orchestra.run.smithi117.stdout: "testlkey": { 2024-03-19T03:42:12.855 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "testlkey", 2024-03-19T03:42:12.855 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.855 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.855 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.855 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.855 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.855 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.855 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.856 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.856 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.856 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.856 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.856 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.856 INFO:teuthology.orchestra.run.smithi117.stdout: "testnewline": { 2024-03-19T03:42:12.856 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "testnewline", 2024-03-19T03:42:12.856 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.856 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.856 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.856 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.856 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.856 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.856 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.856 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.857 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.857 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.857 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.857 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.857 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.857 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.857 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.857 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "snap_schedule", 2024-03-19T03:42:12.857 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.857 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.857 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.857 INFO:teuthology.orchestra.run.smithi117.stdout: "allow_m_granularity": { 2024-03-19T03:42:12.857 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "allow_m_granularity", 2024-03-19T03:42:12.857 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.858 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.858 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.858 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.858 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.858 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.858 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.858 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "allow minute scheduled snapshots", 2024-03-19T03:42:12.858 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.858 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.858 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.858 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.858 INFO:teuthology.orchestra.run.smithi117.stdout: "dump_on_update": { 2024-03-19T03:42:12.858 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "dump_on_update", 2024-03-19T03:42:12.858 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.858 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.859 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.859 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.859 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.859 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.859 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.859 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "dump database to debug log on update", 2024-03-19T03:42:12.859 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.859 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.859 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.859 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.859 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.859 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.859 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.859 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.859 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.860 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.860 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.860 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.860 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.860 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.860 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.860 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.860 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.860 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.860 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.860 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.860 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.860 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.860 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.860 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.861 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.861 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.861 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.861 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.861 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.861 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.861 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.861 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.861 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.861 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.861 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.861 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.861 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.861 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.862 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.862 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.862 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.862 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.862 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.862 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.862 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.862 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.862 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.862 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.862 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.862 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.863 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.863 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.863 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.863 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.863 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.863 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.863 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.863 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.863 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.863 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.863 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.863 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.863 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.863 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.863 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.864 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.864 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.864 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.864 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.864 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.864 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.864 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.864 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.864 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.864 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.864 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.864 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.864 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "stats", 2024-03-19T03:42:12.864 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.865 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.865 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.865 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.865 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.865 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.865 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.865 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.865 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.865 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.865 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.865 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.865 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.865 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.865 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.866 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.866 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.866 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.866 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.866 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.866 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.866 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.866 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.866 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.866 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.866 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.866 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.866 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.866 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.867 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.867 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.867 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.867 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.867 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.867 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.867 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.867 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.867 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.867 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.867 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.867 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.867 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.867 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.867 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.868 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.868 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.868 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.868 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.868 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.868 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.868 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.868 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.868 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.868 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.868 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.868 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.868 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.869 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.869 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.869 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.869 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.869 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.869 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.869 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.869 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.869 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.869 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.869 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.869 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.869 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.869 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.869 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.870 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.870 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.870 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.870 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.870 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "status", 2024-03-19T03:42:12.870 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.870 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.870 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.870 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.870 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.870 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.870 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.870 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.870 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.871 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.871 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.871 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.871 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.871 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.871 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.871 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.871 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.871 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.871 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.871 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.871 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.871 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.871 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.871 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.872 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.872 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.872 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.872 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.872 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.872 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.872 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.872 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.872 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.872 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.872 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.872 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.872 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.872 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.872 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.872 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.873 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.873 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.873 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.873 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.873 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.873 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.873 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.873 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.873 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.873 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.873 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.873 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.873 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.873 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.873 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.874 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.874 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.874 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.874 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.874 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.874 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.874 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.874 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.874 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.874 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.874 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.874 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.874 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.875 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.875 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.875 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.875 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.875 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.875 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.875 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.875 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.875 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "telegraf", 2024-03-19T03:42:12.875 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.875 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.875 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.875 INFO:teuthology.orchestra.run.smithi117.stdout: "address": { 2024-03-19T03:42:12.875 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "address", 2024-03-19T03:42:12.876 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.876 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.876 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.876 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-19T03:42:12.876 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.876 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.876 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.876 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.876 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.876 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.876 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.876 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.876 INFO:teuthology.orchestra.run.smithi117.stdout: "interval": { 2024-03-19T03:42:12.877 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "interval", 2024-03-19T03:42:12.877 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.877 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.877 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.877 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "15", 2024-03-19T03:42:12.877 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.877 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.877 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.877 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.877 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.877 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.877 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.877 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.877 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.877 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.877 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.878 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.878 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.878 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.878 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.878 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.878 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.878 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.878 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.878 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.878 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.878 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.878 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.878 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.879 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.879 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.879 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.879 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.879 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.879 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.879 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.879 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.879 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.879 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.879 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.879 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.879 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.879 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.879 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.880 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.880 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.880 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.880 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.880 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.880 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.880 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.880 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.880 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.880 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.880 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.880 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.880 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.880 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.881 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.881 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.881 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.881 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.881 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.881 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.881 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.881 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.881 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.881 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.881 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.881 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.881 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.881 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.881 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.881 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.882 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.882 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.882 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.882 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.882 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.882 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.882 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.882 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.882 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.882 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.882 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.882 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.882 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "telemetry", 2024-03-19T03:42:12.882 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.883 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.883 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.883 INFO:teuthology.orchestra.run.smithi117.stdout: "channel_basic": { 2024-03-19T03:42:12.883 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "channel_basic", 2024-03-19T03:42:12.883 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.883 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.883 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.883 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.883 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.883 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.883 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.883 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-19T03:42:12.883 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.884 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.884 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.884 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.884 INFO:teuthology.orchestra.run.smithi117.stdout: "channel_crash": { 2024-03-19T03:42:12.884 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "channel_crash", 2024-03-19T03:42:12.884 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.884 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.884 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.884 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.884 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.884 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.884 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.884 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-19T03:42:12.885 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.885 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.885 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.885 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.885 INFO:teuthology.orchestra.run.smithi117.stdout: "channel_device": { 2024-03-19T03:42:12.885 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "channel_device", 2024-03-19T03:42:12.885 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.885 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.885 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.885 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.885 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.885 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.885 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.885 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-19T03:42:12.886 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.886 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.886 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.886 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.886 INFO:teuthology.orchestra.run.smithi117.stdout: "channel_ident": { 2024-03-19T03:42:12.886 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "channel_ident", 2024-03-19T03:42:12.886 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.886 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.886 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.886 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.886 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.886 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.886 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.887 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-19T03:42:12.887 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.887 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.887 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.887 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.887 INFO:teuthology.orchestra.run.smithi117.stdout: "channel_perf": { 2024-03-19T03:42:12.887 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "channel_perf", 2024-03-19T03:42:12.887 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.887 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.887 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.887 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.887 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.887 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.888 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.888 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-19T03:42:12.888 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.888 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.888 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.888 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.888 INFO:teuthology.orchestra.run.smithi117.stdout: "contact": { 2024-03-19T03:42:12.888 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "contact", 2024-03-19T03:42:12.888 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.888 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.888 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.888 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.888 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.888 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.889 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.889 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.889 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.889 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.889 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.889 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.889 INFO:teuthology.orchestra.run.smithi117.stdout: "description": { 2024-03-19T03:42:12.889 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "description", 2024-03-19T03:42:12.889 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.889 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.889 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.889 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.889 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.890 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.890 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.890 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.890 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.890 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.890 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.890 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.890 INFO:teuthology.orchestra.run.smithi117.stdout: "device_url": { 2024-03-19T03:42:12.890 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "device_url", 2024-03-19T03:42:12.890 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.890 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.890 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.890 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-19T03:42:12.890 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.891 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.891 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.891 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.891 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.891 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.891 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.891 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.891 INFO:teuthology.orchestra.run.smithi117.stdout: "enabled": { 2024-03-19T03:42:12.891 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "enabled", 2024-03-19T03:42:12.891 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.891 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.891 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.891 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.891 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.892 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.892 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.892 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.892 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.892 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.892 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.892 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.892 INFO:teuthology.orchestra.run.smithi117.stdout: "interval": { 2024-03-19T03:42:12.892 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "interval", 2024-03-19T03:42:12.892 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.892 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.892 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.892 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "24", 2024-03-19T03:42:12.893 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "8", 2024-03-19T03:42:12.893 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.893 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.893 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.893 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.893 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.893 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.893 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.893 INFO:teuthology.orchestra.run.smithi117.stdout: "last_opt_revision": { 2024-03-19T03:42:12.893 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "last_opt_revision", 2024-03-19T03:42:12.893 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.893 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.893 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.893 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "1", 2024-03-19T03:42:12.894 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.894 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.894 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.894 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.894 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.894 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.894 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.894 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.894 INFO:teuthology.orchestra.run.smithi117.stdout: "leaderboard": { 2024-03-19T03:42:12.894 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "leaderboard", 2024-03-19T03:42:12.894 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.894 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.894 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.895 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.895 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.895 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.895 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.895 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.895 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.895 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.895 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.895 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.895 INFO:teuthology.orchestra.run.smithi117.stdout: "leaderboard_description": { 2024-03-19T03:42:12.895 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "leaderboard_description", 2024-03-19T03:42:12.895 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.895 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.895 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.896 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.896 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.896 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.896 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.896 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.896 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.896 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.896 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.896 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.896 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.896 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.896 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.896 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.896 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.897 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.897 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.897 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.897 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.897 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.897 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.897 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.897 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.897 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.897 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.897 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.897 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.897 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.897 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.898 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.898 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.898 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.898 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.898 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.898 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.898 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.898 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.898 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.898 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.898 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.898 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.898 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.898 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.899 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.899 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.899 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.899 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.899 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.899 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.899 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.899 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.899 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.899 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.899 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.899 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.899 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.899 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.900 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.900 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.900 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.900 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.900 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.900 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.900 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.900 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.900 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.900 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.900 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.900 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.900 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.900 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.901 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.901 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.901 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.901 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.901 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.901 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.901 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.901 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.901 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.901 INFO:teuthology.orchestra.run.smithi117.stdout: "organization": { 2024-03-19T03:42:12.901 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "organization", 2024-03-19T03:42:12.901 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.901 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.901 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.902 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.902 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.902 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.902 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.902 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.902 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.902 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.902 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.902 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.902 INFO:teuthology.orchestra.run.smithi117.stdout: "proxy": { 2024-03-19T03:42:12.902 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "proxy", 2024-03-19T03:42:12.902 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.902 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.902 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.903 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.903 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.903 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.903 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.903 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.903 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.903 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.903 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.903 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.903 INFO:teuthology.orchestra.run.smithi117.stdout: "url": { 2024-03-19T03:42:12.903 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "url", 2024-03-19T03:42:12.903 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.903 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.904 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.904 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-19T03:42:12.904 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.904 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.904 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.904 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.904 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.904 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.904 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.904 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.904 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.904 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.904 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.904 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "test_orchestrator", 2024-03-19T03:42:12.905 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.905 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.905 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.905 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.905 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.905 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.905 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.905 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.905 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.905 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.905 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.905 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.905 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.905 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.906 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.906 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.906 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.906 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.906 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.906 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.906 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.906 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.906 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.906 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.906 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.906 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.907 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.907 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.907 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.907 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.907 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.907 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.907 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.907 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.907 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.907 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.907 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.907 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.907 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.908 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.908 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.908 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.908 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.908 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.908 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.908 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.908 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.908 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.908 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.908 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.908 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.908 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.909 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.909 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.909 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.909 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.909 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.909 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.909 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.909 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.909 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.909 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.909 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.909 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.909 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.909 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.910 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.910 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.910 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.910 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.910 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.910 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.910 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.910 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.910 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.910 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.910 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "volumes", 2024-03-19T03:42:12.910 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.910 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.911 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.911 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.911 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.911 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.911 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.911 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.911 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.911 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.911 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.911 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.911 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.911 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.911 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.911 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.911 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.911 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.911 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.912 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.912 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.912 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.912 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.912 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.912 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.912 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.912 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.912 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.912 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.912 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.912 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.912 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.912 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.912 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.913 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.913 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.913 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.913 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.913 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.913 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.913 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.913 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.913 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.913 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.913 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.913 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.913 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.913 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.913 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.913 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.914 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.914 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.914 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.914 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.914 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.914 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.914 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.914 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.914 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.914 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.914 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.914 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.914 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.915 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.915 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.915 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.915 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.915 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.915 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.915 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.915 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.915 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.915 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.915 INFO:teuthology.orchestra.run.smithi117.stdout: "max_concurrent_clones": { 2024-03-19T03:42:12.915 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "max_concurrent_clones", 2024-03-19T03:42:12.915 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.915 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.915 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.915 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "4", 2024-03-19T03:42:12.916 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.916 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.916 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.916 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-19T03:42:12.916 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.916 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.916 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.916 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.916 INFO:teuthology.orchestra.run.smithi117.stdout: "periodic_async_work": { 2024-03-19T03:42:12.916 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "periodic_async_work", 2024-03-19T03:42:12.916 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.916 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.916 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.916 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.916 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.916 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.917 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.917 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Periodically check for async work", 2024-03-19T03:42:12.917 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.917 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.917 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.917 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.917 INFO:teuthology.orchestra.run.smithi117.stdout: "snapshot_clone_delay": { 2024-03-19T03:42:12.917 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "snapshot_clone_delay", 2024-03-19T03:42:12.917 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.917 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.917 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.917 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "0", 2024-03-19T03:42:12.917 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.917 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.918 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.918 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-19T03:42:12.918 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.918 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.918 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.918 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.918 INFO:teuthology.orchestra.run.smithi117.stdout: "snapshot_clone_no_wait": { 2024-03-19T03:42:12.918 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "snapshot_clone_no_wait", 2024-03-19T03:42:12.918 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.918 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.918 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.918 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.918 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.918 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.919 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.919 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-19T03:42:12.919 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.919 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.919 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.919 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.919 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.919 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.919 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.919 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "zabbix", 2024-03-19T03:42:12.919 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.919 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.919 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.919 INFO:teuthology.orchestra.run.smithi117.stdout: "discovery_interval": { 2024-03-19T03:42:12.920 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "discovery_interval", 2024-03-19T03:42:12.920 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "uint", 2024-03-19T03:42:12.920 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.920 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.920 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "100", 2024-03-19T03:42:12.920 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.920 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.920 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.920 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.920 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.920 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.920 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.920 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.920 INFO:teuthology.orchestra.run.smithi117.stdout: "identifier": { 2024-03-19T03:42:12.921 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "identifier", 2024-03-19T03:42:12.921 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.921 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.921 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.921 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.921 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.921 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.921 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.921 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.921 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.921 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.921 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.921 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.921 INFO:teuthology.orchestra.run.smithi117.stdout: "interval": { 2024-03-19T03:42:12.922 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "interval", 2024-03-19T03:42:12.922 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.922 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.922 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.922 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "60", 2024-03-19T03:42:12.922 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.922 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.922 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.922 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.922 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.922 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.922 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.922 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.922 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.923 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.923 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.923 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.923 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.923 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.923 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.923 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.923 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.923 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.923 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.923 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.923 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.923 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.923 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.923 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.924 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.924 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.924 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.924 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.924 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.924 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.924 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.924 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.924 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.924 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.924 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.924 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.924 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.924 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.924 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.925 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.925 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.925 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.925 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.925 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.925 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.925 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.925 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.925 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.925 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.925 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.925 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.925 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.925 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.926 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.926 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.926 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.926 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.926 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.926 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.926 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.926 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.926 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.926 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.926 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.926 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.926 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.926 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.926 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.927 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.927 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.927 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.927 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.927 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.927 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.927 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.927 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.927 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.927 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.927 INFO:teuthology.orchestra.run.smithi117.stdout: "zabbix_host": { 2024-03-19T03:42:12.927 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "zabbix_host", 2024-03-19T03:42:12.927 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.927 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.928 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.928 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.928 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.928 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.928 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.928 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.928 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.928 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.928 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.928 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.928 INFO:teuthology.orchestra.run.smithi117.stdout: "zabbix_port": { 2024-03-19T03:42:12.928 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "zabbix_port", 2024-03-19T03:42:12.928 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.928 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.928 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.929 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10051", 2024-03-19T03:42:12.929 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.929 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.929 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.929 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.929 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.929 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.929 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.929 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.929 INFO:teuthology.orchestra.run.smithi117.stdout: "zabbix_sender": { 2024-03-19T03:42:12.929 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "zabbix_sender", 2024-03-19T03:42:12.929 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.929 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.929 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.929 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-19T03:42:12.930 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.930 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.930 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.930 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.930 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.930 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.930 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.930 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.930 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.930 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.930 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-03-19T03:42:12.930 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.930 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.930 INFO:teuthology.orchestra.run.smithi117.stdout: "gid": 24113, 2024-03-19T03:42:12.931 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "y", 2024-03-19T03:42:12.931 INFO:teuthology.orchestra.run.smithi117.stdout: "mgr_features": 4540701547738038271, 2024-03-19T03:42:12.931 INFO:teuthology.orchestra.run.smithi117.stdout: "available_modules": [ 2024-03-19T03:42:12.931 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.931 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "alerts", 2024-03-19T03:42:12.931 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.931 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.931 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.931 INFO:teuthology.orchestra.run.smithi117.stdout: "interval": { 2024-03-19T03:42:12.931 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "interval", 2024-03-19T03:42:12.931 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.931 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.931 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.932 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "60", 2024-03-19T03:42:12.932 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.932 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.932 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.932 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "How frequently to reexamine health status", 2024-03-19T03:42:12.932 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.932 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.932 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.932 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.932 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.932 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.932 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.932 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.932 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.932 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.933 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.933 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.933 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.933 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.933 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.933 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.933 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.933 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.933 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.933 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.933 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.933 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.933 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.933 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.934 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.934 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.934 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.934 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.934 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.934 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.934 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.934 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.934 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.934 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.934 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.934 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.934 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.934 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.935 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.935 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.935 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.935 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.935 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.935 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.935 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.935 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.935 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.935 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.935 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.935 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.935 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.935 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.936 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.936 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.936 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.936 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.936 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.936 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.936 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.936 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.936 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.936 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.936 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.936 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.936 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.936 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.937 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.937 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.937 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.937 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.937 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.937 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.937 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.937 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.937 INFO:teuthology.orchestra.run.smithi117.stdout: "smtp_destination": { 2024-03-19T03:42:12.937 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "smtp_destination", 2024-03-19T03:42:12.937 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.937 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.937 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.937 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.937 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.938 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.938 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.938 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Email address to send alerts to", 2024-03-19T03:42:12.938 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.938 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.938 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.938 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.938 INFO:teuthology.orchestra.run.smithi117.stdout: "smtp_from_name": { 2024-03-19T03:42:12.938 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "smtp_from_name", 2024-03-19T03:42:12.938 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.938 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.938 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.938 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "Ceph", 2024-03-19T03:42:12.938 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.939 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.939 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.939 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Email From: name", 2024-03-19T03:42:12.939 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.939 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.939 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.939 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.939 INFO:teuthology.orchestra.run.smithi117.stdout: "smtp_host": { 2024-03-19T03:42:12.939 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "smtp_host", 2024-03-19T03:42:12.939 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.939 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.939 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.939 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.939 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.939 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.940 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.940 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "SMTP server", 2024-03-19T03:42:12.940 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.940 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.940 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.940 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.940 INFO:teuthology.orchestra.run.smithi117.stdout: "smtp_password": { 2024-03-19T03:42:12.940 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "smtp_password", 2024-03-19T03:42:12.940 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.940 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.940 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.940 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.940 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.940 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.941 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.941 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Password to authenticate with", 2024-03-19T03:42:12.941 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.941 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.941 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.941 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.941 INFO:teuthology.orchestra.run.smithi117.stdout: "smtp_port": { 2024-03-19T03:42:12.941 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "smtp_port", 2024-03-19T03:42:12.941 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.941 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.941 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.941 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "465", 2024-03-19T03:42:12.941 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.941 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.941 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.942 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "SMTP port", 2024-03-19T03:42:12.942 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.942 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.942 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.942 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.942 INFO:teuthology.orchestra.run.smithi117.stdout: "smtp_sender": { 2024-03-19T03:42:12.942 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "smtp_sender", 2024-03-19T03:42:12.942 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.942 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.942 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.942 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.942 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.942 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.943 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.943 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "SMTP envelope sender", 2024-03-19T03:42:12.943 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.943 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.943 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.943 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.943 INFO:teuthology.orchestra.run.smithi117.stdout: "smtp_ssl": { 2024-03-19T03:42:12.943 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "smtp_ssl", 2024-03-19T03:42:12.943 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.943 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.943 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.943 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.943 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.943 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.943 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.943 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-19T03:42:12.944 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.944 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.944 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.944 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.944 INFO:teuthology.orchestra.run.smithi117.stdout: "smtp_user": { 2024-03-19T03:42:12.944 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "smtp_user", 2024-03-19T03:42:12.944 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.944 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.944 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.944 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.944 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.944 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.945 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.945 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "User to authenticate as", 2024-03-19T03:42:12.945 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.945 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.945 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.945 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.945 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.945 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.945 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.945 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "balancer", 2024-03-19T03:42:12.945 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.945 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.945 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.945 INFO:teuthology.orchestra.run.smithi117.stdout: "active": { 2024-03-19T03:42:12.946 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "active", 2024-03-19T03:42:12.946 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.946 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.946 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.946 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.946 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.946 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.946 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.946 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "automatically balance PGs across cluster", 2024-03-19T03:42:12.946 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.946 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.946 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.946 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.946 INFO:teuthology.orchestra.run.smithi117.stdout: "begin_time": { 2024-03-19T03:42:12.947 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "begin_time", 2024-03-19T03:42:12.947 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.947 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.947 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.947 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "0000", 2024-03-19T03:42:12.947 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.947 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.947 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.947 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "beginning time of day to automatically balance", 2024-03-19T03:42:12.947 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-19T03:42:12.947 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.947 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.947 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.947 INFO:teuthology.orchestra.run.smithi117.stdout: "begin_weekday": { 2024-03-19T03:42:12.948 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "begin_weekday", 2024-03-19T03:42:12.948 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "uint", 2024-03-19T03:42:12.948 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.948 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.948 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "0", 2024-03-19T03:42:12.948 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "0", 2024-03-19T03:42:12.948 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "6", 2024-03-19T03:42:12.948 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.948 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-19T03:42:12.948 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-19T03:42:12.948 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.948 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.948 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.949 INFO:teuthology.orchestra.run.smithi117.stdout: "crush_compat_max_iterations": { 2024-03-19T03:42:12.949 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "crush_compat_max_iterations", 2024-03-19T03:42:12.949 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "uint", 2024-03-19T03:42:12.949 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.949 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.949 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "25", 2024-03-19T03:42:12.949 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "1", 2024-03-19T03:42:12.949 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "250", 2024-03-19T03:42:12.949 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.949 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-19T03:42:12.949 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.949 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.949 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.949 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.949 INFO:teuthology.orchestra.run.smithi117.stdout: "crush_compat_metrics": { 2024-03-19T03:42:12.950 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "crush_compat_metrics", 2024-03-19T03:42:12.950 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.950 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.950 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.950 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "pgs,objects,bytes", 2024-03-19T03:42:12.950 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.950 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.950 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.950 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-19T03:42:12.950 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-19T03:42:12.950 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.950 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.950 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.950 INFO:teuthology.orchestra.run.smithi117.stdout: "crush_compat_step": { 2024-03-19T03:42:12.951 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "crush_compat_step", 2024-03-19T03:42:12.951 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "float", 2024-03-19T03:42:12.951 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.951 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.951 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "0.5", 2024-03-19T03:42:12.951 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "0.001", 2024-03-19T03:42:12.951 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "0.999", 2024-03-19T03:42:12.951 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.951 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "aggressiveness of optimization", 2024-03-19T03:42:12.951 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-19T03:42:12.951 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.951 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.951 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.951 INFO:teuthology.orchestra.run.smithi117.stdout: "end_time": { 2024-03-19T03:42:12.951 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "end_time", 2024-03-19T03:42:12.952 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.952 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.952 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.952 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "2359", 2024-03-19T03:42:12.952 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.952 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.952 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.952 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "ending time of day to automatically balance", 2024-03-19T03:42:12.952 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-19T03:42:12.952 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.952 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.952 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.952 INFO:teuthology.orchestra.run.smithi117.stdout: "end_weekday": { 2024-03-19T03:42:12.952 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "end_weekday", 2024-03-19T03:42:12.952 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "uint", 2024-03-19T03:42:12.952 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.953 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.953 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "0", 2024-03-19T03:42:12.953 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "0", 2024-03-19T03:42:12.953 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "6", 2024-03-19T03:42:12.953 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.953 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-19T03:42:12.953 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-19T03:42:12.953 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.953 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.953 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.953 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.953 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.953 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.953 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.953 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.953 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.953 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.954 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.954 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.954 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.954 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.954 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.954 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.954 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.954 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.954 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.954 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.954 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.954 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.954 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.955 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.955 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:12.955 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:12.955 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.955 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.955 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.955 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.955 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.955 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.955 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.955 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.955 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.955 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.955 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.955 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.956 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:12.956 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:12.956 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.956 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.956 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.956 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:12.956 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.956 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.956 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.956 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:12.956 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:12.956 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:12.956 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:12.956 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:12.956 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:12.956 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.957 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.957 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.957 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.957 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.957 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.957 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:12.957 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:12.957 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.957 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.957 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.957 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.957 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.957 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.957 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.958 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:12.958 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.958 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.958 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.958 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.958 INFO:teuthology.orchestra.run.smithi117.stdout: "min_score": { 2024-03-19T03:42:12.958 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "min_score", 2024-03-19T03:42:12.958 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "float", 2024-03-19T03:42:12.958 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.958 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.958 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "0", 2024-03-19T03:42:12.958 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.958 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.959 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.959 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-19T03:42:12.959 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.959 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.959 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.959 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.959 INFO:teuthology.orchestra.run.smithi117.stdout: "mode": { 2024-03-19T03:42:12.959 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "mode", 2024-03-19T03:42:12.959 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.959 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.959 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.959 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "upmap", 2024-03-19T03:42:12.960 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.960 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.960 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.960 INFO:teuthology.orchestra.run.smithi117.stdout: "crush-compat", 2024-03-19T03:42:12.960 INFO:teuthology.orchestra.run.smithi117.stdout: "none", 2024-03-19T03:42:12.960 INFO:teuthology.orchestra.run.smithi117.stdout: "read", 2024-03-19T03:42:12.960 INFO:teuthology.orchestra.run.smithi117.stdout: "upmap", 2024-03-19T03:42:12.960 INFO:teuthology.orchestra.run.smithi117.stdout: "upmap-read" 2024-03-19T03:42:12.960 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.960 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Balancer mode", 2024-03-19T03:42:12.960 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.960 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.960 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.960 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.961 INFO:teuthology.orchestra.run.smithi117.stdout: "pool_ids": { 2024-03-19T03:42:12.961 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "pool_ids", 2024-03-19T03:42:12.961 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.961 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.961 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.961 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.961 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.961 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.961 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.961 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-19T03:42:12.961 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.961 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.961 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.961 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.962 INFO:teuthology.orchestra.run.smithi117.stdout: "sleep_interval": { 2024-03-19T03:42:12.962 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "sleep_interval", 2024-03-19T03:42:12.962 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.962 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.962 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.962 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "60", 2024-03-19T03:42:12.962 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.962 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.962 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.962 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-19T03:42:12.962 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.962 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.962 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.962 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.963 INFO:teuthology.orchestra.run.smithi117.stdout: "upmap_max_deviation": { 2024-03-19T03:42:12.963 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "upmap_max_deviation", 2024-03-19T03:42:12.963 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.963 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.963 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.963 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "5", 2024-03-19T03:42:12.963 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "1", 2024-03-19T03:42:12.963 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.963 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.963 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-19T03:42:12.963 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-19T03:42:12.963 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.963 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.963 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.963 INFO:teuthology.orchestra.run.smithi117.stdout: "upmap_max_optimizations": { 2024-03-19T03:42:12.964 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "upmap_max_optimizations", 2024-03-19T03:42:12.964 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "uint", 2024-03-19T03:42:12.964 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.964 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.964 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10", 2024-03-19T03:42:12.964 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.964 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.964 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.964 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-19T03:42:12.964 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.964 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.964 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.964 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.964 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:12.965 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.965 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:12.965 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "cephadm", 2024-03-19T03:42:12.965 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:12.965 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:12.965 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:12.965 INFO:teuthology.orchestra.run.smithi117.stdout: "agent_down_multiplier": { 2024-03-19T03:42:12.965 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "agent_down_multiplier", 2024-03-19T03:42:12.965 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "float", 2024-03-19T03:42:12.965 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.965 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.965 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "3.0", 2024-03-19T03:42:12.965 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.965 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.965 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.966 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-19T03:42:12.966 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.966 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.966 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.966 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.966 INFO:teuthology.orchestra.run.smithi117.stdout: "agent_refresh_rate": { 2024-03-19T03:42:12.966 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "agent_refresh_rate", 2024-03-19T03:42:12.966 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.966 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.966 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.966 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "20", 2024-03-19T03:42:12.966 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.966 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.967 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.967 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-19T03:42:12.967 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.967 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.967 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.967 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.967 INFO:teuthology.orchestra.run.smithi117.stdout: "agent_starting_port": { 2024-03-19T03:42:12.967 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "agent_starting_port", 2024-03-19T03:42:12.967 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.967 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.967 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.967 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "4721", 2024-03-19T03:42:12.967 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.967 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.967 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.968 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-19T03:42:12.968 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.968 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.968 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.968 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.968 INFO:teuthology.orchestra.run.smithi117.stdout: "allow_ptrace": { 2024-03-19T03:42:12.968 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "allow_ptrace", 2024-03-19T03:42:12.968 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.968 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.968 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.968 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.968 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.968 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.968 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.968 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-19T03:42:12.969 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-19T03:42:12.969 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.969 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.969 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.969 INFO:teuthology.orchestra.run.smithi117.stdout: "autotune_interval": { 2024-03-19T03:42:12.969 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "autotune_interval", 2024-03-19T03:42:12.969 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.969 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.969 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.969 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "600", 2024-03-19T03:42:12.969 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.969 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.969 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.970 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-19T03:42:12.970 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.970 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.970 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.970 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.970 INFO:teuthology.orchestra.run.smithi117.stdout: "autotune_memory_target_ratio": { 2024-03-19T03:42:12.970 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "autotune_memory_target_ratio", 2024-03-19T03:42:12.970 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "float", 2024-03-19T03:42:12.970 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.970 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.970 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "0.7", 2024-03-19T03:42:12.970 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.970 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.970 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.971 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-19T03:42:12.971 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.971 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.971 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.971 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.971 INFO:teuthology.orchestra.run.smithi117.stdout: "cephadm_log_destination": { 2024-03-19T03:42:12.971 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "cephadm_log_destination", 2024-03-19T03:42:12.971 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.971 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.971 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.971 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.971 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.971 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.971 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:12.972 INFO:teuthology.orchestra.run.smithi117.stdout: "file", 2024-03-19T03:42:12.972 INFO:teuthology.orchestra.run.smithi117.stdout: "file,syslog", 2024-03-19T03:42:12.972 INFO:teuthology.orchestra.run.smithi117.stdout: "syslog" 2024-03-19T03:42:12.972 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:12.972 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-19T03:42:12.972 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.972 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.972 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.972 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.972 INFO:teuthology.orchestra.run.smithi117.stdout: "cgroups_split": { 2024-03-19T03:42:12.972 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "cgroups_split", 2024-03-19T03:42:12.972 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.972 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.972 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.972 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.973 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.973 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.973 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.973 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-19T03:42:12.973 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.973 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.973 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.973 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.973 INFO:teuthology.orchestra.run.smithi117.stdout: "config_checks_enabled": { 2024-03-19T03:42:12.973 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "config_checks_enabled", 2024-03-19T03:42:12.973 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.973 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.973 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.973 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.973 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.974 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.974 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.974 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-19T03:42:12.974 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.974 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.974 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.974 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.974 INFO:teuthology.orchestra.run.smithi117.stdout: "config_dashboard": { 2024-03-19T03:42:12.974 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "config_dashboard", 2024-03-19T03:42:12.974 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.974 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.974 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.974 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.974 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.975 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.975 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.975 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-19T03:42:12.975 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.975 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.975 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.975 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.975 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_alertmanager": { 2024-03-19T03:42:12.975 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_alertmanager", 2024-03-19T03:42:12.975 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.975 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.975 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.975 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-19T03:42:12.975 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.975 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.976 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.976 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Prometheus container image", 2024-03-19T03:42:12.976 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.976 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.976 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.976 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.976 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_base": { 2024-03-19T03:42:12.976 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_base", 2024-03-19T03:42:12.976 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.976 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.976 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.976 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-19T03:42:12.976 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.976 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.977 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.977 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Container image name, without the tag", 2024-03-19T03:42:12.977 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.977 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.977 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.977 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.977 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_elasticsearch": { 2024-03-19T03:42:12.977 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_elasticsearch", 2024-03-19T03:42:12.977 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.977 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.977 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.977 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-19T03:42:12.977 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.977 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.977 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.977 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "elasticsearch container image", 2024-03-19T03:42:12.978 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.978 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.978 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.978 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.978 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_grafana": { 2024-03-19T03:42:12.978 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_grafana", 2024-03-19T03:42:12.978 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.978 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.978 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.978 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-19T03:42:12.978 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.978 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.978 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.978 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Prometheus container image", 2024-03-19T03:42:12.978 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.979 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.979 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.979 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.979 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_haproxy": { 2024-03-19T03:42:12.979 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_haproxy", 2024-03-19T03:42:12.979 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.979 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.979 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.979 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-19T03:42:12.979 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.979 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.979 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.979 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "HAproxy container image", 2024-03-19T03:42:12.979 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.979 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.980 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.980 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.980 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_jaeger_agent": { 2024-03-19T03:42:12.980 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_jaeger_agent", 2024-03-19T03:42:12.980 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.980 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.980 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.980 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-19T03:42:12.980 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.980 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.980 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.980 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Jaeger agent container image", 2024-03-19T03:42:12.980 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.981 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.981 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.981 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.981 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_jaeger_collector": { 2024-03-19T03:42:12.981 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_jaeger_collector", 2024-03-19T03:42:12.981 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.981 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.981 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.981 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-19T03:42:12.981 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.981 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.981 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.981 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Jaeger collector container image", 2024-03-19T03:42:12.981 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.981 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.982 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.982 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.982 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_jaeger_query": { 2024-03-19T03:42:12.982 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_jaeger_query", 2024-03-19T03:42:12.982 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.982 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.982 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.982 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-19T03:42:12.982 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.982 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.982 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.982 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Jaeger query container image", 2024-03-19T03:42:12.982 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.982 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.982 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.983 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.983 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_keepalived": { 2024-03-19T03:42:12.983 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_keepalived", 2024-03-19T03:42:12.983 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.983 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.983 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.983 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-19T03:42:12.983 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.983 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.983 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.983 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Keepalived container image", 2024-03-19T03:42:12.983 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.983 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.983 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.983 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.984 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_loki": { 2024-03-19T03:42:12.984 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_loki", 2024-03-19T03:42:12.984 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.984 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.984 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.984 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-19T03:42:12.984 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.984 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.984 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.984 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Loki container image", 2024-03-19T03:42:12.984 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.984 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.984 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.984 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.984 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_node_exporter": { 2024-03-19T03:42:12.985 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_node_exporter", 2024-03-19T03:42:12.985 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.985 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.985 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.985 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-19T03:42:12.985 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.985 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.985 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.985 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Prometheus container image", 2024-03-19T03:42:12.985 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.985 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.985 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.985 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.985 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_nvmeof": { 2024-03-19T03:42:12.986 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_nvmeof", 2024-03-19T03:42:12.986 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.986 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.986 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.986 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-19T03:42:12.986 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.986 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.986 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.986 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Nvme-of container image", 2024-03-19T03:42:12.986 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.986 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.986 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.986 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.986 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_prometheus": { 2024-03-19T03:42:12.986 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_prometheus", 2024-03-19T03:42:12.987 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.987 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.987 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.987 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-19T03:42:12.987 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.987 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.987 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.987 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Prometheus container image", 2024-03-19T03:42:12.987 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.987 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.987 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.987 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.987 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_promtail": { 2024-03-19T03:42:12.987 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_promtail", 2024-03-19T03:42:12.987 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.988 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.988 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.988 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-19T03:42:12.988 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.988 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.988 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.988 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Promtail container image", 2024-03-19T03:42:12.988 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.988 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.988 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.988 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.988 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_snmp_gateway": { 2024-03-19T03:42:12.988 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_snmp_gateway", 2024-03-19T03:42:12.988 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.988 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.989 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.989 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-19T03:42:12.989 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.989 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.989 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.989 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "SNMP Gateway container image", 2024-03-19T03:42:12.989 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.989 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.989 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.989 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.989 INFO:teuthology.orchestra.run.smithi117.stdout: "container_init": { 2024-03-19T03:42:12.989 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_init", 2024-03-19T03:42:12.990 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.990 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.990 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.990 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:12.990 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.990 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.990 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.990 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Run podman/docker with `--init`", 2024-03-19T03:42:12.990 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.990 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.990 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.990 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.990 INFO:teuthology.orchestra.run.smithi117.stdout: "daemon_cache_timeout": { 2024-03-19T03:42:12.990 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "daemon_cache_timeout", 2024-03-19T03:42:12.991 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.991 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.991 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.991 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "600", 2024-03-19T03:42:12.991 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.991 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.991 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.991 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-19T03:42:12.991 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.991 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.991 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.991 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.991 INFO:teuthology.orchestra.run.smithi117.stdout: "default_cephadm_command_timeout": { 2024-03-19T03:42:12.991 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "default_cephadm_command_timeout", 2024-03-19T03:42:12.991 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:12.992 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.992 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.992 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "900", 2024-03-19T03:42:12.992 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.992 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.992 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.992 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-19T03:42:12.992 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.992 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.992 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.992 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.992 INFO:teuthology.orchestra.run.smithi117.stdout: "default_registry": { 2024-03-19T03:42:12.992 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "default_registry", 2024-03-19T03:42:12.992 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.992 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.993 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.993 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "docker.io", 2024-03-19T03:42:12.993 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.993 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.993 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.993 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-19T03:42:12.993 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.993 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.993 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.993 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.993 INFO:teuthology.orchestra.run.smithi117.stdout: "device_cache_timeout": { 2024-03-19T03:42:12.993 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "device_cache_timeout", 2024-03-19T03:42:12.993 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.993 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.993 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.993 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "1800", 2024-03-19T03:42:12.994 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.994 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.994 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.994 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "seconds to cache device inventory", 2024-03-19T03:42:12.994 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.994 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.994 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.994 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.994 INFO:teuthology.orchestra.run.smithi117.stdout: "device_enhanced_scan": { 2024-03-19T03:42:12.994 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "device_enhanced_scan", 2024-03-19T03:42:12.994 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.994 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.994 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.994 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.994 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.995 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.995 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.995 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-19T03:42:12.995 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.995 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.995 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.995 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.995 INFO:teuthology.orchestra.run.smithi117.stdout: "facts_cache_timeout": { 2024-03-19T03:42:12.995 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "facts_cache_timeout", 2024-03-19T03:42:12.995 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.995 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.995 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.995 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "60", 2024-03-19T03:42:12.995 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.995 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.996 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.996 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "seconds to cache host facts data", 2024-03-19T03:42:12.996 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.996 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.996 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.996 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.996 INFO:teuthology.orchestra.run.smithi117.stdout: "host_check_interval": { 2024-03-19T03:42:12.996 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "host_check_interval", 2024-03-19T03:42:12.996 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:12.996 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.996 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.996 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "600", 2024-03-19T03:42:12.996 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.996 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.996 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.997 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "how frequently to perform a host check", 2024-03-19T03:42:12.997 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.997 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.997 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.997 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.997 INFO:teuthology.orchestra.run.smithi117.stdout: "hw_monitoring": { 2024-03-19T03:42:12.997 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "hw_monitoring", 2024-03-19T03:42:12.997 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.997 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.997 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.997 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.997 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.997 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.997 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.998 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-19T03:42:12.998 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.998 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.998 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.998 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.998 INFO:teuthology.orchestra.run.smithi117.stdout: "inventory_list_all": { 2024-03-19T03:42:12.998 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "inventory_list_all", 2024-03-19T03:42:12.998 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:12.998 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.998 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:12.998 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:12.998 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.998 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.998 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:12.999 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-19T03:42:12.999 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:12.999 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:12.999 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:12.999 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:12.999 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:12.999 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:12.999 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:12.999 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:12.999 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:12.999 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:12.999 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:12.999 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:12.999 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.000 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.000 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.000 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.000 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.000 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.000 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.000 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.000 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.000 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.000 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.000 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.000 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.000 INFO:teuthology.orchestra.run.smithi117.stdout: "log_refresh_metadata": { 2024-03-19T03:42:13.000 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_refresh_metadata", 2024-03-19T03:42:13.000 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.001 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.001 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.001 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.001 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.001 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.001 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.001 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-19T03:42:13.001 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.001 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.001 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.001 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.001 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.001 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.001 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.002 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.002 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.002 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.002 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.002 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.002 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.002 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-19T03:42:13.002 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.002 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.002 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.002 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.002 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.003 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.003 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.003 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.003 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.003 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.003 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.003 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.003 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.003 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.003 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.004 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.004 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.004 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.004 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.004 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.004 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.004 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.004 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.004 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.004 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.004 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.004 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.004 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.004 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.005 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.005 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.005 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.005 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.005 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.005 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.005 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.005 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.005 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.005 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.005 INFO:teuthology.orchestra.run.smithi117.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-19T03:42:13.005 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-19T03:42:13.005 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.005 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.005 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.006 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.006 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.006 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.006 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.006 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-19T03:42:13.006 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.006 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.006 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.006 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.006 INFO:teuthology.orchestra.run.smithi117.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-19T03:42:13.006 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-19T03:42:13.006 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.006 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.006 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.006 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "*", 2024-03-19T03:42:13.007 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.007 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.007 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.007 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-19T03:42:13.007 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.007 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.007 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.007 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.007 INFO:teuthology.orchestra.run.smithi117.stdout: "max_count_per_host": { 2024-03-19T03:42:13.007 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "max_count_per_host", 2024-03-19T03:42:13.007 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.007 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.007 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.007 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10", 2024-03-19T03:42:13.008 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.008 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.008 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.008 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "max number of daemons per service per host", 2024-03-19T03:42:13.008 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.008 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.008 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.008 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.008 INFO:teuthology.orchestra.run.smithi117.stdout: "max_osd_draining_count": { 2024-03-19T03:42:13.008 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "max_osd_draining_count", 2024-03-19T03:42:13.008 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.008 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.008 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.008 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10", 2024-03-19T03:42:13.008 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.009 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.009 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.009 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-19T03:42:13.009 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.009 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.009 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.009 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.009 INFO:teuthology.orchestra.run.smithi117.stdout: "migration_current": { 2024-03-19T03:42:13.009 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "migration_current", 2024-03-19T03:42:13.009 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.009 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.009 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.009 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.009 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.009 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.010 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.010 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "internal - do not modify", 2024-03-19T03:42:13.010 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.010 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.010 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.010 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.010 INFO:teuthology.orchestra.run.smithi117.stdout: "mode": { 2024-03-19T03:42:13.010 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "mode", 2024-03-19T03:42:13.010 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.010 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.010 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.010 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "root", 2024-03-19T03:42:13.010 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.010 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.010 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.011 INFO:teuthology.orchestra.run.smithi117.stdout: "cephadm-package", 2024-03-19T03:42:13.011 INFO:teuthology.orchestra.run.smithi117.stdout: "root" 2024-03-19T03:42:13.011 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.011 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "mode for remote execution of cephadm", 2024-03-19T03:42:13.011 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.011 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.011 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.011 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.011 INFO:teuthology.orchestra.run.smithi117.stdout: "oob_default_addr": { 2024-03-19T03:42:13.011 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "oob_default_addr", 2024-03-19T03:42:13.011 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.011 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.011 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.011 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "169.254.1.1", 2024-03-19T03:42:13.011 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.012 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.012 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.012 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-19T03:42:13.012 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.012 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.012 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.012 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.012 INFO:teuthology.orchestra.run.smithi117.stdout: "prometheus_alerts_path": { 2024-03-19T03:42:13.012 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "prometheus_alerts_path", 2024-03-19T03:42:13.012 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.012 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.012 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.012 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-19T03:42:13.012 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.012 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.013 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.013 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-19T03:42:13.013 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.013 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.013 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.013 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.013 INFO:teuthology.orchestra.run.smithi117.stdout: "registry_insecure": { 2024-03-19T03:42:13.013 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "registry_insecure", 2024-03-19T03:42:13.013 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.013 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.013 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.013 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.013 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.013 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.014 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.014 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-19T03:42:13.014 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.014 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.014 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.014 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.014 INFO:teuthology.orchestra.run.smithi117.stdout: "registry_password": { 2024-03-19T03:42:13.014 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "registry_password", 2024-03-19T03:42:13.014 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.014 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.014 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.014 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.014 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.015 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.015 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.015 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-19T03:42:13.017 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.017 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.017 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.018 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.018 INFO:teuthology.orchestra.run.smithi117.stdout: "registry_url": { 2024-03-19T03:42:13.018 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "registry_url", 2024-03-19T03:42:13.018 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.018 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.018 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.018 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.018 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.018 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.018 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.018 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-19T03:42:13.018 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.018 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.019 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.019 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.019 INFO:teuthology.orchestra.run.smithi117.stdout: "registry_username": { 2024-03-19T03:42:13.019 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "registry_username", 2024-03-19T03:42:13.019 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.019 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.019 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.019 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.019 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.019 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.019 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.019 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-19T03:42:13.019 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.020 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.020 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.020 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.020 INFO:teuthology.orchestra.run.smithi117.stdout: "secure_monitoring_stack": { 2024-03-19T03:42:13.020 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "secure_monitoring_stack", 2024-03-19T03:42:13.020 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.020 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.020 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.020 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.020 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.020 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.020 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.020 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-19T03:42:13.020 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.021 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.021 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.021 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.021 INFO:teuthology.orchestra.run.smithi117.stdout: "service_discovery_port": { 2024-03-19T03:42:13.021 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "service_discovery_port", 2024-03-19T03:42:13.021 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.021 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.021 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.021 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "8765", 2024-03-19T03:42:13.021 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.021 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.021 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.021 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "cephadm service discovery port", 2024-03-19T03:42:13.021 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.022 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.022 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.022 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.022 INFO:teuthology.orchestra.run.smithi117.stdout: "ssh_config_file": { 2024-03-19T03:42:13.022 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ssh_config_file", 2024-03-19T03:42:13.022 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.022 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.022 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.022 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.022 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.022 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.022 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.022 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-19T03:42:13.022 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.023 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.023 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.023 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.023 INFO:teuthology.orchestra.run.smithi117.stdout: "use_agent": { 2024-03-19T03:42:13.023 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "use_agent", 2024-03-19T03:42:13.023 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.023 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.023 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.023 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.023 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.023 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.023 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.023 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-19T03:42:13.023 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.024 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.024 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.024 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.024 INFO:teuthology.orchestra.run.smithi117.stdout: "use_repo_digest": { 2024-03-19T03:42:13.024 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "use_repo_digest", 2024-03-19T03:42:13.024 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.024 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.024 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.024 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.024 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.024 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.024 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.024 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-19T03:42:13.024 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.025 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.025 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.025 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.025 INFO:teuthology.orchestra.run.smithi117.stdout: "warn_on_failed_host_check": { 2024-03-19T03:42:13.025 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "warn_on_failed_host_check", 2024-03-19T03:42:13.025 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.025 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.025 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.025 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.025 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.025 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.025 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.026 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "raise a health warning if the host check fails", 2024-03-19T03:42:13.026 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.026 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.026 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.026 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.026 INFO:teuthology.orchestra.run.smithi117.stdout: "warn_on_stray_daemons": { 2024-03-19T03:42:13.026 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "warn_on_stray_daemons", 2024-03-19T03:42:13.026 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.026 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.026 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.026 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.026 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.026 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.026 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.027 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-19T03:42:13.027 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.027 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.027 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.027 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.027 INFO:teuthology.orchestra.run.smithi117.stdout: "warn_on_stray_hosts": { 2024-03-19T03:42:13.027 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "warn_on_stray_hosts", 2024-03-19T03:42:13.027 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.027 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.027 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.027 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.027 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.027 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.028 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.028 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-19T03:42:13.028 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.028 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.028 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.028 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.028 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.028 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.028 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.028 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "crash", 2024-03-19T03:42:13.028 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.028 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.028 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.028 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.029 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.029 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.029 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.029 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.029 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.029 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.029 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.029 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.029 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.029 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.029 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.030 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.030 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.030 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.030 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.030 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.030 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.030 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.030 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.030 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.030 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.030 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.030 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.030 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.030 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.030 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.031 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.031 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.031 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.031 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.031 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.031 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.031 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.031 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.031 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.031 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.031 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.031 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.031 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.031 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.031 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.032 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.032 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.032 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.032 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.032 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.032 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.032 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.032 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.032 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.032 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.032 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.032 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.032 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.032 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.033 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.033 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.033 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.033 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.033 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.033 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.033 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.033 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.033 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.033 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.033 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.033 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.033 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.033 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.033 INFO:teuthology.orchestra.run.smithi117.stdout: "retain_interval": { 2024-03-19T03:42:13.034 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "retain_interval", 2024-03-19T03:42:13.034 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.034 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.034 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.034 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "31536000", 2024-03-19T03:42:13.034 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.034 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.034 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.034 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-19T03:42:13.034 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.034 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.034 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.034 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.034 INFO:teuthology.orchestra.run.smithi117.stdout: "warn_recent_interval": { 2024-03-19T03:42:13.035 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "warn_recent_interval", 2024-03-19T03:42:13.035 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.035 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.035 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.035 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "1209600", 2024-03-19T03:42:13.035 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.035 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.035 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.035 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-19T03:42:13.035 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.035 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.035 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.035 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.036 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.036 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.036 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.036 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "dashboard", 2024-03-19T03:42:13.036 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.036 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.036 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.036 INFO:teuthology.orchestra.run.smithi117.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-19T03:42:13.036 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-19T03:42:13.036 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.036 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.036 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.036 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10", 2024-03-19T03:42:13.037 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.037 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.037 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.037 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.037 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.037 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.037 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.037 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.037 INFO:teuthology.orchestra.run.smithi117.stdout: "ALERTMANAGER_API_HOST": { 2024-03-19T03:42:13.037 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-19T03:42:13.037 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.037 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.037 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.037 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.037 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.038 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.038 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.038 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.038 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.038 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.038 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.038 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.038 INFO:teuthology.orchestra.run.smithi117.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-19T03:42:13.038 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-19T03:42:13.038 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.038 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.038 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.038 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.038 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.039 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.039 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.039 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.039 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.039 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.039 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.039 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.039 INFO:teuthology.orchestra.run.smithi117.stdout: "AUDIT_API_ENABLED": { 2024-03-19T03:42:13.039 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "AUDIT_API_ENABLED", 2024-03-19T03:42:13.039 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.039 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.039 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.039 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.039 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.040 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.040 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.040 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.040 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.040 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.040 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.040 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.040 INFO:teuthology.orchestra.run.smithi117.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-19T03:42:13.040 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-19T03:42:13.040 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.040 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.040 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.040 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.041 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.041 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.041 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.041 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.041 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.041 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.041 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.041 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.041 INFO:teuthology.orchestra.run.smithi117.stdout: "ENABLE_BROWSABLE_API": { 2024-03-19T03:42:13.041 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-19T03:42:13.041 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.041 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.042 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.042 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.042 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.042 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.042 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.042 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.042 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.042 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.042 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.042 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.042 INFO:teuthology.orchestra.run.smithi117.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-19T03:42:13.042 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-19T03:42:13.042 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.042 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.043 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.043 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.043 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.043 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.043 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.043 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.043 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.043 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.043 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.043 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.043 INFO:teuthology.orchestra.run.smithi117.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-19T03:42:13.043 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-19T03:42:13.043 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.044 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.044 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.044 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.044 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.044 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.044 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.044 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.044 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.044 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.044 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.044 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.044 INFO:teuthology.orchestra.run.smithi117.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-19T03:42:13.044 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-19T03:42:13.044 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.044 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.045 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.045 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.045 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.045 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.045 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.045 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.045 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.045 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.045 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.045 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.045 INFO:teuthology.orchestra.run.smithi117.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-19T03:42:13.046 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-19T03:42:13.046 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.046 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.046 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.046 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.046 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.046 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.047 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.047 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.047 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.047 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.047 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.047 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.047 INFO:teuthology.orchestra.run.smithi117.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-19T03:42:13.047 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-19T03:42:13.047 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.047 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.047 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.047 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.048 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.048 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.048 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.048 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.048 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.048 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.048 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.048 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.048 INFO:teuthology.orchestra.run.smithi117.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-19T03:42:13.048 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-19T03:42:13.048 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.048 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.048 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.048 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.048 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.049 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.049 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.049 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.049 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.049 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.049 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.049 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.049 INFO:teuthology.orchestra.run.smithi117.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-19T03:42:13.049 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-19T03:42:13.049 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.049 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.049 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.049 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.049 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.050 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.050 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.050 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.050 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.050 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.050 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.050 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.050 INFO:teuthology.orchestra.run.smithi117.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-19T03:42:13.050 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-19T03:42:13.050 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.050 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.050 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.050 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.050 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.051 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.051 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.051 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.051 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.051 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.051 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.051 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.051 INFO:teuthology.orchestra.run.smithi117.stdout: "GRAFANA_API_PASSWORD": { 2024-03-19T03:42:13.051 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-19T03:42:13.051 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.051 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.051 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.051 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "admin", 2024-03-19T03:42:13.051 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.051 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.052 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.052 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.052 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.052 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.052 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.052 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.052 INFO:teuthology.orchestra.run.smithi117.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-19T03:42:13.052 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-19T03:42:13.052 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.052 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.052 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.052 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.052 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.052 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.052 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.052 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.053 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.053 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.053 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.053 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.053 INFO:teuthology.orchestra.run.smithi117.stdout: "GRAFANA_API_URL": { 2024-03-19T03:42:13.053 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "GRAFANA_API_URL", 2024-03-19T03:42:13.053 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.053 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.053 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.053 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.053 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.053 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.053 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.053 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.053 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.054 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.054 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.054 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.054 INFO:teuthology.orchestra.run.smithi117.stdout: "GRAFANA_API_USERNAME": { 2024-03-19T03:42:13.054 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-19T03:42:13.054 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.054 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.054 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.054 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "admin", 2024-03-19T03:42:13.054 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.054 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.054 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.054 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.054 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.055 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.055 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.055 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.055 INFO:teuthology.orchestra.run.smithi117.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-19T03:42:13.055 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-19T03:42:13.055 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.055 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.055 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.055 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.055 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.055 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.055 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.055 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.055 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.055 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.056 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.056 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.056 INFO:teuthology.orchestra.run.smithi117.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-19T03:42:13.056 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-19T03:42:13.056 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.056 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.056 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.056 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.056 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.056 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.056 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.056 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.056 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.056 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.056 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.057 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.057 INFO:teuthology.orchestra.run.smithi117.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-19T03:42:13.057 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-19T03:42:13.057 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.057 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.057 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.057 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.057 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.057 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.057 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.057 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.057 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.057 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.057 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.057 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.057 INFO:teuthology.orchestra.run.smithi117.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-19T03:42:13.058 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-19T03:42:13.058 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.058 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.058 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.058 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.058 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.058 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.058 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.058 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.058 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.058 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.058 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.058 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.059 INFO:teuthology.orchestra.run.smithi117.stdout: "MULTICLUSTER_CONFIG": { 2024-03-19T03:42:13.059 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-19T03:42:13.059 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.059 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.059 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.059 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "{}", 2024-03-19T03:42:13.059 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.059 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.059 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.059 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.059 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.059 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.060 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.060 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.060 INFO:teuthology.orchestra.run.smithi117.stdout: "PROMETHEUS_API_HOST": { 2024-03-19T03:42:13.060 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-19T03:42:13.060 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.060 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.060 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.060 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.060 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.060 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.060 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.060 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.060 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.060 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.061 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.061 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.061 INFO:teuthology.orchestra.run.smithi117.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-19T03:42:13.061 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-19T03:42:13.061 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.061 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.061 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.061 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.061 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.061 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.061 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.061 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.061 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.061 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.062 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.062 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.062 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-19T03:42:13.062 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-19T03:42:13.062 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.062 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.062 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.062 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.063 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.063 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.063 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.063 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.063 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.063 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.063 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.063 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.063 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-19T03:42:13.063 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-19T03:42:13.063 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.063 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.064 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.064 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.064 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.064 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.064 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.064 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.064 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.064 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.064 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.064 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.064 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-19T03:42:13.064 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-19T03:42:13.064 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.064 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.064 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.065 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.065 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.065 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.065 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.065 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.065 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.065 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.065 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.065 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.065 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-19T03:42:13.065 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-19T03:42:13.065 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.065 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.065 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.065 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.066 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.066 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.066 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.066 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.066 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.066 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.066 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.066 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.066 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-19T03:42:13.066 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-19T03:42:13.066 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.066 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.066 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.066 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.067 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.067 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.067 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.067 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.067 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.067 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.067 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.067 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.067 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-19T03:42:13.067 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-19T03:42:13.067 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.067 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.067 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.067 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.067 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.067 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.068 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.068 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.068 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.068 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.068 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.068 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.068 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-19T03:42:13.068 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-19T03:42:13.068 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.068 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.068 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.068 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.068 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.068 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.068 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.069 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.069 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.069 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.069 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.069 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.069 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_ENABLED": { 2024-03-19T03:42:13.069 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-19T03:42:13.069 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.069 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.069 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.069 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.069 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.069 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.069 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.069 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.070 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.070 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.070 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.070 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.070 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-19T03:42:13.070 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-19T03:42:13.070 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.070 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.070 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.070 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-19T03:42:13.070 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.070 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.070 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.070 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.071 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.071 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.071 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.071 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.071 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-19T03:42:13.071 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-19T03:42:13.071 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.071 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.071 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.071 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10", 2024-03-19T03:42:13.071 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.071 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.071 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.071 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.071 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.071 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.072 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.072 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.072 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-19T03:42:13.072 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-19T03:42:13.072 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.072 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.072 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.072 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "8", 2024-03-19T03:42:13.072 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.072 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.072 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.072 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.072 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.072 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.072 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.072 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.073 INFO:teuthology.orchestra.run.smithi117.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-19T03:42:13.073 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-19T03:42:13.073 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.073 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.073 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.073 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "45", 2024-03-19T03:42:13.073 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.073 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.073 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.073 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.073 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.073 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.073 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.073 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.073 INFO:teuthology.orchestra.run.smithi117.stdout: "RGW_API_ACCESS_KEY": { 2024-03-19T03:42:13.073 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-19T03:42:13.074 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.074 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.074 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.074 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.074 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.074 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.074 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.074 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.074 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.074 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.074 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.074 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.074 INFO:teuthology.orchestra.run.smithi117.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-19T03:42:13.074 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-19T03:42:13.075 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.075 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.075 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.075 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "admin", 2024-03-19T03:42:13.075 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.075 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.075 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.075 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.075 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.075 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.075 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.075 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.075 INFO:teuthology.orchestra.run.smithi117.stdout: "RGW_API_SECRET_KEY": { 2024-03-19T03:42:13.075 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-19T03:42:13.075 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.076 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.076 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.076 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.076 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.076 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.076 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.076 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.076 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.076 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.076 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.076 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.076 INFO:teuthology.orchestra.run.smithi117.stdout: "RGW_API_SSL_VERIFY": { 2024-03-19T03:42:13.076 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-19T03:42:13.076 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.076 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.077 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.077 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.077 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.077 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.077 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.077 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.077 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.077 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.077 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.077 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.077 INFO:teuthology.orchestra.run.smithi117.stdout: "UNSAFE_TLS_v1_2": { 2024-03-19T03:42:13.077 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-19T03:42:13.077 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.077 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.078 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.078 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.078 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.078 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.078 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.078 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.078 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.078 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.078 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.078 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.078 INFO:teuthology.orchestra.run.smithi117.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-19T03:42:13.078 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-19T03:42:13.078 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.078 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.079 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.079 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "0", 2024-03-19T03:42:13.079 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.079 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.079 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.079 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.079 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.079 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.079 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.079 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.079 INFO:teuthology.orchestra.run.smithi117.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-19T03:42:13.079 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-19T03:42:13.079 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.079 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.079 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.080 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10", 2024-03-19T03:42:13.080 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.080 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.080 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.080 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.080 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.080 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.080 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.080 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.080 INFO:teuthology.orchestra.run.smithi117.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-19T03:42:13.080 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-19T03:42:13.080 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.080 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.080 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.080 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "5", 2024-03-19T03:42:13.080 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.081 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.081 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.081 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.081 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.081 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.081 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.081 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.081 INFO:teuthology.orchestra.run.smithi117.stdout: "cross_origin_url": { 2024-03-19T03:42:13.081 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "cross_origin_url", 2024-03-19T03:42:13.081 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.081 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.081 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.081 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.081 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.081 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.081 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.081 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.082 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.082 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.082 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.082 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.082 INFO:teuthology.orchestra.run.smithi117.stdout: "crt_file": { 2024-03-19T03:42:13.082 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "crt_file", 2024-03-19T03:42:13.082 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.082 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.082 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.082 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.082 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.082 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.082 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.082 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.082 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.083 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.083 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.083 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.083 INFO:teuthology.orchestra.run.smithi117.stdout: "debug": { 2024-03-19T03:42:13.083 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "debug", 2024-03-19T03:42:13.083 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.083 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.083 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.083 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.083 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.083 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.083 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.083 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Enable/disable debug options", 2024-03-19T03:42:13.083 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.084 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.084 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.084 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.084 INFO:teuthology.orchestra.run.smithi117.stdout: "jwt_token_ttl": { 2024-03-19T03:42:13.084 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "jwt_token_ttl", 2024-03-19T03:42:13.084 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.084 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.084 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.084 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "28800", 2024-03-19T03:42:13.084 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.084 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.084 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.084 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.084 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.085 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.085 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.085 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.085 INFO:teuthology.orchestra.run.smithi117.stdout: "key_file": { 2024-03-19T03:42:13.085 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "key_file", 2024-03-19T03:42:13.085 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.085 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.085 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.085 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.085 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.085 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.085 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.085 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.085 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.085 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.086 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.086 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.086 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.086 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.086 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.086 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.086 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.086 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.086 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.086 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.086 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.086 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.086 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.086 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.087 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.087 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.087 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.087 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.087 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.087 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.087 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.087 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.087 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.087 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.087 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.087 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.087 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.087 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.087 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.087 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.088 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.088 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.088 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.088 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.088 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.088 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.088 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.088 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.088 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.088 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.088 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.088 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.089 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.089 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.089 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.089 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.089 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.089 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.089 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.089 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.089 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.089 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.089 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.089 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.089 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.089 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.090 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.090 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.090 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.090 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.090 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.090 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.090 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.090 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.090 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.090 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.090 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.090 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.090 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.090 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.091 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.091 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.091 INFO:teuthology.orchestra.run.smithi117.stdout: "motd": { 2024-03-19T03:42:13.091 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "motd", 2024-03-19T03:42:13.091 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.091 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.091 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.091 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.091 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.091 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.091 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.091 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "The message of the day", 2024-03-19T03:42:13.091 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.091 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.091 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.092 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.092 INFO:teuthology.orchestra.run.smithi117.stdout: "redirect_resolve_ip_addr": { 2024-03-19T03:42:13.092 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "redirect_resolve_ip_addr", 2024-03-19T03:42:13.092 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.092 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.092 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.092 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.092 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.092 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.092 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.092 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.092 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.092 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.092 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.092 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.093 INFO:teuthology.orchestra.run.smithi117.stdout: "server_addr": { 2024-03-19T03:42:13.093 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "server_addr", 2024-03-19T03:42:13.093 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.093 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.093 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.093 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "::", 2024-03-19T03:42:13.093 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.093 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.093 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.093 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.093 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.093 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.093 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.093 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.093 INFO:teuthology.orchestra.run.smithi117.stdout: "server_port": { 2024-03-19T03:42:13.094 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "server_port", 2024-03-19T03:42:13.094 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.094 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.094 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.094 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "8080", 2024-03-19T03:42:13.094 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.094 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.094 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.094 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.094 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.094 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.094 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.094 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.094 INFO:teuthology.orchestra.run.smithi117.stdout: "ssl": { 2024-03-19T03:42:13.095 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ssl", 2024-03-19T03:42:13.095 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.095 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.095 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.095 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.095 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.095 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.095 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.095 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.095 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.095 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.095 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.095 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.095 INFO:teuthology.orchestra.run.smithi117.stdout: "ssl_server_port": { 2024-03-19T03:42:13.096 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ssl_server_port", 2024-03-19T03:42:13.096 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.096 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.096 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.096 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "8443", 2024-03-19T03:42:13.096 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.096 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.096 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.096 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.096 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.096 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.096 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.096 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.096 INFO:teuthology.orchestra.run.smithi117.stdout: "standby_behaviour": { 2024-03-19T03:42:13.096 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "standby_behaviour", 2024-03-19T03:42:13.096 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.097 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.097 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.097 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "redirect", 2024-03-19T03:42:13.097 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.097 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.097 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.097 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.097 INFO:teuthology.orchestra.run.smithi117.stdout: "redirect" 2024-03-19T03:42:13.097 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.097 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.097 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.097 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.097 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.098 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.098 INFO:teuthology.orchestra.run.smithi117.stdout: "standby_error_status_code": { 2024-03-19T03:42:13.098 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "standby_error_status_code", 2024-03-19T03:42:13.098 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.098 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.098 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.098 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "500", 2024-03-19T03:42:13.098 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "400", 2024-03-19T03:42:13.098 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "599", 2024-03-19T03:42:13.098 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.098 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.098 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.098 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.099 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.099 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.099 INFO:teuthology.orchestra.run.smithi117.stdout: "url_prefix": { 2024-03-19T03:42:13.099 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "url_prefix", 2024-03-19T03:42:13.099 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.099 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.099 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.099 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.099 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.099 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.099 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.099 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.099 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.100 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.100 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.100 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.100 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.100 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.100 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.100 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "devicehealth", 2024-03-19T03:42:13.100 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.100 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.100 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.100 INFO:teuthology.orchestra.run.smithi117.stdout: "enable_monitoring": { 2024-03-19T03:42:13.100 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "enable_monitoring", 2024-03-19T03:42:13.100 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.100 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.101 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.101 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.101 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.101 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.101 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.101 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "monitor device health metrics", 2024-03-19T03:42:13.101 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.101 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.101 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.101 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.101 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.101 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.101 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.101 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.101 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.102 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.102 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.102 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.102 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.102 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.102 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.102 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.102 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.102 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.102 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.102 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.102 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.102 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.102 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.103 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.103 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.103 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.103 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.103 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.103 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.103 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.103 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.103 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.103 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.103 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.103 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.103 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.103 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.103 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.104 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.104 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.104 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.104 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.104 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.104 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.104 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.104 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.104 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.104 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.104 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.104 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.104 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.104 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.105 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.105 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.105 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.105 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.105 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.105 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.105 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.105 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.105 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.105 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.105 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.105 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.105 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.105 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.105 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.106 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.106 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.106 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.106 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.106 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.106 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.106 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.106 INFO:teuthology.orchestra.run.smithi117.stdout: "mark_out_threshold": { 2024-03-19T03:42:13.106 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "mark_out_threshold", 2024-03-19T03:42:13.106 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.106 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.106 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.106 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "2419200", 2024-03-19T03:42:13.106 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.107 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.107 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.107 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-19T03:42:13.107 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.107 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.107 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.107 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.107 INFO:teuthology.orchestra.run.smithi117.stdout: "pool_name": { 2024-03-19T03:42:13.107 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "pool_name", 2024-03-19T03:42:13.107 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.107 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.107 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.107 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "device_health_metrics", 2024-03-19T03:42:13.107 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.107 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.107 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.108 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-19T03:42:13.108 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.108 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.108 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.108 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.108 INFO:teuthology.orchestra.run.smithi117.stdout: "retention_period": { 2024-03-19T03:42:13.108 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "retention_period", 2024-03-19T03:42:13.108 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.108 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.108 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.108 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "15552000", 2024-03-19T03:42:13.108 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.108 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.108 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.108 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "how long to retain device health metrics", 2024-03-19T03:42:13.109 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.109 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.109 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.109 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.109 INFO:teuthology.orchestra.run.smithi117.stdout: "scrape_frequency": { 2024-03-19T03:42:13.109 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "scrape_frequency", 2024-03-19T03:42:13.109 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.109 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.109 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.109 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "86400", 2024-03-19T03:42:13.109 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.109 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.109 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.109 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-19T03:42:13.109 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.109 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.110 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.110 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.110 INFO:teuthology.orchestra.run.smithi117.stdout: "self_heal": { 2024-03-19T03:42:13.110 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "self_heal", 2024-03-19T03:42:13.110 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.110 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.110 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.110 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.110 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.110 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.110 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.110 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-19T03:42:13.111 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.111 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.111 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.111 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.111 INFO:teuthology.orchestra.run.smithi117.stdout: "sleep_interval": { 2024-03-19T03:42:13.111 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "sleep_interval", 2024-03-19T03:42:13.111 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.111 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.111 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.111 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "600", 2024-03-19T03:42:13.111 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.111 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.111 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.112 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "how frequently to wake up and check device health", 2024-03-19T03:42:13.112 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.112 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.112 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.112 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.112 INFO:teuthology.orchestra.run.smithi117.stdout: "warn_threshold": { 2024-03-19T03:42:13.112 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "warn_threshold", 2024-03-19T03:42:13.112 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.112 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.112 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.112 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "7257600", 2024-03-19T03:42:13.112 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.112 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.113 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.113 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-19T03:42:13.113 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.113 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.113 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.113 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.113 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.113 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.113 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.113 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "diskprediction_local", 2024-03-19T03:42:13.113 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.113 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.113 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.113 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.114 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.114 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.114 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.114 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.114 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.114 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.114 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.114 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.114 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.114 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.114 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.114 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.114 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.114 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.114 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.115 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.115 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.115 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.115 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.115 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.115 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.115 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.115 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.115 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.115 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.115 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.115 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.115 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.115 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.115 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.116 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.116 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.116 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.116 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.116 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.116 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.116 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.116 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.116 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.116 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.116 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.116 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.116 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.116 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.116 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.117 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.117 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.117 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.117 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.117 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.117 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.117 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.117 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.117 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.117 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.117 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.117 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.117 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.117 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.117 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.117 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.118 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.118 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.118 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.118 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.118 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.118 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.118 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.118 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.118 INFO:teuthology.orchestra.run.smithi117.stdout: "predict_interval": { 2024-03-19T03:42:13.118 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "predict_interval", 2024-03-19T03:42:13.118 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.118 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.118 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.118 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "86400", 2024-03-19T03:42:13.119 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.119 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.119 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.119 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.119 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.119 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.119 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.119 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.119 INFO:teuthology.orchestra.run.smithi117.stdout: "predictor_model": { 2024-03-19T03:42:13.119 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "predictor_model", 2024-03-19T03:42:13.119 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.119 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.119 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.119 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "prophetstor", 2024-03-19T03:42:13.120 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.120 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.120 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.120 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.120 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.120 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.120 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.120 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.120 INFO:teuthology.orchestra.run.smithi117.stdout: "sleep_interval": { 2024-03-19T03:42:13.120 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "sleep_interval", 2024-03-19T03:42:13.120 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.120 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.120 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.120 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "600", 2024-03-19T03:42:13.121 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.121 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.121 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.121 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.121 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.121 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.121 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.121 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.121 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.121 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.121 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.121 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "influx", 2024-03-19T03:42:13.121 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": false, 2024-03-19T03:42:13.121 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "influxdb python module not found", 2024-03-19T03:42:13.121 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.122 INFO:teuthology.orchestra.run.smithi117.stdout: "batch_size": { 2024-03-19T03:42:13.122 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "batch_size", 2024-03-19T03:42:13.122 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.122 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.122 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.122 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "5000", 2024-03-19T03:42:13.122 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.122 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.122 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.122 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-19T03:42:13.122 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.122 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.122 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.123 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.123 INFO:teuthology.orchestra.run.smithi117.stdout: "database": { 2024-03-19T03:42:13.123 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "database", 2024-03-19T03:42:13.123 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.123 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.123 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.123 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "ceph", 2024-03-19T03:42:13.123 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.123 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.123 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.123 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-19T03:42:13.123 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.123 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.123 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.124 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.124 INFO:teuthology.orchestra.run.smithi117.stdout: "hostname": { 2024-03-19T03:42:13.124 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "hostname", 2024-03-19T03:42:13.124 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.124 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.124 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.124 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.124 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.124 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.124 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.124 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "InfluxDB server hostname", 2024-03-19T03:42:13.124 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.124 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.124 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.125 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.125 INFO:teuthology.orchestra.run.smithi117.stdout: "interval": { 2024-03-19T03:42:13.125 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "interval", 2024-03-19T03:42:13.125 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.125 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.125 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.125 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "30", 2024-03-19T03:42:13.125 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "5", 2024-03-19T03:42:13.125 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.125 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.125 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-19T03:42:13.125 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.125 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.126 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.126 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.126 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.126 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.126 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.126 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.126 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.126 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.126 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.126 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.126 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.126 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.126 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.126 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.127 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.127 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.127 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.127 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.127 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.127 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.127 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.127 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.127 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.127 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.127 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.127 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.127 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.128 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.128 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.128 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.128 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.128 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.128 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.128 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.128 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.128 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.128 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.128 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.128 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.128 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.128 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.129 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.129 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.129 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.129 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.129 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.129 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.129 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.129 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.129 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.129 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.129 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.129 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.129 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.130 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.130 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.130 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.130 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.130 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.130 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.130 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.130 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.130 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.130 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.130 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.130 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.130 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.131 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.131 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.131 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.131 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.131 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.131 INFO:teuthology.orchestra.run.smithi117.stdout: "password": { 2024-03-19T03:42:13.131 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "password", 2024-03-19T03:42:13.131 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.131 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.131 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.131 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.131 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.131 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.131 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.132 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "password of InfluxDB server user", 2024-03-19T03:42:13.132 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.132 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.132 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.132 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.132 INFO:teuthology.orchestra.run.smithi117.stdout: "port": { 2024-03-19T03:42:13.132 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "port", 2024-03-19T03:42:13.132 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.132 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.132 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.132 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "8086", 2024-03-19T03:42:13.132 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.132 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.132 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.132 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "InfluxDB server port", 2024-03-19T03:42:13.133 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.133 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.133 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.133 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.133 INFO:teuthology.orchestra.run.smithi117.stdout: "ssl": { 2024-03-19T03:42:13.133 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ssl", 2024-03-19T03:42:13.133 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.133 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.133 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.133 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "false", 2024-03-19T03:42:13.133 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.133 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.133 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.133 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-19T03:42:13.134 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.134 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.134 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.134 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.134 INFO:teuthology.orchestra.run.smithi117.stdout: "threads": { 2024-03-19T03:42:13.134 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "threads", 2024-03-19T03:42:13.134 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.134 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.134 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.134 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "5", 2024-03-19T03:42:13.134 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "1", 2024-03-19T03:42:13.134 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "32", 2024-03-19T03:42:13.134 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.134 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-19T03:42:13.134 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.135 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.135 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.135 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.135 INFO:teuthology.orchestra.run.smithi117.stdout: "username": { 2024-03-19T03:42:13.135 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "username", 2024-03-19T03:42:13.135 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.135 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.135 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.135 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.135 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.135 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.135 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.135 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "username of InfluxDB server user", 2024-03-19T03:42:13.135 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.136 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.136 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.136 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.136 INFO:teuthology.orchestra.run.smithi117.stdout: "verify_ssl": { 2024-03-19T03:42:13.136 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "verify_ssl", 2024-03-19T03:42:13.136 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.136 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.136 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.136 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "true", 2024-03-19T03:42:13.136 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.136 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.136 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.136 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-19T03:42:13.136 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.136 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.137 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.137 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.137 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.137 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.137 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.137 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "insights", 2024-03-19T03:42:13.137 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.137 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.137 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.137 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.137 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.137 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.137 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.137 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.138 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.138 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.138 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.138 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.138 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.138 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.138 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.138 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.138 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.138 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.138 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.138 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.138 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.138 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.139 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.139 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.139 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.139 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.139 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.139 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.139 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.139 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.139 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.139 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.139 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.139 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.139 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.139 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.139 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.140 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.140 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.140 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.140 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.140 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.140 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.140 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.140 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.140 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.140 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.140 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.140 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.140 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.140 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.141 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.141 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.141 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.141 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.141 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.141 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.141 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.141 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.141 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.141 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.141 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.141 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.141 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.141 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.141 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.142 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.142 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.142 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.142 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.142 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.142 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.142 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.142 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.142 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.142 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.142 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "iostat", 2024-03-19T03:42:13.142 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.142 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.142 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.142 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.143 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.143 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.143 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.143 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.143 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.143 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.143 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.143 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.143 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.143 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.143 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.143 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.143 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.143 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.143 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.144 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.144 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.144 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.144 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.144 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.144 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.144 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.144 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.144 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.144 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.144 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.144 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.144 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.144 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.144 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.144 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.144 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.145 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.145 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.145 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.145 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.145 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.145 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.145 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.145 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.145 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.145 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.145 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.145 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.145 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.145 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.145 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.146 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.146 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.146 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.146 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.146 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.146 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.146 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.146 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.146 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.146 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.146 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.146 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.146 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.146 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.147 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.147 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.147 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.147 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.147 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.147 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.147 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.147 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.147 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.147 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.147 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.147 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "k8sevents", 2024-03-19T03:42:13.147 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.147 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.147 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.147 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_event_retention_days": { 2024-03-19T03:42:13.148 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ceph_event_retention_days", 2024-03-19T03:42:13.148 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.148 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.148 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.148 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "7", 2024-03-19T03:42:13.148 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.148 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.148 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.148 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-19T03:42:13.148 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.148 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.148 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.148 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.148 INFO:teuthology.orchestra.run.smithi117.stdout: "config_check_secs": { 2024-03-19T03:42:13.148 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "config_check_secs", 2024-03-19T03:42:13.149 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.149 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.149 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.149 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10", 2024-03-19T03:42:13.149 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "10", 2024-03-19T03:42:13.149 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.149 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.149 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-19T03:42:13.149 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.149 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.149 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.149 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.149 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.149 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.149 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.149 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.150 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.150 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.150 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.150 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.150 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.150 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.150 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.150 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.150 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.150 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.150 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.150 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.151 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.151 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.151 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.151 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.151 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.151 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.151 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.151 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.151 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.151 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.151 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.151 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.151 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.151 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.151 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.152 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.152 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.152 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.152 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.152 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.152 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.152 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.152 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.152 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.152 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.152 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.152 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.152 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.153 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.153 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.153 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.153 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.153 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.153 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.153 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.153 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.153 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.153 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.153 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.153 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.153 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.153 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.153 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.154 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.154 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.154 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.154 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.154 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.154 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.154 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.154 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.154 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.154 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.154 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.154 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.154 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.154 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.155 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "localpool", 2024-03-19T03:42:13.155 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.155 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.155 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.155 INFO:teuthology.orchestra.run.smithi117.stdout: "failure_domain": { 2024-03-19T03:42:13.155 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "failure_domain", 2024-03-19T03:42:13.155 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.155 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.155 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.155 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "host", 2024-03-19T03:42:13.155 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.155 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.155 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.155 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "failure domain for any created local pool", 2024-03-19T03:42:13.156 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-19T03:42:13.156 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.156 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.156 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.156 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.156 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.156 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.156 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.156 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.156 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.156 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.156 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.156 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.156 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.157 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.157 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.157 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.157 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.157 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.157 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.157 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.157 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.157 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.157 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.157 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.157 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.157 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.157 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.157 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.157 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.158 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.158 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.158 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.158 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.158 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.158 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.158 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.158 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.158 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.158 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.158 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.158 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.158 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.158 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.158 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.159 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.159 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.159 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.159 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.159 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.159 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.159 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.159 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.159 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.159 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.159 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.159 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.159 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.159 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.160 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.160 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.160 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.160 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.160 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.160 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.160 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.160 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.160 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.160 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.160 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.160 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.160 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.161 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.161 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.161 INFO:teuthology.orchestra.run.smithi117.stdout: "min_size": { 2024-03-19T03:42:13.161 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "min_size", 2024-03-19T03:42:13.161 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.161 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.161 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.161 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.161 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.161 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.161 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.161 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "default min_size for any created local pool", 2024-03-19T03:42:13.161 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-19T03:42:13.161 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.162 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.162 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.162 INFO:teuthology.orchestra.run.smithi117.stdout: "num_rep": { 2024-03-19T03:42:13.162 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "num_rep", 2024-03-19T03:42:13.162 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.162 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.162 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.162 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "3", 2024-03-19T03:42:13.162 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.162 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.162 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.162 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "default replica count for any created local pool", 2024-03-19T03:42:13.162 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.162 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.163 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.163 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.163 INFO:teuthology.orchestra.run.smithi117.stdout: "pg_num": { 2024-03-19T03:42:13.163 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "pg_num", 2024-03-19T03:42:13.163 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.163 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.163 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.163 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "128", 2024-03-19T03:42:13.163 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.163 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.163 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.163 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "default pg_num for any created local pool", 2024-03-19T03:42:13.163 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.164 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.164 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.164 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.164 INFO:teuthology.orchestra.run.smithi117.stdout: "prefix": { 2024-03-19T03:42:13.164 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "prefix", 2024-03-19T03:42:13.164 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.164 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.164 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.164 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.164 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.165 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.165 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.165 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "name prefix for any created local pool", 2024-03-19T03:42:13.165 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.165 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.165 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.165 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.165 INFO:teuthology.orchestra.run.smithi117.stdout: "subtree": { 2024-03-19T03:42:13.165 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "subtree", 2024-03-19T03:42:13.165 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.165 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.165 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.165 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "rack", 2024-03-19T03:42:13.165 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.165 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.165 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.166 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-19T03:42:13.166 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-19T03:42:13.166 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.166 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.166 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.166 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.166 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.166 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.166 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "mds_autoscaler", 2024-03-19T03:42:13.166 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.166 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.166 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.167 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.167 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.167 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.167 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.167 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.167 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.167 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.167 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.167 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.167 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.167 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.167 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.167 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.168 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.168 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.168 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.168 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.168 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.168 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.168 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.168 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.168 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.168 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.168 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.168 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.168 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.168 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.168 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.168 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.169 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.169 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.169 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.169 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.169 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.169 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.169 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.169 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.169 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.169 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.169 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.169 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.169 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.169 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.169 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.169 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.169 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.170 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.170 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.170 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.170 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.170 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.170 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.170 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.170 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.170 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.170 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.170 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.170 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.170 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.170 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.171 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.171 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.171 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.171 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.171 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.171 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.171 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.171 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.171 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.171 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.171 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.171 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.171 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.171 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "mirroring", 2024-03-19T03:42:13.171 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.171 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.172 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.172 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.172 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.172 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.172 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.172 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.172 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.172 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.172 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.172 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.172 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.172 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.172 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.172 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.172 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.172 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.173 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.173 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.173 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.173 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.173 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.173 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.173 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.173 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.173 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.173 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.173 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.173 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.173 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.173 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.173 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.173 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.173 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.174 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.174 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.174 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.174 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.174 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.174 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.174 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.174 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.174 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.174 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.174 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.174 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.174 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.174 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.174 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.174 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.175 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.175 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.175 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.175 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.175 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.175 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.175 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.175 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.175 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.175 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.175 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.175 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.175 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.175 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.175 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.175 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.176 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.176 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.176 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.176 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.176 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.176 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.176 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.176 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.176 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.176 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "nfs", 2024-03-19T03:42:13.176 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.176 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.176 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.177 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.177 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.177 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.177 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.177 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.177 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.177 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.177 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.177 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.177 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.177 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.177 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.178 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.178 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.178 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.178 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.178 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.178 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.178 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.178 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.178 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.178 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.178 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.178 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.178 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.178 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.179 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.179 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.179 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.179 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.179 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.179 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.179 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.179 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.179 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.179 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.179 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.179 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.180 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.180 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.180 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.180 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.180 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.180 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.180 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.180 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.180 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.180 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.180 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.180 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.180 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.181 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.181 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.181 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.181 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.181 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.181 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.181 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.181 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.181 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.181 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.182 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.182 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.182 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.182 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.182 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.182 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.182 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.182 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.182 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.182 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.182 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.182 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.183 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "orchestrator", 2024-03-19T03:42:13.183 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.183 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.183 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.183 INFO:teuthology.orchestra.run.smithi117.stdout: "fail_fs": { 2024-03-19T03:42:13.183 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "fail_fs", 2024-03-19T03:42:13.183 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.183 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.183 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.183 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.183 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.183 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.184 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.184 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-19T03:42:13.184 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.184 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.184 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.184 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.184 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.184 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.184 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.184 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.184 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.184 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.184 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.185 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.185 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.185 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.185 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.185 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.185 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.185 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.185 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.185 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.185 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.185 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.185 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.185 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.186 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.186 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.186 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.186 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.186 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.186 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.186 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.186 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.186 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.186 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.186 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.186 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.186 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.186 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.187 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.187 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.187 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.187 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.187 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.187 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.187 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.187 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.187 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.187 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.187 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.187 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.187 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.187 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.187 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.188 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.188 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.188 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.188 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.188 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.188 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.188 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.188 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.188 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.188 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.188 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.188 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.188 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.188 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.188 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.189 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.189 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.189 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.189 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.189 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.189 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.189 INFO:teuthology.orchestra.run.smithi117.stdout: "orchestrator": { 2024-03-19T03:42:13.189 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "orchestrator", 2024-03-19T03:42:13.189 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.189 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.189 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.189 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.189 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.189 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.190 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.190 INFO:teuthology.orchestra.run.smithi117.stdout: "cephadm", 2024-03-19T03:42:13.190 INFO:teuthology.orchestra.run.smithi117.stdout: "rook", 2024-03-19T03:42:13.190 INFO:teuthology.orchestra.run.smithi117.stdout: "test_orchestrator" 2024-03-19T03:42:13.190 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.190 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Orchestrator backend", 2024-03-19T03:42:13.190 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.190 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.190 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.190 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.190 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.190 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.190 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.190 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "osd_perf_query", 2024-03-19T03:42:13.191 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.191 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.191 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.191 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.191 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.191 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.191 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.191 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.191 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.191 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.191 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.191 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.191 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.191 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.192 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.192 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.192 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.192 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.192 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.192 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.192 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.192 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.192 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.192 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.192 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.192 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.192 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.192 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.192 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.193 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.193 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.193 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.193 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.193 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.193 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.193 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.193 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.193 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.193 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.193 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.193 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.193 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.193 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.194 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.194 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.194 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.194 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.194 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.194 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.194 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.194 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.194 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.194 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.194 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.194 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.194 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.195 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.195 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.195 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.195 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.195 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.195 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.195 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.195 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.195 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.195 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.195 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.195 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.195 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.195 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.196 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.196 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.196 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.196 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.196 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.196 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.196 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "osd_support", 2024-03-19T03:42:13.196 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.196 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.196 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.196 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.196 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.196 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.197 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.197 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.197 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.197 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.197 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.197 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.197 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.197 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.197 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.197 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.197 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.197 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.197 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.197 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.198 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.198 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.198 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.198 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.198 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.198 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.198 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.198 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.198 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.198 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.198 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.198 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.198 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.198 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.199 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.199 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.199 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.199 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.199 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.199 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.199 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.199 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.199 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.199 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.199 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.199 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.199 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.200 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.200 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.200 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.200 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.200 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.200 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.200 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.200 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.200 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.200 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.200 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.200 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.200 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.200 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.201 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.201 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.201 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.201 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.201 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.201 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.201 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.201 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.201 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.201 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.201 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.201 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.201 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.201 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.201 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.201 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "pg_autoscaler", 2024-03-19T03:42:13.202 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.202 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.202 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.202 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.202 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.202 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.202 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.202 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.202 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.202 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.202 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.202 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.202 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.202 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.202 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.203 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.203 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.203 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.203 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.203 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.203 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.203 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.203 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.203 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.203 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.203 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.203 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.203 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.203 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.203 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.203 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.203 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.204 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.204 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.204 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.204 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.204 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.204 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.204 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.204 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.204 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.204 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.204 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.204 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.204 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.204 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.204 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.204 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.204 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.205 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.205 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.205 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.205 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.205 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.205 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.205 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.205 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.205 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.205 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.205 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.205 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.205 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.205 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.205 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.206 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.206 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.206 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.206 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.206 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.206 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.206 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.206 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.206 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.206 INFO:teuthology.orchestra.run.smithi117.stdout: "sleep_interval": { 2024-03-19T03:42:13.206 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "sleep_interval", 2024-03-19T03:42:13.206 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.207 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.207 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.207 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "60", 2024-03-19T03:42:13.207 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.207 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.207 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.207 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.207 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.207 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.207 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.207 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.207 INFO:teuthology.orchestra.run.smithi117.stdout: "threshold": { 2024-03-19T03:42:13.207 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "threshold", 2024-03-19T03:42:13.207 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "float", 2024-03-19T03:42:13.207 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.207 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.208 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "3.0", 2024-03-19T03:42:13.208 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "1.0", 2024-03-19T03:42:13.208 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.208 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.208 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "scaling threshold", 2024-03-19T03:42:13.208 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-19T03:42:13.208 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.208 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.208 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.208 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.208 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.208 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.208 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "progress", 2024-03-19T03:42:13.208 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.208 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.209 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.209 INFO:teuthology.orchestra.run.smithi117.stdout: "allow_pg_recovery_event": { 2024-03-19T03:42:13.209 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "allow_pg_recovery_event", 2024-03-19T03:42:13.209 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.209 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.209 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.209 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.209 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.209 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.209 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.209 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-19T03:42:13.209 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.209 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.210 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.210 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.210 INFO:teuthology.orchestra.run.smithi117.stdout: "enabled": { 2024-03-19T03:42:13.210 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "enabled", 2024-03-19T03:42:13.210 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.210 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.210 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.210 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.210 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.210 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.210 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.210 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.210 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.211 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.211 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.211 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.211 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.211 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.211 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.211 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.211 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.211 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.211 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.211 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.211 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.212 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.212 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.212 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.212 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.212 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.212 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.212 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.212 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.212 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.212 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.212 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.212 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.212 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.212 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.213 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.213 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.213 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.213 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.213 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.213 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.213 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.213 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.213 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.213 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.213 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.213 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.214 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.214 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.214 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.214 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.214 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.214 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.214 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.214 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.214 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.214 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.214 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.214 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.215 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.215 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.215 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.215 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.215 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.215 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.215 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.215 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.215 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.215 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.215 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.215 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.215 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.215 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.216 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.216 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.216 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.216 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.216 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.216 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.216 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.216 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.216 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.216 INFO:teuthology.orchestra.run.smithi117.stdout: "max_completed_events": { 2024-03-19T03:42:13.216 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "max_completed_events", 2024-03-19T03:42:13.216 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.216 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.216 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.217 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "50", 2024-03-19T03:42:13.217 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.217 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.217 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.217 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "number of past completed events to remember", 2024-03-19T03:42:13.217 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.217 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.217 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.217 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.217 INFO:teuthology.orchestra.run.smithi117.stdout: "sleep_interval": { 2024-03-19T03:42:13.217 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "sleep_interval", 2024-03-19T03:42:13.217 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.217 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.217 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.217 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "5", 2024-03-19T03:42:13.218 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.218 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.218 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.218 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "how long the module is going to sleep", 2024-03-19T03:42:13.218 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.218 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.218 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.218 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.218 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.218 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.218 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.218 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "prometheus", 2024-03-19T03:42:13.218 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.218 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.219 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.219 INFO:teuthology.orchestra.run.smithi117.stdout: "cache": { 2024-03-19T03:42:13.219 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "cache", 2024-03-19T03:42:13.219 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.219 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.221 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.221 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.221 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.221 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.221 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.221 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.221 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.221 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.222 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.222 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.222 INFO:teuthology.orchestra.run.smithi117.stdout: "exclude_perf_counters": { 2024-03-19T03:42:13.222 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "exclude_perf_counters", 2024-03-19T03:42:13.222 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.222 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.222 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.222 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.222 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.222 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.222 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.222 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-19T03:42:13.222 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-19T03:42:13.222 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.223 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.223 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.223 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.223 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.223 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.223 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.223 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.223 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.223 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.223 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.223 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.223 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.223 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.223 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.223 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.223 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.224 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.224 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.224 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.224 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.224 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.224 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.224 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.224 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.224 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.224 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.224 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.224 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.224 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.224 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.224 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.225 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.225 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.225 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.225 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.225 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.225 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.225 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.225 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.225 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.225 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.225 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.225 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.225 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.225 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.225 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.225 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.226 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.226 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.226 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.226 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.226 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.226 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.226 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.226 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.226 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.226 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.226 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.226 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.226 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.226 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.227 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.227 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.227 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.227 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.227 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.227 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.227 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.227 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.227 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.227 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.227 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.227 INFO:teuthology.orchestra.run.smithi117.stdout: "rbd_stats_pools": { 2024-03-19T03:42:13.227 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rbd_stats_pools", 2024-03-19T03:42:13.227 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.227 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.227 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.228 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.228 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.228 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.228 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.228 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.228 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.228 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.228 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.228 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.228 INFO:teuthology.orchestra.run.smithi117.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-19T03:42:13.228 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-19T03:42:13.228 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.228 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.228 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.228 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "300", 2024-03-19T03:42:13.229 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.229 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.229 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.229 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.229 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.229 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.229 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.229 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.229 INFO:teuthology.orchestra.run.smithi117.stdout: "scrape_interval": { 2024-03-19T03:42:13.229 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "scrape_interval", 2024-03-19T03:42:13.229 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "float", 2024-03-19T03:42:13.229 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.229 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.229 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "15.0", 2024-03-19T03:42:13.229 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.229 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.230 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.230 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.230 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.230 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.230 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.230 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.230 INFO:teuthology.orchestra.run.smithi117.stdout: "server_addr": { 2024-03-19T03:42:13.230 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "server_addr", 2024-03-19T03:42:13.230 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.230 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.230 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.230 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "::", 2024-03-19T03:42:13.230 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.230 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.230 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.231 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-19T03:42:13.231 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.231 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.231 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.231 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.231 INFO:teuthology.orchestra.run.smithi117.stdout: "server_port": { 2024-03-19T03:42:13.231 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "server_port", 2024-03-19T03:42:13.231 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.231 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.231 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.231 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "9283", 2024-03-19T03:42:13.231 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.231 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.231 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.231 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-19T03:42:13.231 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.232 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.232 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.232 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.232 INFO:teuthology.orchestra.run.smithi117.stdout: "stale_cache_strategy": { 2024-03-19T03:42:13.232 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "stale_cache_strategy", 2024-03-19T03:42:13.232 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.232 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.232 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.232 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "log", 2024-03-19T03:42:13.232 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.232 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.232 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.232 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.232 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.232 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.232 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.233 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.233 INFO:teuthology.orchestra.run.smithi117.stdout: "standby_behaviour": { 2024-03-19T03:42:13.233 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "standby_behaviour", 2024-03-19T03:42:13.233 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.233 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.233 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.233 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "default", 2024-03-19T03:42:13.233 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.233 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.233 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.233 INFO:teuthology.orchestra.run.smithi117.stdout: "default", 2024-03-19T03:42:13.233 INFO:teuthology.orchestra.run.smithi117.stdout: "error" 2024-03-19T03:42:13.233 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.233 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.234 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.234 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.234 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.234 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.234 INFO:teuthology.orchestra.run.smithi117.stdout: "standby_error_status_code": { 2024-03-19T03:42:13.234 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "standby_error_status_code", 2024-03-19T03:42:13.234 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.234 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.234 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.234 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "500", 2024-03-19T03:42:13.234 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "400", 2024-03-19T03:42:13.234 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "599", 2024-03-19T03:42:13.234 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.234 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.235 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.235 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.235 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.235 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.235 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.235 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.235 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.235 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rbd_support", 2024-03-19T03:42:13.235 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.235 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.235 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.235 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.235 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.235 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.235 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.235 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.236 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.236 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.236 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.236 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.236 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.236 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.236 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.236 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.236 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.236 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.236 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.236 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.236 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.236 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.236 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.236 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.237 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.237 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.237 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.237 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.237 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.237 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.237 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.237 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.237 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.237 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.237 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.237 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.237 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.237 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.238 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.238 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.238 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.238 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.238 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.238 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.238 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.238 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.238 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.238 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.238 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.238 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.238 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.238 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.239 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.239 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.239 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.239 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.239 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.239 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.239 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.239 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.239 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.239 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.239 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.239 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.239 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.239 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.240 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.240 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.240 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.240 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.240 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.240 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.240 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.240 INFO:teuthology.orchestra.run.smithi117.stdout: "max_concurrent_snap_create": { 2024-03-19T03:42:13.240 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "max_concurrent_snap_create", 2024-03-19T03:42:13.240 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.240 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.240 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.240 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10", 2024-03-19T03:42:13.240 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.240 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.241 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.241 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.241 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.241 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.241 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.241 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.241 INFO:teuthology.orchestra.run.smithi117.stdout: "mirror_snapshot_schedule": { 2024-03-19T03:42:13.241 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "mirror_snapshot_schedule", 2024-03-19T03:42:13.241 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.241 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.241 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.241 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.241 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.242 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.242 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.242 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.242 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.242 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.242 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.242 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.242 INFO:teuthology.orchestra.run.smithi117.stdout: "trash_purge_schedule": { 2024-03-19T03:42:13.242 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "trash_purge_schedule", 2024-03-19T03:42:13.242 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.242 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.242 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.242 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.243 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.243 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.243 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.243 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.243 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.243 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.243 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.243 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.243 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.243 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.243 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.243 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "restful", 2024-03-19T03:42:13.243 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.243 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.243 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.244 INFO:teuthology.orchestra.run.smithi117.stdout: "enable_auth": { 2024-03-19T03:42:13.244 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "enable_auth", 2024-03-19T03:42:13.244 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.244 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.244 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.244 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.244 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.244 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.244 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.244 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.244 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.244 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.245 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.245 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.245 INFO:teuthology.orchestra.run.smithi117.stdout: "key_file": { 2024-03-19T03:42:13.245 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "key_file", 2024-03-19T03:42:13.245 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.245 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.245 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.245 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.245 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.245 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.245 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.245 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.245 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.245 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.246 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.246 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.246 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.246 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.246 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.246 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.246 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.246 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.246 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.246 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.246 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.246 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.246 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.246 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.247 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.247 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.247 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.247 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.247 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.247 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.247 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.247 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.247 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.247 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.247 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.247 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.247 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.247 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.247 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.248 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.248 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.248 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.248 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.248 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.248 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.248 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.248 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.248 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.248 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.248 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.248 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.248 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.248 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.249 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.249 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.249 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.249 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.249 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.249 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.249 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.249 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.249 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.249 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.249 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.249 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.249 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.249 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.249 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.250 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.250 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.250 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.250 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.250 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.250 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.250 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.250 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.250 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.250 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.250 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.250 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.250 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.250 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.251 INFO:teuthology.orchestra.run.smithi117.stdout: "server_addr": { 2024-03-19T03:42:13.251 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "server_addr", 2024-03-19T03:42:13.251 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.251 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.251 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.251 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.251 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.251 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.251 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.251 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.251 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.251 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.251 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.252 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.252 INFO:teuthology.orchestra.run.smithi117.stdout: "server_port": { 2024-03-19T03:42:13.252 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "server_port", 2024-03-19T03:42:13.252 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.252 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.252 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.252 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.252 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.252 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.252 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.252 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.252 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.252 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.252 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.252 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.253 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.253 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.253 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.253 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rgw", 2024-03-19T03:42:13.253 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.253 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.253 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.253 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.253 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.253 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.253 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.253 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.253 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.253 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.254 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.254 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.254 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.254 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.254 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.254 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.254 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.254 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.254 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.254 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.254 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.254 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.254 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.255 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.255 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.255 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.255 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.255 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.255 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.255 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.255 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.255 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.255 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.255 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.255 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.255 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.255 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.255 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.256 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.256 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.256 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.256 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.256 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.256 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.256 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.256 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.256 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.256 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.256 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.256 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.256 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.256 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.256 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.257 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.257 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.257 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.257 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.257 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.257 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.257 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.257 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.257 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.257 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.257 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.257 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.257 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.257 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.257 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.257 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.258 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.258 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.258 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.258 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.258 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.258 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.258 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.258 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rook", 2024-03-19T03:42:13.258 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.258 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.258 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.258 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.258 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.258 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.258 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.259 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.259 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.259 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.259 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.259 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.259 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.259 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.259 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.259 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.259 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.259 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.259 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.259 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.259 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.259 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.259 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.260 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.260 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.260 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.260 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.260 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.260 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.260 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.260 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.260 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.260 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.260 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.260 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.260 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.260 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.260 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.261 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.261 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.261 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.261 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.261 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.261 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.261 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.261 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.261 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.261 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.261 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.261 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.261 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.262 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.262 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.262 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.262 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.262 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.262 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.262 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.262 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.262 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.262 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.262 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.262 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.262 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.262 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.263 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.263 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.263 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.263 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.263 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.263 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.263 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.263 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.263 INFO:teuthology.orchestra.run.smithi117.stdout: "storage_class": { 2024-03-19T03:42:13.263 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "storage_class", 2024-03-19T03:42:13.263 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.263 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.263 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.263 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "local", 2024-03-19T03:42:13.264 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.264 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.264 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.264 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-19T03:42:13.264 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.264 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.264 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.264 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.264 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.264 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.264 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.264 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "selftest", 2024-03-19T03:42:13.264 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.264 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.264 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.265 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.265 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.265 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.265 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.265 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.265 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.265 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.265 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.265 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.265 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.265 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.265 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.265 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.265 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.266 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.266 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.266 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.266 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.266 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.266 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.266 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.266 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.266 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.266 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.266 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.266 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.266 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.266 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.267 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.267 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.267 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.267 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.267 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.267 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.267 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.267 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.267 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.267 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.267 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.267 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.267 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.267 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.267 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.267 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.267 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.268 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.268 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.268 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.268 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.268 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.268 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.268 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.268 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.268 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.268 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.268 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.268 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.268 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.268 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.268 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.268 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.269 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.269 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.269 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.269 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.269 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.269 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.269 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.269 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.269 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.269 INFO:teuthology.orchestra.run.smithi117.stdout: "roption1": { 2024-03-19T03:42:13.269 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "roption1", 2024-03-19T03:42:13.269 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.269 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.269 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.269 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.270 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.270 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.270 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.270 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.270 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.270 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.270 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.270 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.270 INFO:teuthology.orchestra.run.smithi117.stdout: "roption2": { 2024-03-19T03:42:13.270 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "roption2", 2024-03-19T03:42:13.270 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.270 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.270 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.270 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "xyz", 2024-03-19T03:42:13.270 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.271 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.271 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.271 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.271 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.271 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.271 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.271 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.271 INFO:teuthology.orchestra.run.smithi117.stdout: "rwoption1": { 2024-03-19T03:42:13.271 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rwoption1", 2024-03-19T03:42:13.271 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.271 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.271 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.272 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.272 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.272 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.272 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.272 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.272 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.272 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.272 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.272 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.272 INFO:teuthology.orchestra.run.smithi117.stdout: "rwoption2": { 2024-03-19T03:42:13.272 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rwoption2", 2024-03-19T03:42:13.272 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.272 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.272 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.272 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.272 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.273 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.273 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.273 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.273 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.273 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.273 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.273 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.273 INFO:teuthology.orchestra.run.smithi117.stdout: "rwoption3": { 2024-03-19T03:42:13.273 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rwoption3", 2024-03-19T03:42:13.273 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "float", 2024-03-19T03:42:13.273 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.273 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.273 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.273 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.273 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.274 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.274 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.274 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.274 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.274 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.274 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.274 INFO:teuthology.orchestra.run.smithi117.stdout: "rwoption4": { 2024-03-19T03:42:13.274 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rwoption4", 2024-03-19T03:42:13.274 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.274 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.274 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.274 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.274 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.274 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.275 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.275 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.275 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.275 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.275 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.275 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.275 INFO:teuthology.orchestra.run.smithi117.stdout: "rwoption5": { 2024-03-19T03:42:13.275 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rwoption5", 2024-03-19T03:42:13.275 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.275 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.275 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.275 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.275 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.275 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.275 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.276 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.276 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.276 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.276 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.276 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.276 INFO:teuthology.orchestra.run.smithi117.stdout: "rwoption6": { 2024-03-19T03:42:13.276 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rwoption6", 2024-03-19T03:42:13.276 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.276 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.276 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.276 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.276 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.276 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.276 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.276 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.277 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.277 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.277 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.277 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.277 INFO:teuthology.orchestra.run.smithi117.stdout: "rwoption7": { 2024-03-19T03:42:13.277 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rwoption7", 2024-03-19T03:42:13.277 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.277 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.277 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.277 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.277 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "1", 2024-03-19T03:42:13.277 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "42", 2024-03-19T03:42:13.277 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.277 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.277 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.278 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.278 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.278 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.278 INFO:teuthology.orchestra.run.smithi117.stdout: "testkey": { 2024-03-19T03:42:13.278 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "testkey", 2024-03-19T03:42:13.278 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.278 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.278 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.278 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.278 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.278 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.278 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.278 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.279 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.279 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.279 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.279 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.279 INFO:teuthology.orchestra.run.smithi117.stdout: "testlkey": { 2024-03-19T03:42:13.279 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "testlkey", 2024-03-19T03:42:13.279 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.279 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.279 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.279 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.279 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.279 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.279 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.279 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.280 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.280 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.280 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.280 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.280 INFO:teuthology.orchestra.run.smithi117.stdout: "testnewline": { 2024-03-19T03:42:13.280 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "testnewline", 2024-03-19T03:42:13.280 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.280 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.280 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.280 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.280 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.280 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.280 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.280 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.281 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.281 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.281 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.281 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.281 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.281 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.281 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.281 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "snap_schedule", 2024-03-19T03:42:13.281 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.281 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.281 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.281 INFO:teuthology.orchestra.run.smithi117.stdout: "allow_m_granularity": { 2024-03-19T03:42:13.281 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "allow_m_granularity", 2024-03-19T03:42:13.281 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.282 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.282 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.282 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.282 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.282 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.282 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.282 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "allow minute scheduled snapshots", 2024-03-19T03:42:13.282 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.282 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.282 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.282 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.282 INFO:teuthology.orchestra.run.smithi117.stdout: "dump_on_update": { 2024-03-19T03:42:13.282 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "dump_on_update", 2024-03-19T03:42:13.282 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.283 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.283 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.283 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.283 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.283 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.283 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.283 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "dump database to debug log on update", 2024-03-19T03:42:13.283 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.283 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.283 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.283 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.283 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.283 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.283 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.284 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.284 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.284 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.284 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.284 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.284 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.284 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.284 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.284 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.284 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.284 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.284 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.284 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.285 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.285 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.285 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.285 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.285 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.285 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.285 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.285 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.285 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.285 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.285 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.285 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.285 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.285 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.286 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.286 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.286 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.286 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.286 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.286 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.286 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.286 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.286 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.286 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.286 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.286 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.286 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.286 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.287 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.287 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.287 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.287 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.287 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.287 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.287 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.287 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.287 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.287 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.287 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.287 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.287 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.287 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.288 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.288 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.288 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.288 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.288 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.288 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.288 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.288 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.288 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.288 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.288 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.288 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.288 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.288 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.288 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.289 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "stats", 2024-03-19T03:42:13.289 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.289 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.289 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.289 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.289 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.289 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.289 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.289 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.289 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.289 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.289 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.289 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.289 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.289 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.290 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.290 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.290 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.290 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.290 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.290 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.290 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.290 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.290 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.290 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.290 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.290 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.290 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.291 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.291 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.291 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.291 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.291 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.291 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.291 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.291 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.291 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.291 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.291 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.291 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.291 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.291 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.291 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.292 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.292 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.292 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.292 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.292 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.292 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.292 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.292 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.292 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.292 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.292 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.292 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.292 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.292 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.292 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.293 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.293 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.293 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.293 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.293 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.293 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.293 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.293 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.293 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.293 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.293 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.293 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.293 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.293 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.293 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.294 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.294 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.294 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.294 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.294 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "status", 2024-03-19T03:42:13.294 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.294 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.294 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.294 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.294 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.294 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.294 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.294 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.294 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.295 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.295 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.295 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.295 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.295 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.295 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.295 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.295 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.295 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.295 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.295 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.295 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.295 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.295 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.295 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.296 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.296 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.296 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.296 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.296 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.296 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.296 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.296 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.296 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.296 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.296 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.296 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.296 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.296 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.297 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.297 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.297 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.297 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.297 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.297 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.297 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.297 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.297 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.297 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.297 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.297 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.297 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.298 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.298 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.298 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.298 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.298 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.298 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.298 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.298 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.298 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.298 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.298 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.298 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.298 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.299 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.299 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.299 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.299 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.299 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.299 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.299 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.299 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.299 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.299 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.299 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.300 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.300 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "telegraf", 2024-03-19T03:42:13.300 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.300 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.300 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.300 INFO:teuthology.orchestra.run.smithi117.stdout: "address": { 2024-03-19T03:42:13.300 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "address", 2024-03-19T03:42:13.300 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.300 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.300 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.300 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-19T03:42:13.300 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.300 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.300 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.301 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.301 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.301 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.301 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.301 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.301 INFO:teuthology.orchestra.run.smithi117.stdout: "interval": { 2024-03-19T03:42:13.301 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "interval", 2024-03-19T03:42:13.301 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.301 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.301 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.301 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "15", 2024-03-19T03:42:13.301 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.301 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.301 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.302 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.302 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.302 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.302 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.302 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.302 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.302 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.302 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.302 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.302 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.302 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.302 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.302 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.303 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.303 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.303 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.303 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.303 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.303 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.303 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.303 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.303 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.303 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.303 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.303 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.303 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.303 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.304 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.304 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.304 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.304 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.304 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.304 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.304 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.304 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.304 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.304 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.304 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.304 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.304 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.304 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.305 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.305 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.305 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.305 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.305 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.305 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.305 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.305 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.305 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.305 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.305 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.305 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.305 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.305 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.305 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.306 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.306 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.306 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.306 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.306 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.306 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.306 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.306 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.306 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.306 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.306 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.306 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.306 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.306 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.307 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.307 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.307 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.307 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.307 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.307 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.307 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.307 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.307 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "telemetry", 2024-03-19T03:42:13.307 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.307 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.307 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.307 INFO:teuthology.orchestra.run.smithi117.stdout: "channel_basic": { 2024-03-19T03:42:13.307 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "channel_basic", 2024-03-19T03:42:13.308 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.308 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.308 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.308 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.308 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.308 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.308 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.308 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-19T03:42:13.308 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.309 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.309 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.309 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.309 INFO:teuthology.orchestra.run.smithi117.stdout: "channel_crash": { 2024-03-19T03:42:13.309 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "channel_crash", 2024-03-19T03:42:13.309 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.309 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.309 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.309 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.309 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.309 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.309 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.309 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-19T03:42:13.309 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.310 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.310 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.310 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.310 INFO:teuthology.orchestra.run.smithi117.stdout: "channel_device": { 2024-03-19T03:42:13.310 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "channel_device", 2024-03-19T03:42:13.310 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.310 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.310 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.310 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.310 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.310 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.310 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.310 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-19T03:42:13.310 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.311 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.311 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.311 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.311 INFO:teuthology.orchestra.run.smithi117.stdout: "channel_ident": { 2024-03-19T03:42:13.311 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "channel_ident", 2024-03-19T03:42:13.311 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.311 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.311 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.311 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.311 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.311 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.311 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.311 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-19T03:42:13.311 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.311 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.312 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.312 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.312 INFO:teuthology.orchestra.run.smithi117.stdout: "channel_perf": { 2024-03-19T03:42:13.312 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "channel_perf", 2024-03-19T03:42:13.312 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.312 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.312 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.312 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.312 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.312 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.312 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.312 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-19T03:42:13.312 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.312 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.313 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.313 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.313 INFO:teuthology.orchestra.run.smithi117.stdout: "contact": { 2024-03-19T03:42:13.313 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "contact", 2024-03-19T03:42:13.313 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.313 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.313 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.313 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.313 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.313 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.313 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.313 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.313 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.313 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.313 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.314 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.314 INFO:teuthology.orchestra.run.smithi117.stdout: "description": { 2024-03-19T03:42:13.314 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "description", 2024-03-19T03:42:13.314 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.314 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.314 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.314 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.314 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.314 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.314 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.314 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.314 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.314 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.315 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.315 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.315 INFO:teuthology.orchestra.run.smithi117.stdout: "device_url": { 2024-03-19T03:42:13.315 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "device_url", 2024-03-19T03:42:13.315 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.315 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.315 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.315 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-19T03:42:13.315 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.315 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.315 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.315 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.315 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.316 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.316 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.316 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.316 INFO:teuthology.orchestra.run.smithi117.stdout: "enabled": { 2024-03-19T03:42:13.316 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "enabled", 2024-03-19T03:42:13.316 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.316 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.316 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.316 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.316 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.316 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.316 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.316 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.316 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.317 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.317 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.317 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.317 INFO:teuthology.orchestra.run.smithi117.stdout: "interval": { 2024-03-19T03:42:13.317 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "interval", 2024-03-19T03:42:13.317 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.317 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.317 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.317 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "24", 2024-03-19T03:42:13.317 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "8", 2024-03-19T03:42:13.317 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.317 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.317 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.317 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.318 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.318 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.318 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.318 INFO:teuthology.orchestra.run.smithi117.stdout: "last_opt_revision": { 2024-03-19T03:42:13.318 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "last_opt_revision", 2024-03-19T03:42:13.318 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.318 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.318 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.318 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "1", 2024-03-19T03:42:13.318 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.318 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.318 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.319 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.319 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.319 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.319 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.319 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.319 INFO:teuthology.orchestra.run.smithi117.stdout: "leaderboard": { 2024-03-19T03:42:13.319 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "leaderboard", 2024-03-19T03:42:13.319 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.319 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.319 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.319 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.319 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.319 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.319 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.320 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.320 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.320 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.320 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.320 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.320 INFO:teuthology.orchestra.run.smithi117.stdout: "leaderboard_description": { 2024-03-19T03:42:13.320 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "leaderboard_description", 2024-03-19T03:42:13.320 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.320 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.320 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.320 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.320 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.320 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.320 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.321 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.321 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.321 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.321 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.321 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.321 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.321 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.321 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.321 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.321 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.321 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.321 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.321 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.321 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.322 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.322 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.322 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.322 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.322 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.322 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.322 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.322 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.322 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.322 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.322 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.322 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.322 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.322 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.323 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.323 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.323 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.323 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.323 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.323 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.323 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.323 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.323 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.323 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.323 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.323 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.323 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.323 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.323 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.323 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.324 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.324 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.324 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.324 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.324 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.324 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.324 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.324 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.324 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.324 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.324 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.324 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.324 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.325 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.325 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.325 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.325 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.325 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.325 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.325 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.325 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.325 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.325 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.325 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.325 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.325 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.325 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.325 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.326 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.326 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.326 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.326 INFO:teuthology.orchestra.run.smithi117.stdout: "organization": { 2024-03-19T03:42:13.326 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "organization", 2024-03-19T03:42:13.326 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.326 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.326 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.326 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.326 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.326 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.326 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.326 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.326 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.326 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.327 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.327 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.327 INFO:teuthology.orchestra.run.smithi117.stdout: "proxy": { 2024-03-19T03:42:13.327 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "proxy", 2024-03-19T03:42:13.327 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.327 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.327 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.327 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.327 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.327 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.327 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.327 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.327 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.327 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.327 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.327 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.327 INFO:teuthology.orchestra.run.smithi117.stdout: "url": { 2024-03-19T03:42:13.328 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "url", 2024-03-19T03:42:13.328 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.328 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.328 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.328 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-19T03:42:13.328 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.328 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.328 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.328 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.328 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.328 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.328 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.328 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.328 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.328 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.328 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.329 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "test_orchestrator", 2024-03-19T03:42:13.329 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.329 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.329 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.329 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.329 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.329 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.329 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.329 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.329 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.329 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.329 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.329 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.329 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.329 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.330 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.330 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.330 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.330 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.330 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.330 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.330 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.330 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.330 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.330 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.330 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.330 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.330 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.330 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.331 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.331 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.331 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.331 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.331 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.331 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.331 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.331 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.331 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.331 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.331 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.331 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.331 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.331 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.331 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.331 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.331 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.332 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.332 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.332 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.332 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.332 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.332 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.332 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.332 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.332 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.332 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.332 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.332 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.332 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.332 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.332 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.333 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.333 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.333 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.333 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.333 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.333 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.333 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.333 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.333 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.333 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.333 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.333 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.333 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.333 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.333 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.333 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.334 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "volumes", 2024-03-19T03:42:13.334 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.334 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.334 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.334 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.334 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.334 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.334 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.334 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.334 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.334 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.334 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.334 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.334 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.334 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.335 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.335 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.335 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.335 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.335 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.335 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.335 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.335 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.335 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.335 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.335 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.335 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.335 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.336 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.336 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.336 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.336 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.336 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.336 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.336 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.336 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.336 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.336 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.336 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.336 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.336 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.336 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.337 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.337 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.337 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.337 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.337 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.337 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.337 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.337 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.337 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.337 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.337 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.338 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.338 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.338 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.338 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.338 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.338 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.338 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.338 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.338 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.338 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.338 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.338 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.338 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.338 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.339 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.339 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.339 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.339 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.339 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.339 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.339 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.339 INFO:teuthology.orchestra.run.smithi117.stdout: "max_concurrent_clones": { 2024-03-19T03:42:13.339 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "max_concurrent_clones", 2024-03-19T03:42:13.339 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.339 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.339 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.339 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "4", 2024-03-19T03:42:13.340 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.340 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.340 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.340 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-19T03:42:13.340 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.340 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.340 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.340 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.340 INFO:teuthology.orchestra.run.smithi117.stdout: "periodic_async_work": { 2024-03-19T03:42:13.340 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "periodic_async_work", 2024-03-19T03:42:13.340 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.340 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.340 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.340 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.340 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.341 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.341 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.341 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Periodically check for async work", 2024-03-19T03:42:13.341 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.341 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.341 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.341 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.341 INFO:teuthology.orchestra.run.smithi117.stdout: "snapshot_clone_delay": { 2024-03-19T03:42:13.341 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "snapshot_clone_delay", 2024-03-19T03:42:13.341 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.341 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.341 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.341 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "0", 2024-03-19T03:42:13.342 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.342 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.342 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.342 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-19T03:42:13.342 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.342 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.342 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.342 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.342 INFO:teuthology.orchestra.run.smithi117.stdout: "snapshot_clone_no_wait": { 2024-03-19T03:42:13.342 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "snapshot_clone_no_wait", 2024-03-19T03:42:13.342 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.342 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.342 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.343 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.343 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.343 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.343 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.343 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-19T03:42:13.343 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.343 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.343 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.343 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.343 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.343 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.343 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.343 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "zabbix", 2024-03-19T03:42:13.344 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.344 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.344 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.344 INFO:teuthology.orchestra.run.smithi117.stdout: "discovery_interval": { 2024-03-19T03:42:13.344 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "discovery_interval", 2024-03-19T03:42:13.344 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "uint", 2024-03-19T03:42:13.344 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.344 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.344 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "100", 2024-03-19T03:42:13.344 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.344 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.344 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.344 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.344 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.344 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.345 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.345 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.345 INFO:teuthology.orchestra.run.smithi117.stdout: "identifier": { 2024-03-19T03:42:13.345 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "identifier", 2024-03-19T03:42:13.345 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.345 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.345 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.345 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.345 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.345 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.345 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.345 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.345 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.345 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.346 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.346 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.346 INFO:teuthology.orchestra.run.smithi117.stdout: "interval": { 2024-03-19T03:42:13.346 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "interval", 2024-03-19T03:42:13.346 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.346 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.346 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.346 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "60", 2024-03-19T03:42:13.346 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.346 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.346 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.346 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.346 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.347 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.347 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.347 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.347 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.347 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.347 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.347 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.347 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.347 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.347 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.347 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.347 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.347 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.347 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.347 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.348 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.348 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.348 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.348 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.348 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.348 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.348 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.348 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.348 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.348 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.348 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.348 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.348 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.348 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.348 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.349 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.349 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.349 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.349 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.349 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.349 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.349 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.349 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.349 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.349 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.349 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.349 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.349 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.349 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.350 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.350 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.350 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.350 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.350 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.350 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.350 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.350 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.350 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.350 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.350 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.350 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.350 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.350 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.351 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.351 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.351 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.351 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.351 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.351 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.351 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.351 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.351 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.351 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.351 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.351 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.351 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.351 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.352 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.352 INFO:teuthology.orchestra.run.smithi117.stdout: "zabbix_host": { 2024-03-19T03:42:13.352 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "zabbix_host", 2024-03-19T03:42:13.352 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.352 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.352 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.352 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.352 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.352 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.352 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.352 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.352 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.352 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.352 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.352 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.353 INFO:teuthology.orchestra.run.smithi117.stdout: "zabbix_port": { 2024-03-19T03:42:13.353 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "zabbix_port", 2024-03-19T03:42:13.353 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.353 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.353 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.353 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10051", 2024-03-19T03:42:13.353 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.353 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.353 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.353 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.353 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.353 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.354 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.354 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.354 INFO:teuthology.orchestra.run.smithi117.stdout: "zabbix_sender": { 2024-03-19T03:42:13.354 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "zabbix_sender", 2024-03-19T03:42:13.354 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.354 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.354 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.354 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-19T03:42:13.354 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.354 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.354 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.354 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.355 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.355 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.355 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.355 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.355 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.355 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.355 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-03-19T03:42:13.355 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.355 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.355 INFO:teuthology.orchestra.run.smithi117.stdout: "modules": [ 2024-03-19T03:42:13.355 INFO:teuthology.orchestra.run.smithi117.stdout: "cephadm", 2024-03-19T03:42:13.355 INFO:teuthology.orchestra.run.smithi117.stdout: "dashboard", 2024-03-19T03:42:13.356 INFO:teuthology.orchestra.run.smithi117.stdout: "iostat", 2024-03-19T03:42:13.356 INFO:teuthology.orchestra.run.smithi117.stdout: "nfs", 2024-03-19T03:42:13.356 INFO:teuthology.orchestra.run.smithi117.stdout: "restful" 2024-03-19T03:42:13.356 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.356 INFO:teuthology.orchestra.run.smithi117.stdout: "available_modules": [ 2024-03-19T03:42:13.356 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.356 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "alerts", 2024-03-19T03:42:13.356 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.356 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.356 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.357 INFO:teuthology.orchestra.run.smithi117.stdout: "interval": { 2024-03-19T03:42:13.357 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "interval", 2024-03-19T03:42:13.357 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.357 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.357 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.357 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "60", 2024-03-19T03:42:13.357 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.357 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.357 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.357 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "How frequently to reexamine health status", 2024-03-19T03:42:13.357 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.358 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.358 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.358 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.358 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.358 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.358 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.358 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.358 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.358 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.358 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.358 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.359 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.359 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.359 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.359 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.359 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.359 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.359 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.359 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.359 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.359 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.359 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.359 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.359 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.360 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.360 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.360 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.360 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.360 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.360 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.360 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.360 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.360 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.360 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.360 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.360 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.360 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.361 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.361 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.361 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.361 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.361 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.361 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.361 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.361 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.361 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.361 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.361 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.361 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.361 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.361 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.362 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.362 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.362 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.362 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.362 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.362 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.362 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.362 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.362 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.362 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.362 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.362 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.363 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.363 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.363 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.363 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.363 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.363 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.363 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.363 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.363 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.363 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.363 INFO:teuthology.orchestra.run.smithi117.stdout: "smtp_destination": { 2024-03-19T03:42:13.363 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "smtp_destination", 2024-03-19T03:42:13.364 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.364 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.364 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.364 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.364 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.364 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.364 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.364 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Email address to send alerts to", 2024-03-19T03:42:13.364 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.364 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.364 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.364 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.364 INFO:teuthology.orchestra.run.smithi117.stdout: "smtp_from_name": { 2024-03-19T03:42:13.364 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "smtp_from_name", 2024-03-19T03:42:13.364 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.364 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.365 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.365 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "Ceph", 2024-03-19T03:42:13.365 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.365 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.365 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.365 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Email From: name", 2024-03-19T03:42:13.365 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.365 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.365 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.365 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.365 INFO:teuthology.orchestra.run.smithi117.stdout: "smtp_host": { 2024-03-19T03:42:13.365 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "smtp_host", 2024-03-19T03:42:13.366 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.366 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.366 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.366 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.366 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.366 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.366 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.366 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "SMTP server", 2024-03-19T03:42:13.366 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.366 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.366 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.366 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.366 INFO:teuthology.orchestra.run.smithi117.stdout: "smtp_password": { 2024-03-19T03:42:13.366 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "smtp_password", 2024-03-19T03:42:13.367 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.367 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.367 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.367 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.367 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.367 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.367 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.367 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Password to authenticate with", 2024-03-19T03:42:13.367 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.367 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.367 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.367 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.367 INFO:teuthology.orchestra.run.smithi117.stdout: "smtp_port": { 2024-03-19T03:42:13.367 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "smtp_port", 2024-03-19T03:42:13.367 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.368 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.368 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.368 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "465", 2024-03-19T03:42:13.368 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.368 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.368 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.368 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "SMTP port", 2024-03-19T03:42:13.368 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.368 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.368 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.368 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.368 INFO:teuthology.orchestra.run.smithi117.stdout: "smtp_sender": { 2024-03-19T03:42:13.368 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "smtp_sender", 2024-03-19T03:42:13.369 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.369 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.369 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.369 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.369 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.369 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.369 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.369 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "SMTP envelope sender", 2024-03-19T03:42:13.369 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.369 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.369 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.369 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.369 INFO:teuthology.orchestra.run.smithi117.stdout: "smtp_ssl": { 2024-03-19T03:42:13.369 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "smtp_ssl", 2024-03-19T03:42:13.369 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.370 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.370 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.370 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.370 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.370 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.370 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.370 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-19T03:42:13.370 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.370 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.370 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.370 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.370 INFO:teuthology.orchestra.run.smithi117.stdout: "smtp_user": { 2024-03-19T03:42:13.370 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "smtp_user", 2024-03-19T03:42:13.370 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.371 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.371 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.371 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.371 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.371 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.371 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.371 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "User to authenticate as", 2024-03-19T03:42:13.371 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.371 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.371 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.371 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.371 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.372 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.372 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.372 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "balancer", 2024-03-19T03:42:13.372 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.372 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.372 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.372 INFO:teuthology.orchestra.run.smithi117.stdout: "active": { 2024-03-19T03:42:13.372 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "active", 2024-03-19T03:42:13.372 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.372 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.372 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.372 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.372 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.372 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.373 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.373 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "automatically balance PGs across cluster", 2024-03-19T03:42:13.373 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.373 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.373 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.373 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.373 INFO:teuthology.orchestra.run.smithi117.stdout: "begin_time": { 2024-03-19T03:42:13.373 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "begin_time", 2024-03-19T03:42:13.373 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.373 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.373 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.373 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "0000", 2024-03-19T03:42:13.373 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.373 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.374 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.374 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "beginning time of day to automatically balance", 2024-03-19T03:42:13.374 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-19T03:42:13.374 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.374 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.374 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.374 INFO:teuthology.orchestra.run.smithi117.stdout: "begin_weekday": { 2024-03-19T03:42:13.374 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "begin_weekday", 2024-03-19T03:42:13.374 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "uint", 2024-03-19T03:42:13.374 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.374 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.374 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "0", 2024-03-19T03:42:13.374 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "0", 2024-03-19T03:42:13.374 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "6", 2024-03-19T03:42:13.375 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.375 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-19T03:42:13.375 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-19T03:42:13.375 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.375 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.375 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.375 INFO:teuthology.orchestra.run.smithi117.stdout: "crush_compat_max_iterations": { 2024-03-19T03:42:13.375 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "crush_compat_max_iterations", 2024-03-19T03:42:13.375 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "uint", 2024-03-19T03:42:13.375 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.375 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.375 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "25", 2024-03-19T03:42:13.375 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "1", 2024-03-19T03:42:13.376 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "250", 2024-03-19T03:42:13.376 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.376 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-19T03:42:13.376 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.376 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.376 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.376 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.376 INFO:teuthology.orchestra.run.smithi117.stdout: "crush_compat_metrics": { 2024-03-19T03:42:13.376 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "crush_compat_metrics", 2024-03-19T03:42:13.376 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.376 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.376 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.377 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "pgs,objects,bytes", 2024-03-19T03:42:13.377 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.377 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.377 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.377 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-19T03:42:13.377 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-19T03:42:13.377 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.377 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.377 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.377 INFO:teuthology.orchestra.run.smithi117.stdout: "crush_compat_step": { 2024-03-19T03:42:13.377 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "crush_compat_step", 2024-03-19T03:42:13.377 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "float", 2024-03-19T03:42:13.377 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.378 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.378 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "0.5", 2024-03-19T03:42:13.378 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "0.001", 2024-03-19T03:42:13.378 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "0.999", 2024-03-19T03:42:13.378 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.378 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "aggressiveness of optimization", 2024-03-19T03:42:13.378 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-19T03:42:13.378 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.378 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.378 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.378 INFO:teuthology.orchestra.run.smithi117.stdout: "end_time": { 2024-03-19T03:42:13.378 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "end_time", 2024-03-19T03:42:13.378 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.378 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.379 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.379 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "2359", 2024-03-19T03:42:13.379 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.379 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.379 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.379 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "ending time of day to automatically balance", 2024-03-19T03:42:13.379 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-19T03:42:13.379 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.379 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.379 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.379 INFO:teuthology.orchestra.run.smithi117.stdout: "end_weekday": { 2024-03-19T03:42:13.379 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "end_weekday", 2024-03-19T03:42:13.379 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "uint", 2024-03-19T03:42:13.379 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.380 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.380 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "0", 2024-03-19T03:42:13.380 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "0", 2024-03-19T03:42:13.380 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "6", 2024-03-19T03:42:13.380 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.380 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-19T03:42:13.380 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-19T03:42:13.380 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.380 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.380 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.380 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.380 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.380 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.381 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.381 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.381 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.381 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.381 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.381 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.381 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.381 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.381 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.381 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.381 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.381 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.381 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.382 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.382 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.382 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.382 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.382 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.382 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.382 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.382 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.382 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.382 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.382 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.382 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.382 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.383 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.383 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.383 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.383 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.383 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.383 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.383 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.383 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.383 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.383 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.383 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.383 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.383 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.383 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.383 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.384 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.384 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.384 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.384 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.384 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.384 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.384 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.384 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.384 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.384 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.384 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.384 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.384 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.384 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.385 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.385 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.385 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.385 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.385 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.385 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.385 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.385 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.385 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.385 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.385 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.386 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.386 INFO:teuthology.orchestra.run.smithi117.stdout: "min_score": { 2024-03-19T03:42:13.386 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "min_score", 2024-03-19T03:42:13.386 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "float", 2024-03-19T03:42:13.386 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.386 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.386 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "0", 2024-03-19T03:42:13.386 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.386 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.386 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.386 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-19T03:42:13.386 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.387 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.387 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.387 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.387 INFO:teuthology.orchestra.run.smithi117.stdout: "mode": { 2024-03-19T03:42:13.387 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "mode", 2024-03-19T03:42:13.387 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.387 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.387 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.387 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "upmap", 2024-03-19T03:42:13.387 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.387 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.387 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.387 INFO:teuthology.orchestra.run.smithi117.stdout: "crush-compat", 2024-03-19T03:42:13.387 INFO:teuthology.orchestra.run.smithi117.stdout: "none", 2024-03-19T03:42:13.387 INFO:teuthology.orchestra.run.smithi117.stdout: "read", 2024-03-19T03:42:13.387 INFO:teuthology.orchestra.run.smithi117.stdout: "upmap", 2024-03-19T03:42:13.387 INFO:teuthology.orchestra.run.smithi117.stdout: "upmap-read" 2024-03-19T03:42:13.388 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.388 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Balancer mode", 2024-03-19T03:42:13.388 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.388 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.388 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.388 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.388 INFO:teuthology.orchestra.run.smithi117.stdout: "pool_ids": { 2024-03-19T03:42:13.388 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "pool_ids", 2024-03-19T03:42:13.388 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.388 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.388 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.388 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.388 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.389 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.389 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.389 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-19T03:42:13.389 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.389 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.389 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.389 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.389 INFO:teuthology.orchestra.run.smithi117.stdout: "sleep_interval": { 2024-03-19T03:42:13.389 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "sleep_interval", 2024-03-19T03:42:13.389 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.389 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.389 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.389 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "60", 2024-03-19T03:42:13.390 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.390 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.390 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.390 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-19T03:42:13.390 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.390 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.390 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.390 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.390 INFO:teuthology.orchestra.run.smithi117.stdout: "upmap_max_deviation": { 2024-03-19T03:42:13.390 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "upmap_max_deviation", 2024-03-19T03:42:13.390 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.390 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.390 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.390 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "5", 2024-03-19T03:42:13.391 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "1", 2024-03-19T03:42:13.391 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.391 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.391 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-19T03:42:13.391 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-19T03:42:13.391 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.391 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.391 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.391 INFO:teuthology.orchestra.run.smithi117.stdout: "upmap_max_optimizations": { 2024-03-19T03:42:13.391 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "upmap_max_optimizations", 2024-03-19T03:42:13.391 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "uint", 2024-03-19T03:42:13.392 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.392 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.392 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10", 2024-03-19T03:42:13.392 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.392 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.392 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.392 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-19T03:42:13.392 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.392 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.392 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.392 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.392 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.392 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.392 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.393 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "cephadm", 2024-03-19T03:42:13.393 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.393 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.393 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.393 INFO:teuthology.orchestra.run.smithi117.stdout: "agent_down_multiplier": { 2024-03-19T03:42:13.393 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "agent_down_multiplier", 2024-03-19T03:42:13.393 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "float", 2024-03-19T03:42:13.393 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.393 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.393 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "3.0", 2024-03-19T03:42:13.393 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.393 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.394 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.394 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-19T03:42:13.394 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.394 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.394 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.394 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.394 INFO:teuthology.orchestra.run.smithi117.stdout: "agent_refresh_rate": { 2024-03-19T03:42:13.394 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "agent_refresh_rate", 2024-03-19T03:42:13.394 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.394 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.394 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.394 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "20", 2024-03-19T03:42:13.394 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.394 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.395 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.395 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-19T03:42:13.395 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.395 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.395 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.395 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.395 INFO:teuthology.orchestra.run.smithi117.stdout: "agent_starting_port": { 2024-03-19T03:42:13.395 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "agent_starting_port", 2024-03-19T03:42:13.395 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.395 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.395 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.395 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "4721", 2024-03-19T03:42:13.395 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.396 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.396 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.396 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-19T03:42:13.396 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.396 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.396 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.396 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.396 INFO:teuthology.orchestra.run.smithi117.stdout: "allow_ptrace": { 2024-03-19T03:42:13.396 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "allow_ptrace", 2024-03-19T03:42:13.396 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.396 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.396 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.397 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.397 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.397 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.397 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.397 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-19T03:42:13.397 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-19T03:42:13.397 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.397 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.397 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.397 INFO:teuthology.orchestra.run.smithi117.stdout: "autotune_interval": { 2024-03-19T03:42:13.397 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "autotune_interval", 2024-03-19T03:42:13.397 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.397 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.397 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.398 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "600", 2024-03-19T03:42:13.398 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.398 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.398 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.398 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-19T03:42:13.398 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.398 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.398 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.398 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.398 INFO:teuthology.orchestra.run.smithi117.stdout: "autotune_memory_target_ratio": { 2024-03-19T03:42:13.398 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "autotune_memory_target_ratio", 2024-03-19T03:42:13.398 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "float", 2024-03-19T03:42:13.398 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.398 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.399 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "0.7", 2024-03-19T03:42:13.399 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.399 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.399 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.399 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-19T03:42:13.399 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.399 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.399 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.399 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.399 INFO:teuthology.orchestra.run.smithi117.stdout: "cephadm_log_destination": { 2024-03-19T03:42:13.399 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "cephadm_log_destination", 2024-03-19T03:42:13.399 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.399 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.399 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.400 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.400 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.400 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.400 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.400 INFO:teuthology.orchestra.run.smithi117.stdout: "file", 2024-03-19T03:42:13.400 INFO:teuthology.orchestra.run.smithi117.stdout: "file,syslog", 2024-03-19T03:42:13.400 INFO:teuthology.orchestra.run.smithi117.stdout: "syslog" 2024-03-19T03:42:13.400 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.400 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-19T03:42:13.400 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.400 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.400 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.400 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.400 INFO:teuthology.orchestra.run.smithi117.stdout: "cgroups_split": { 2024-03-19T03:42:13.400 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "cgroups_split", 2024-03-19T03:42:13.401 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.401 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.401 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.401 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.401 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.401 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.401 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.401 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-19T03:42:13.401 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.401 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.401 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.401 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.401 INFO:teuthology.orchestra.run.smithi117.stdout: "config_checks_enabled": { 2024-03-19T03:42:13.402 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "config_checks_enabled", 2024-03-19T03:42:13.402 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.402 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.402 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.402 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.402 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.402 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.402 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.402 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-19T03:42:13.402 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.402 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.402 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.402 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.403 INFO:teuthology.orchestra.run.smithi117.stdout: "config_dashboard": { 2024-03-19T03:42:13.403 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "config_dashboard", 2024-03-19T03:42:13.403 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.403 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.403 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.403 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.403 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.403 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.403 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.403 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-19T03:42:13.403 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.403 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.403 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.404 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.404 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_alertmanager": { 2024-03-19T03:42:13.404 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_alertmanager", 2024-03-19T03:42:13.404 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.404 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.404 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.404 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-19T03:42:13.404 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.404 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.404 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.404 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Prometheus container image", 2024-03-19T03:42:13.405 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.405 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.405 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.405 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.405 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_base": { 2024-03-19T03:42:13.405 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_base", 2024-03-19T03:42:13.405 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.405 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.405 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.405 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-19T03:42:13.405 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.405 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.405 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.406 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Container image name, without the tag", 2024-03-19T03:42:13.406 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.406 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.406 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.406 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.406 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_elasticsearch": { 2024-03-19T03:42:13.406 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_elasticsearch", 2024-03-19T03:42:13.406 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.406 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.406 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.406 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-19T03:42:13.406 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.407 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.407 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.407 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "elasticsearch container image", 2024-03-19T03:42:13.407 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.407 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.407 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.407 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.407 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_grafana": { 2024-03-19T03:42:13.407 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_grafana", 2024-03-19T03:42:13.407 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.407 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.407 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.407 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-19T03:42:13.407 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.408 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.408 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.408 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Prometheus container image", 2024-03-19T03:42:13.408 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.408 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.408 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.408 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.408 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_haproxy": { 2024-03-19T03:42:13.408 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_haproxy", 2024-03-19T03:42:13.408 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.408 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.409 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.409 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-19T03:42:13.409 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.409 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.409 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.409 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "HAproxy container image", 2024-03-19T03:42:13.409 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.409 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.409 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.409 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.409 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_jaeger_agent": { 2024-03-19T03:42:13.409 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_jaeger_agent", 2024-03-19T03:42:13.409 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.409 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.410 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.410 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-19T03:42:13.410 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.410 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.410 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.410 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Jaeger agent container image", 2024-03-19T03:42:13.410 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.410 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.410 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.410 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.410 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_jaeger_collector": { 2024-03-19T03:42:13.410 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_jaeger_collector", 2024-03-19T03:42:13.410 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.411 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.411 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.411 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-19T03:42:13.411 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.411 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.411 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.411 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Jaeger collector container image", 2024-03-19T03:42:13.411 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.411 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.411 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.411 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.411 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_jaeger_query": { 2024-03-19T03:42:13.411 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_jaeger_query", 2024-03-19T03:42:13.411 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.411 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.412 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.412 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-19T03:42:13.412 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.412 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.412 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.412 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Jaeger query container image", 2024-03-19T03:42:13.412 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.412 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.412 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.412 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.412 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_keepalived": { 2024-03-19T03:42:13.412 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_keepalived", 2024-03-19T03:42:13.412 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.413 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.413 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.413 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-19T03:42:13.413 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.413 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.413 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.413 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Keepalived container image", 2024-03-19T03:42:13.413 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.413 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.413 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.413 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.413 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_loki": { 2024-03-19T03:42:13.414 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_loki", 2024-03-19T03:42:13.414 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.414 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.414 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.414 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-19T03:42:13.414 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.414 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.414 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.414 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Loki container image", 2024-03-19T03:42:13.414 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.414 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.414 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.414 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.415 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_node_exporter": { 2024-03-19T03:42:13.415 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_node_exporter", 2024-03-19T03:42:13.415 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.415 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.415 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.415 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-19T03:42:13.415 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.415 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.415 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.415 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Prometheus container image", 2024-03-19T03:42:13.415 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.415 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.416 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.416 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.416 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_nvmeof": { 2024-03-19T03:42:13.416 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_nvmeof", 2024-03-19T03:42:13.416 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.416 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.416 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.416 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-19T03:42:13.416 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.416 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.416 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.417 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Nvme-of container image", 2024-03-19T03:42:13.417 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.417 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.417 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.417 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.417 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_prometheus": { 2024-03-19T03:42:13.417 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_prometheus", 2024-03-19T03:42:13.417 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.417 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.417 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.417 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-19T03:42:13.417 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.417 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.418 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.418 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Prometheus container image", 2024-03-19T03:42:13.418 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.418 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.418 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.418 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.418 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_promtail": { 2024-03-19T03:42:13.418 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_promtail", 2024-03-19T03:42:13.418 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.418 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.418 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.418 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-19T03:42:13.419 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.419 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.419 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.419 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Promtail container image", 2024-03-19T03:42:13.419 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.419 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.419 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.419 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.419 INFO:teuthology.orchestra.run.smithi117.stdout: "container_image_snmp_gateway": { 2024-03-19T03:42:13.419 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_image_snmp_gateway", 2024-03-19T03:42:13.419 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.419 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.419 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.420 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-19T03:42:13.420 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.420 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.420 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.420 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "SNMP Gateway container image", 2024-03-19T03:42:13.420 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.420 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.420 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.420 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.420 INFO:teuthology.orchestra.run.smithi117.stdout: "container_init": { 2024-03-19T03:42:13.420 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "container_init", 2024-03-19T03:42:13.420 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.420 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.421 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.421 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.421 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.421 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.421 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.421 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Run podman/docker with `--init`", 2024-03-19T03:42:13.421 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.421 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.421 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.421 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.421 INFO:teuthology.orchestra.run.smithi117.stdout: "daemon_cache_timeout": { 2024-03-19T03:42:13.422 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "daemon_cache_timeout", 2024-03-19T03:42:13.422 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.422 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.422 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.422 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "600", 2024-03-19T03:42:13.422 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.422 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.425 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.425 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-19T03:42:13.425 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.425 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.425 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.425 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.425 INFO:teuthology.orchestra.run.smithi117.stdout: "default_cephadm_command_timeout": { 2024-03-19T03:42:13.425 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "default_cephadm_command_timeout", 2024-03-19T03:42:13.425 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.426 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.426 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.426 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "900", 2024-03-19T03:42:13.426 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.426 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.426 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.426 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-19T03:42:13.426 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.426 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.426 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.426 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.426 INFO:teuthology.orchestra.run.smithi117.stdout: "default_registry": { 2024-03-19T03:42:13.426 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "default_registry", 2024-03-19T03:42:13.426 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.427 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.427 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.427 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "docker.io", 2024-03-19T03:42:13.427 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.427 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.427 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.427 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-19T03:42:13.427 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.427 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.427 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.427 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.427 INFO:teuthology.orchestra.run.smithi117.stdout: "device_cache_timeout": { 2024-03-19T03:42:13.427 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "device_cache_timeout", 2024-03-19T03:42:13.427 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.428 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.428 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.428 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "1800", 2024-03-19T03:42:13.428 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.428 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.428 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.428 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "seconds to cache device inventory", 2024-03-19T03:42:13.428 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.428 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.428 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.428 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.428 INFO:teuthology.orchestra.run.smithi117.stdout: "device_enhanced_scan": { 2024-03-19T03:42:13.428 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "device_enhanced_scan", 2024-03-19T03:42:13.428 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.428 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.429 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.429 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.429 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.429 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.429 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.429 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-19T03:42:13.429 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.429 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.429 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.429 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.429 INFO:teuthology.orchestra.run.smithi117.stdout: "facts_cache_timeout": { 2024-03-19T03:42:13.429 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "facts_cache_timeout", 2024-03-19T03:42:13.430 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.430 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.430 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.430 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "60", 2024-03-19T03:42:13.430 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.430 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.430 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.430 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "seconds to cache host facts data", 2024-03-19T03:42:13.430 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.430 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.430 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.430 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.430 INFO:teuthology.orchestra.run.smithi117.stdout: "host_check_interval": { 2024-03-19T03:42:13.431 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "host_check_interval", 2024-03-19T03:42:13.431 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.431 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.431 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.431 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "600", 2024-03-19T03:42:13.431 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.431 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.431 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.431 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "how frequently to perform a host check", 2024-03-19T03:42:13.431 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.431 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.431 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.431 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.431 INFO:teuthology.orchestra.run.smithi117.stdout: "hw_monitoring": { 2024-03-19T03:42:13.432 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "hw_monitoring", 2024-03-19T03:42:13.432 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.432 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.432 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.432 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.432 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.432 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.432 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.432 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-19T03:42:13.432 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.432 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.432 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.432 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.432 INFO:teuthology.orchestra.run.smithi117.stdout: "inventory_list_all": { 2024-03-19T03:42:13.433 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "inventory_list_all", 2024-03-19T03:42:13.433 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.433 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.433 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.433 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.433 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.433 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.433 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.433 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-19T03:42:13.433 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.433 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.433 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.433 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.433 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.433 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.434 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.434 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.434 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.434 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.434 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.434 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.434 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.434 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.434 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.434 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.434 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.434 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.434 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.434 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.434 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.435 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.435 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.435 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.435 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.435 INFO:teuthology.orchestra.run.smithi117.stdout: "log_refresh_metadata": { 2024-03-19T03:42:13.435 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_refresh_metadata", 2024-03-19T03:42:13.435 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.435 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.435 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.435 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.435 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.435 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.435 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.435 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-19T03:42:13.436 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.436 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.436 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.436 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.436 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.436 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.436 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.436 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.436 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.436 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.436 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.436 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.436 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.436 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-19T03:42:13.436 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.436 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.437 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.437 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.437 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.437 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.437 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.437 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.437 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.437 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.437 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.437 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.437 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.437 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.437 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.437 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.438 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.438 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.438 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.438 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.438 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.438 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.438 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.438 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.438 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.438 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.438 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.438 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.438 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.439 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.439 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.439 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.439 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.439 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.439 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.439 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.439 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.439 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.439 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.439 INFO:teuthology.orchestra.run.smithi117.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-19T03:42:13.439 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-19T03:42:13.439 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.439 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.439 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.440 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.440 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.440 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.440 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.440 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-19T03:42:13.440 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.440 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.440 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.440 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.440 INFO:teuthology.orchestra.run.smithi117.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-19T03:42:13.440 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-19T03:42:13.440 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.440 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.440 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.441 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "*", 2024-03-19T03:42:13.441 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.441 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.441 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.441 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-19T03:42:13.441 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.441 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.441 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.441 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.441 INFO:teuthology.orchestra.run.smithi117.stdout: "max_count_per_host": { 2024-03-19T03:42:13.441 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "max_count_per_host", 2024-03-19T03:42:13.441 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.441 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.441 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.441 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10", 2024-03-19T03:42:13.442 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.442 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.442 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.442 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "max number of daemons per service per host", 2024-03-19T03:42:13.442 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.442 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.442 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.442 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.442 INFO:teuthology.orchestra.run.smithi117.stdout: "max_osd_draining_count": { 2024-03-19T03:42:13.442 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "max_osd_draining_count", 2024-03-19T03:42:13.442 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.442 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.442 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.442 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10", 2024-03-19T03:42:13.443 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.443 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.443 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.443 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-19T03:42:13.443 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.443 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.443 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.443 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.443 INFO:teuthology.orchestra.run.smithi117.stdout: "migration_current": { 2024-03-19T03:42:13.443 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "migration_current", 2024-03-19T03:42:13.443 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.443 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.443 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.443 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.444 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.444 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.444 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.444 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "internal - do not modify", 2024-03-19T03:42:13.444 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.444 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.444 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.444 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.444 INFO:teuthology.orchestra.run.smithi117.stdout: "mode": { 2024-03-19T03:42:13.444 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "mode", 2024-03-19T03:42:13.444 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.444 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.444 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.444 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "root", 2024-03-19T03:42:13.444 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.445 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.445 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.445 INFO:teuthology.orchestra.run.smithi117.stdout: "cephadm-package", 2024-03-19T03:42:13.445 INFO:teuthology.orchestra.run.smithi117.stdout: "root" 2024-03-19T03:42:13.445 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.445 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "mode for remote execution of cephadm", 2024-03-19T03:42:13.445 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.445 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.445 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.445 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.445 INFO:teuthology.orchestra.run.smithi117.stdout: "oob_default_addr": { 2024-03-19T03:42:13.445 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "oob_default_addr", 2024-03-19T03:42:13.445 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.445 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.445 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.446 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "169.254.1.1", 2024-03-19T03:42:13.446 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.446 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.446 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.446 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-19T03:42:13.446 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.446 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.446 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.446 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.446 INFO:teuthology.orchestra.run.smithi117.stdout: "prometheus_alerts_path": { 2024-03-19T03:42:13.446 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "prometheus_alerts_path", 2024-03-19T03:42:13.446 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.446 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.447 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.447 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-19T03:42:13.447 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.447 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.447 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.447 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-19T03:42:13.447 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.447 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.447 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.447 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.447 INFO:teuthology.orchestra.run.smithi117.stdout: "registry_insecure": { 2024-03-19T03:42:13.447 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "registry_insecure", 2024-03-19T03:42:13.447 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.447 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.447 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.448 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.448 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.448 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.448 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.448 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-19T03:42:13.448 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.448 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.448 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.448 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.448 INFO:teuthology.orchestra.run.smithi117.stdout: "registry_password": { 2024-03-19T03:42:13.448 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "registry_password", 2024-03-19T03:42:13.448 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.448 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.448 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.449 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.449 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.449 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.449 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.449 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-19T03:42:13.449 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.449 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.449 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.449 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.449 INFO:teuthology.orchestra.run.smithi117.stdout: "registry_url": { 2024-03-19T03:42:13.449 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "registry_url", 2024-03-19T03:42:13.449 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.449 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.449 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.450 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.450 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.450 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.450 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.450 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-19T03:42:13.450 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.450 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.450 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.450 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.450 INFO:teuthology.orchestra.run.smithi117.stdout: "registry_username": { 2024-03-19T03:42:13.450 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "registry_username", 2024-03-19T03:42:13.450 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.450 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.450 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.450 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.451 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.451 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.451 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.451 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-19T03:42:13.451 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.451 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.451 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.451 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.451 INFO:teuthology.orchestra.run.smithi117.stdout: "secure_monitoring_stack": { 2024-03-19T03:42:13.451 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "secure_monitoring_stack", 2024-03-19T03:42:13.451 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.451 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.451 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.451 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.451 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.452 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.452 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.452 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-19T03:42:13.452 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.452 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.452 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.452 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.452 INFO:teuthology.orchestra.run.smithi117.stdout: "service_discovery_port": { 2024-03-19T03:42:13.452 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "service_discovery_port", 2024-03-19T03:42:13.452 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.452 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.452 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.452 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "8765", 2024-03-19T03:42:13.452 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.452 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.452 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.453 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "cephadm service discovery port", 2024-03-19T03:42:13.453 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.453 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.453 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.453 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.453 INFO:teuthology.orchestra.run.smithi117.stdout: "ssh_config_file": { 2024-03-19T03:42:13.453 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ssh_config_file", 2024-03-19T03:42:13.453 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.453 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.453 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.453 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.453 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.453 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.453 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.453 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-19T03:42:13.454 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.454 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.454 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.454 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.454 INFO:teuthology.orchestra.run.smithi117.stdout: "use_agent": { 2024-03-19T03:42:13.454 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "use_agent", 2024-03-19T03:42:13.454 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.454 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.454 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.454 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.454 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.454 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.454 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.454 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-19T03:42:13.455 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.455 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.455 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.455 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.455 INFO:teuthology.orchestra.run.smithi117.stdout: "use_repo_digest": { 2024-03-19T03:42:13.455 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "use_repo_digest", 2024-03-19T03:42:13.455 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.455 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.455 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.455 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.455 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.455 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.455 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.455 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-19T03:42:13.456 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.456 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.456 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.456 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.456 INFO:teuthology.orchestra.run.smithi117.stdout: "warn_on_failed_host_check": { 2024-03-19T03:42:13.456 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "warn_on_failed_host_check", 2024-03-19T03:42:13.456 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.456 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.456 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.456 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.456 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.456 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.456 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.456 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "raise a health warning if the host check fails", 2024-03-19T03:42:13.457 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.457 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.457 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.457 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.457 INFO:teuthology.orchestra.run.smithi117.stdout: "warn_on_stray_daemons": { 2024-03-19T03:42:13.457 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "warn_on_stray_daemons", 2024-03-19T03:42:13.457 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.457 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.457 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.457 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.457 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.457 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.457 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.457 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-19T03:42:13.457 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.458 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.458 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.458 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.458 INFO:teuthology.orchestra.run.smithi117.stdout: "warn_on_stray_hosts": { 2024-03-19T03:42:13.458 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "warn_on_stray_hosts", 2024-03-19T03:42:13.458 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.458 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.458 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.458 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.458 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.458 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.458 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.458 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-19T03:42:13.458 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.459 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.459 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.459 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.459 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.459 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.459 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.459 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "crash", 2024-03-19T03:42:13.459 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.459 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.459 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.459 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.459 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.459 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.459 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.460 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.460 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.460 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.460 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.460 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.460 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.460 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.460 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.460 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.460 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.460 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.460 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.460 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.460 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.461 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.461 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.461 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.461 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.461 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.461 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.461 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.461 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.461 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.461 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.461 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.461 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.461 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.461 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.461 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.462 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.462 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.462 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.462 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.462 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.462 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.462 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.462 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.462 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.462 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.462 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.462 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.462 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.462 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.462 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.463 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.463 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.463 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.463 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.463 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.463 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.463 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.463 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.463 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.463 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.463 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.463 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.463 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.463 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.463 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.463 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.464 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.464 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.464 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.464 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.464 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.464 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.464 INFO:teuthology.orchestra.run.smithi117.stdout: "retain_interval": { 2024-03-19T03:42:13.464 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "retain_interval", 2024-03-19T03:42:13.464 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.464 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.464 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.464 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "31536000", 2024-03-19T03:42:13.464 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.464 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.464 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.465 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-19T03:42:13.465 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.465 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.465 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.465 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.465 INFO:teuthology.orchestra.run.smithi117.stdout: "warn_recent_interval": { 2024-03-19T03:42:13.465 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "warn_recent_interval", 2024-03-19T03:42:13.465 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.465 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.465 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.465 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "1209600", 2024-03-19T03:42:13.465 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.465 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.465 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.465 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-19T03:42:13.465 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.465 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.466 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.466 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.466 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.466 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.466 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.466 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "dashboard", 2024-03-19T03:42:13.466 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.466 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.466 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.466 INFO:teuthology.orchestra.run.smithi117.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-19T03:42:13.466 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-19T03:42:13.466 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.466 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.466 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.467 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10", 2024-03-19T03:42:13.467 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.467 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.467 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.467 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.467 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.467 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.467 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.467 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.467 INFO:teuthology.orchestra.run.smithi117.stdout: "ALERTMANAGER_API_HOST": { 2024-03-19T03:42:13.467 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-19T03:42:13.467 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.467 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.467 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.467 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.467 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.468 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.468 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.468 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.468 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.468 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.468 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.468 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.468 INFO:teuthology.orchestra.run.smithi117.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-19T03:42:13.468 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-19T03:42:13.468 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.468 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.468 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.468 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.468 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.469 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.469 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.469 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.469 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.469 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.469 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.469 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.469 INFO:teuthology.orchestra.run.smithi117.stdout: "AUDIT_API_ENABLED": { 2024-03-19T03:42:13.469 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "AUDIT_API_ENABLED", 2024-03-19T03:42:13.469 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.469 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.469 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.469 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.469 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.470 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.470 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.470 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.470 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.470 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.470 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.470 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.470 INFO:teuthology.orchestra.run.smithi117.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-19T03:42:13.470 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-19T03:42:13.470 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.470 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.470 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.470 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.471 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.471 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.471 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.471 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.471 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.471 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.471 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.471 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.471 INFO:teuthology.orchestra.run.smithi117.stdout: "ENABLE_BROWSABLE_API": { 2024-03-19T03:42:13.471 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-19T03:42:13.471 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.471 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.471 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.471 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.471 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.472 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.472 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.472 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.472 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.472 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.472 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.472 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.472 INFO:teuthology.orchestra.run.smithi117.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-19T03:42:13.472 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-19T03:42:13.472 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.472 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.472 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.472 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.472 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.472 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.473 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.473 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.473 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.473 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.473 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.473 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.473 INFO:teuthology.orchestra.run.smithi117.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-19T03:42:13.473 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-19T03:42:13.473 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.473 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.473 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.473 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.473 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.473 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.473 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.474 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.474 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.474 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.474 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.474 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.474 INFO:teuthology.orchestra.run.smithi117.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-19T03:42:13.474 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-19T03:42:13.474 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.474 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.474 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.474 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.474 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.474 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.474 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.475 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.475 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.475 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.475 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.475 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.475 INFO:teuthology.orchestra.run.smithi117.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-19T03:42:13.475 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-19T03:42:13.475 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.475 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.475 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.475 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.475 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.475 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.475 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.475 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.476 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.476 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.476 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.476 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.476 INFO:teuthology.orchestra.run.smithi117.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-19T03:42:13.476 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-19T03:42:13.476 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.476 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.476 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.476 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.476 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.476 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.476 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.476 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.476 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.477 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.477 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.477 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.477 INFO:teuthology.orchestra.run.smithi117.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-19T03:42:13.477 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-19T03:42:13.477 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.477 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.477 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.477 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.477 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.477 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.477 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.477 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.477 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.478 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.478 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.478 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.478 INFO:teuthology.orchestra.run.smithi117.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-19T03:42:13.478 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-19T03:42:13.478 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.478 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.478 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.478 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.478 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.478 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.478 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.478 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.478 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.478 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.479 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.479 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.479 INFO:teuthology.orchestra.run.smithi117.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-19T03:42:13.479 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-19T03:42:13.479 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.479 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.479 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.479 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.479 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.479 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.479 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.479 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.479 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.479 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.479 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.480 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.480 INFO:teuthology.orchestra.run.smithi117.stdout: "GRAFANA_API_PASSWORD": { 2024-03-19T03:42:13.480 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-19T03:42:13.480 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.480 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.480 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.480 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "admin", 2024-03-19T03:42:13.480 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.480 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.480 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.480 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.480 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.480 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.480 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.480 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.481 INFO:teuthology.orchestra.run.smithi117.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-19T03:42:13.481 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-19T03:42:13.481 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.481 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.481 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.481 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.481 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.481 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.481 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.481 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.481 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.481 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.481 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.481 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.481 INFO:teuthology.orchestra.run.smithi117.stdout: "GRAFANA_API_URL": { 2024-03-19T03:42:13.482 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "GRAFANA_API_URL", 2024-03-19T03:42:13.482 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.482 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.482 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.482 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.482 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.482 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.482 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.482 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.482 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.482 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.482 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.482 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.483 INFO:teuthology.orchestra.run.smithi117.stdout: "GRAFANA_API_USERNAME": { 2024-03-19T03:42:13.483 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-19T03:42:13.483 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.483 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.483 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.483 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "admin", 2024-03-19T03:42:13.483 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.483 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.483 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.483 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.483 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.483 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.483 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.483 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.484 INFO:teuthology.orchestra.run.smithi117.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-19T03:42:13.484 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-19T03:42:13.484 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.484 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.484 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.484 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.484 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.484 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.484 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.484 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.484 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.484 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.484 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.484 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.485 INFO:teuthology.orchestra.run.smithi117.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-19T03:42:13.485 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-19T03:42:13.485 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.485 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.485 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.485 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.485 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.485 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.485 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.485 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.485 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.485 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.485 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.485 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.485 INFO:teuthology.orchestra.run.smithi117.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-19T03:42:13.486 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-19T03:42:13.486 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.486 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.486 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.486 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.486 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.486 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.486 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.486 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.486 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.486 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.486 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.486 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.486 INFO:teuthology.orchestra.run.smithi117.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-19T03:42:13.487 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-19T03:42:13.487 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.487 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.487 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.487 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.487 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.487 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.487 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.487 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.487 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.487 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.487 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.487 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.487 INFO:teuthology.orchestra.run.smithi117.stdout: "MULTICLUSTER_CONFIG": { 2024-03-19T03:42:13.487 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-19T03:42:13.488 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.488 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.488 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.488 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "{}", 2024-03-19T03:42:13.488 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.488 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.488 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.488 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.488 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.488 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.488 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.488 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.488 INFO:teuthology.orchestra.run.smithi117.stdout: "PROMETHEUS_API_HOST": { 2024-03-19T03:42:13.488 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-19T03:42:13.488 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.489 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.489 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.489 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.489 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.489 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.489 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.489 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.489 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.489 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.489 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.489 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.489 INFO:teuthology.orchestra.run.smithi117.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-19T03:42:13.489 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-19T03:42:13.489 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.489 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.489 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.490 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.490 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.490 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.490 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.490 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.490 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.490 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.490 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.490 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.490 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-19T03:42:13.490 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-19T03:42:13.490 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.490 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.490 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.491 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.491 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.491 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.491 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.491 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.491 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.491 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.491 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.491 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.491 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-19T03:42:13.491 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-19T03:42:13.491 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.491 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.491 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.491 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.491 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.491 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.492 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.492 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.492 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.492 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.492 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.492 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.492 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-19T03:42:13.492 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-19T03:42:13.492 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.492 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.492 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.492 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.492 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.492 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.492 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.492 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.493 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.493 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.493 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.493 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.493 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-19T03:42:13.493 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-19T03:42:13.493 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.493 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.493 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.493 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.493 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.493 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.493 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.493 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.493 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.493 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.494 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.494 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.494 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-19T03:42:13.494 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-19T03:42:13.494 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.494 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.494 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.494 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.494 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.494 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.494 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.494 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.494 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.494 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.494 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.495 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.495 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-19T03:42:13.495 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-19T03:42:13.495 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.495 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.495 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.495 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.495 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.495 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.495 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.495 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.495 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.495 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.495 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.496 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.496 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-19T03:42:13.496 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-19T03:42:13.496 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.496 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.496 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.496 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.496 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.496 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.496 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.496 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.496 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.496 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.497 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.497 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.497 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_ENABLED": { 2024-03-19T03:42:13.497 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-19T03:42:13.497 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.497 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.497 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.497 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.497 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.497 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.497 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.497 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.497 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.497 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.498 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.498 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.498 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-19T03:42:13.498 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-19T03:42:13.498 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.498 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.498 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.498 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-19T03:42:13.498 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.498 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.498 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.498 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.498 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.498 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.499 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.499 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.499 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-19T03:42:13.499 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-19T03:42:13.499 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.499 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.499 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.499 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10", 2024-03-19T03:42:13.499 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.499 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.499 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.499 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.499 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.499 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.499 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.499 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.500 INFO:teuthology.orchestra.run.smithi117.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-19T03:42:13.500 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-19T03:42:13.500 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.500 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.500 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.500 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "8", 2024-03-19T03:42:13.500 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.500 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.500 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.500 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.500 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.500 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.501 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.501 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.501 INFO:teuthology.orchestra.run.smithi117.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-19T03:42:13.501 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-19T03:42:13.501 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.501 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.501 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.501 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "45", 2024-03-19T03:42:13.501 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.501 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.501 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.501 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.501 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.501 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.501 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.502 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.502 INFO:teuthology.orchestra.run.smithi117.stdout: "RGW_API_ACCESS_KEY": { 2024-03-19T03:42:13.502 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-19T03:42:13.502 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.502 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.502 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.502 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.502 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.502 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.502 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.502 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.502 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.502 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.503 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.503 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.503 INFO:teuthology.orchestra.run.smithi117.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-19T03:42:13.503 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-19T03:42:13.503 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.503 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.503 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.503 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "admin", 2024-03-19T03:42:13.503 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.503 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.503 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.503 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.503 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.503 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.504 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.504 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.504 INFO:teuthology.orchestra.run.smithi117.stdout: "RGW_API_SECRET_KEY": { 2024-03-19T03:42:13.504 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-19T03:42:13.504 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.504 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.504 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.504 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.504 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.504 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.504 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.504 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.504 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.504 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.505 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.505 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.505 INFO:teuthology.orchestra.run.smithi117.stdout: "RGW_API_SSL_VERIFY": { 2024-03-19T03:42:13.505 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-19T03:42:13.505 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.505 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.505 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.505 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.505 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.505 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.505 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.505 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.505 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.505 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.506 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.506 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.506 INFO:teuthology.orchestra.run.smithi117.stdout: "UNSAFE_TLS_v1_2": { 2024-03-19T03:42:13.506 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-19T03:42:13.506 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.506 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.506 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.506 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.506 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.506 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.506 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.506 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.506 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.506 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.507 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.507 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.507 INFO:teuthology.orchestra.run.smithi117.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-19T03:42:13.507 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-19T03:42:13.507 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.507 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.507 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.507 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "0", 2024-03-19T03:42:13.507 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.507 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.507 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.507 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.507 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.507 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.508 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.508 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.508 INFO:teuthology.orchestra.run.smithi117.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-19T03:42:13.508 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-19T03:42:13.508 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.508 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.508 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.508 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10", 2024-03-19T03:42:13.508 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.508 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.508 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.508 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.508 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.508 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.509 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.509 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.509 INFO:teuthology.orchestra.run.smithi117.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-19T03:42:13.509 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-19T03:42:13.509 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.509 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.509 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.509 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "5", 2024-03-19T03:42:13.509 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.509 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.509 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.509 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.509 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.509 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.509 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.509 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.510 INFO:teuthology.orchestra.run.smithi117.stdout: "cross_origin_url": { 2024-03-19T03:42:13.510 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "cross_origin_url", 2024-03-19T03:42:13.510 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.510 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.510 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.510 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.510 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.510 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.510 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.510 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.510 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.510 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.510 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.510 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.511 INFO:teuthology.orchestra.run.smithi117.stdout: "crt_file": { 2024-03-19T03:42:13.511 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "crt_file", 2024-03-19T03:42:13.511 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.511 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.511 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.511 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.511 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.511 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.511 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.511 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.511 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.511 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.511 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.511 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.511 INFO:teuthology.orchestra.run.smithi117.stdout: "debug": { 2024-03-19T03:42:13.512 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "debug", 2024-03-19T03:42:13.512 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.512 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.512 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.512 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.512 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.512 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.512 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.512 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Enable/disable debug options", 2024-03-19T03:42:13.512 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.512 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.512 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.512 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.512 INFO:teuthology.orchestra.run.smithi117.stdout: "jwt_token_ttl": { 2024-03-19T03:42:13.512 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "jwt_token_ttl", 2024-03-19T03:42:13.512 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.513 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.513 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.513 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "28800", 2024-03-19T03:42:13.513 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.513 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.513 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.513 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.513 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.513 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.513 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.513 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.513 INFO:teuthology.orchestra.run.smithi117.stdout: "key_file": { 2024-03-19T03:42:13.513 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "key_file", 2024-03-19T03:42:13.513 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.513 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.513 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.514 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.514 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.514 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.514 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.514 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.514 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.514 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.514 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.514 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.514 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.514 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.514 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.514 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.514 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.514 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.515 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.515 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.515 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.515 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.515 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.515 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.515 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.515 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.515 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.515 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.515 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.515 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.515 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.515 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.515 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.515 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.516 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.516 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.516 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.516 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.516 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.516 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.516 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.516 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.516 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.516 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.516 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.516 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.516 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.516 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.516 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.517 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.517 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.517 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.517 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.517 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.517 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.517 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.517 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.517 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.517 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.517 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.517 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.517 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.517 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.517 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.518 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.518 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.518 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.518 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.518 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.518 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.518 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.518 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.518 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.518 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.518 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.518 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.518 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.518 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.518 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.519 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.519 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.519 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.519 INFO:teuthology.orchestra.run.smithi117.stdout: "motd": { 2024-03-19T03:42:13.519 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "motd", 2024-03-19T03:42:13.519 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.519 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.519 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.519 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.519 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.519 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.519 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.519 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "The message of the day", 2024-03-19T03:42:13.519 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.519 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.519 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.520 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.520 INFO:teuthology.orchestra.run.smithi117.stdout: "redirect_resolve_ip_addr": { 2024-03-19T03:42:13.520 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "redirect_resolve_ip_addr", 2024-03-19T03:42:13.520 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.520 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.520 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.520 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.520 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.520 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.520 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.520 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.520 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.520 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.520 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.521 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.521 INFO:teuthology.orchestra.run.smithi117.stdout: "server_addr": { 2024-03-19T03:42:13.521 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "server_addr", 2024-03-19T03:42:13.521 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.521 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.521 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.521 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "::", 2024-03-19T03:42:13.521 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.521 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.521 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.521 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.521 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.521 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.522 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.522 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.522 INFO:teuthology.orchestra.run.smithi117.stdout: "server_port": { 2024-03-19T03:42:13.522 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "server_port", 2024-03-19T03:42:13.522 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.522 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.522 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.522 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "8080", 2024-03-19T03:42:13.522 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.522 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.522 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.522 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.523 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.523 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.523 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.523 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.523 INFO:teuthology.orchestra.run.smithi117.stdout: "ssl": { 2024-03-19T03:42:13.523 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ssl", 2024-03-19T03:42:13.523 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.523 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.523 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.523 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.523 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.524 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.524 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.524 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.524 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.524 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.524 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.524 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.524 INFO:teuthology.orchestra.run.smithi117.stdout: "ssl_server_port": { 2024-03-19T03:42:13.524 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ssl_server_port", 2024-03-19T03:42:13.524 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.524 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.524 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.524 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "8443", 2024-03-19T03:42:13.524 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.524 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.524 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.524 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.525 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.525 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.525 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.525 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.525 INFO:teuthology.orchestra.run.smithi117.stdout: "standby_behaviour": { 2024-03-19T03:42:13.525 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "standby_behaviour", 2024-03-19T03:42:13.525 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.525 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.525 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.525 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "redirect", 2024-03-19T03:42:13.525 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.525 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.525 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.525 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.525 INFO:teuthology.orchestra.run.smithi117.stdout: "redirect" 2024-03-19T03:42:13.525 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.526 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.526 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.526 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.526 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.526 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.526 INFO:teuthology.orchestra.run.smithi117.stdout: "standby_error_status_code": { 2024-03-19T03:42:13.526 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "standby_error_status_code", 2024-03-19T03:42:13.526 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.526 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.526 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.526 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "500", 2024-03-19T03:42:13.526 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "400", 2024-03-19T03:42:13.526 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "599", 2024-03-19T03:42:13.526 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.526 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.527 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.527 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.527 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.527 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.527 INFO:teuthology.orchestra.run.smithi117.stdout: "url_prefix": { 2024-03-19T03:42:13.527 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "url_prefix", 2024-03-19T03:42:13.527 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.527 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.527 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.527 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.527 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.527 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.528 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.528 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.528 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.528 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.528 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.528 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.528 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.528 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.528 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.528 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "devicehealth", 2024-03-19T03:42:13.528 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.528 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.529 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.529 INFO:teuthology.orchestra.run.smithi117.stdout: "enable_monitoring": { 2024-03-19T03:42:13.529 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "enable_monitoring", 2024-03-19T03:42:13.529 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.529 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.529 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.529 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.529 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.529 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.529 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.529 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "monitor device health metrics", 2024-03-19T03:42:13.529 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.529 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.530 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.530 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.530 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.530 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.530 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.530 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.530 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.530 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.530 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.530 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.530 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.530 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.530 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.531 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.531 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.531 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.531 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.531 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.531 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.531 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.531 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.531 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.531 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.531 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.531 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.531 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.532 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.532 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.532 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.532 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.532 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.532 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.532 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.532 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.532 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.532 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.532 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.532 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.532 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.532 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.533 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.533 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.533 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.533 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.533 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.533 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.533 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.533 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.533 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.533 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.533 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.533 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.533 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.533 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.534 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.534 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.534 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.534 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.534 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.534 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.534 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.534 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.534 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.535 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.535 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.535 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.535 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.535 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.535 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.535 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.535 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.535 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.535 INFO:teuthology.orchestra.run.smithi117.stdout: "mark_out_threshold": { 2024-03-19T03:42:13.535 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "mark_out_threshold", 2024-03-19T03:42:13.535 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.536 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.536 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.536 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "2419200", 2024-03-19T03:42:13.536 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.536 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.536 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.536 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-19T03:42:13.536 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.536 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.536 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.536 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.537 INFO:teuthology.orchestra.run.smithi117.stdout: "pool_name": { 2024-03-19T03:42:13.537 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "pool_name", 2024-03-19T03:42:13.537 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.537 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.537 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.537 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "device_health_metrics", 2024-03-19T03:42:13.537 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.537 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.537 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.537 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-19T03:42:13.537 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.537 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.537 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.537 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.537 INFO:teuthology.orchestra.run.smithi117.stdout: "retention_period": { 2024-03-19T03:42:13.538 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "retention_period", 2024-03-19T03:42:13.538 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.538 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.538 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.538 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "15552000", 2024-03-19T03:42:13.538 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.538 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.538 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.538 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "how long to retain device health metrics", 2024-03-19T03:42:13.538 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.538 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.538 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.538 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.538 INFO:teuthology.orchestra.run.smithi117.stdout: "scrape_frequency": { 2024-03-19T03:42:13.539 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "scrape_frequency", 2024-03-19T03:42:13.539 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.539 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.539 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.539 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "86400", 2024-03-19T03:42:13.539 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.539 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.539 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.539 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-19T03:42:13.539 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.539 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.539 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.539 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.539 INFO:teuthology.orchestra.run.smithi117.stdout: "self_heal": { 2024-03-19T03:42:13.539 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "self_heal", 2024-03-19T03:42:13.540 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.540 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.540 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.540 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.540 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.540 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.540 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.540 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-19T03:42:13.540 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.540 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.540 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.540 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.540 INFO:teuthology.orchestra.run.smithi117.stdout: "sleep_interval": { 2024-03-19T03:42:13.540 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "sleep_interval", 2024-03-19T03:42:13.540 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.541 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.541 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.541 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "600", 2024-03-19T03:42:13.541 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.541 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.541 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.541 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "how frequently to wake up and check device health", 2024-03-19T03:42:13.541 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.541 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.541 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.541 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.541 INFO:teuthology.orchestra.run.smithi117.stdout: "warn_threshold": { 2024-03-19T03:42:13.541 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "warn_threshold", 2024-03-19T03:42:13.541 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.542 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.542 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.542 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "7257600", 2024-03-19T03:42:13.542 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.542 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.542 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.542 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-19T03:42:13.542 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.542 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.542 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.542 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.542 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.542 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.542 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.543 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "diskprediction_local", 2024-03-19T03:42:13.543 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.543 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.543 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.543 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.543 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.543 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.543 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.543 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.543 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.543 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.543 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.543 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.543 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.544 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.544 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.544 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.544 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.544 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.544 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.544 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.544 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.544 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.544 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.544 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.544 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.544 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.544 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.545 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.545 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.545 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.545 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.545 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.545 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.545 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.545 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.545 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.545 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.545 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.545 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.545 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.545 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.546 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.546 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.546 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.546 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.546 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.546 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.546 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.546 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.546 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.546 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.546 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.546 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.547 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.547 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.547 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.547 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.547 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.547 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.547 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.547 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.547 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.547 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.547 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.547 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.547 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.547 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.548 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.548 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.548 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.548 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.548 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.548 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.548 INFO:teuthology.orchestra.run.smithi117.stdout: "predict_interval": { 2024-03-19T03:42:13.548 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "predict_interval", 2024-03-19T03:42:13.548 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.548 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.548 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.548 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "86400", 2024-03-19T03:42:13.548 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.548 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.548 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.549 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.549 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.549 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.549 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.549 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.549 INFO:teuthology.orchestra.run.smithi117.stdout: "predictor_model": { 2024-03-19T03:42:13.549 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "predictor_model", 2024-03-19T03:42:13.549 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.549 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.549 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.549 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "prophetstor", 2024-03-19T03:42:13.550 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.550 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.550 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.550 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.550 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.550 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.550 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.550 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.550 INFO:teuthology.orchestra.run.smithi117.stdout: "sleep_interval": { 2024-03-19T03:42:13.550 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "sleep_interval", 2024-03-19T03:42:13.551 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.551 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.551 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.551 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "600", 2024-03-19T03:42:13.551 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.551 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.551 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.551 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.551 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.551 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.551 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.551 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.551 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.552 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.552 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.552 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "influx", 2024-03-19T03:42:13.552 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": false, 2024-03-19T03:42:13.552 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "influxdb python module not found", 2024-03-19T03:42:13.552 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.552 INFO:teuthology.orchestra.run.smithi117.stdout: "batch_size": { 2024-03-19T03:42:13.552 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "batch_size", 2024-03-19T03:42:13.552 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.552 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.552 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.552 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "5000", 2024-03-19T03:42:13.553 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.553 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.553 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.553 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-19T03:42:13.553 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.553 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.553 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.553 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.553 INFO:teuthology.orchestra.run.smithi117.stdout: "database": { 2024-03-19T03:42:13.553 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "database", 2024-03-19T03:42:13.553 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.553 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.553 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.554 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "ceph", 2024-03-19T03:42:13.554 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.554 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.554 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.554 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-19T03:42:13.554 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.554 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.554 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.554 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.554 INFO:teuthology.orchestra.run.smithi117.stdout: "hostname": { 2024-03-19T03:42:13.554 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "hostname", 2024-03-19T03:42:13.554 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.555 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.555 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.555 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.555 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.555 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.555 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.555 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "InfluxDB server hostname", 2024-03-19T03:42:13.555 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.555 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.555 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.555 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.555 INFO:teuthology.orchestra.run.smithi117.stdout: "interval": { 2024-03-19T03:42:13.556 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "interval", 2024-03-19T03:42:13.556 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.556 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.556 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.556 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "30", 2024-03-19T03:42:13.556 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "5", 2024-03-19T03:42:13.556 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.556 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.556 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-19T03:42:13.556 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.556 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.556 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.556 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.556 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.557 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.557 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.557 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.557 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.557 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.557 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.557 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.557 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.557 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.557 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.557 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.557 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.557 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.557 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.558 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.558 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.558 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.558 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.558 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.558 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.558 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.558 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.558 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.558 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.558 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.558 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.558 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.558 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.559 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.559 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.559 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.559 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.559 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.559 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.559 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.559 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.559 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.559 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.559 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.559 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.559 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.559 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.559 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.560 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.560 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.560 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.560 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.560 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.560 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.560 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.560 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.560 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.560 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.560 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.560 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.560 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.560 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.560 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.561 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.561 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.561 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.561 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.561 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.561 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.561 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.561 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.561 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.561 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.561 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.561 INFO:teuthology.orchestra.run.smithi117.stdout: "password": { 2024-03-19T03:42:13.561 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "password", 2024-03-19T03:42:13.561 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.561 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.562 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.562 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.562 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.562 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.562 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.562 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "password of InfluxDB server user", 2024-03-19T03:42:13.562 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.562 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.562 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.562 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.562 INFO:teuthology.orchestra.run.smithi117.stdout: "port": { 2024-03-19T03:42:13.562 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "port", 2024-03-19T03:42:13.563 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.563 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.563 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.563 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "8086", 2024-03-19T03:42:13.563 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.563 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.563 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.563 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "InfluxDB server port", 2024-03-19T03:42:13.563 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.563 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.564 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.564 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.564 INFO:teuthology.orchestra.run.smithi117.stdout: "ssl": { 2024-03-19T03:42:13.564 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ssl", 2024-03-19T03:42:13.564 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.564 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.565 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.565 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "false", 2024-03-19T03:42:13.565 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.565 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.565 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.565 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-19T03:42:13.565 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.566 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.566 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.566 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.566 INFO:teuthology.orchestra.run.smithi117.stdout: "threads": { 2024-03-19T03:42:13.566 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "threads", 2024-03-19T03:42:13.566 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.566 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.566 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.566 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "5", 2024-03-19T03:42:13.566 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "1", 2024-03-19T03:42:13.567 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "32", 2024-03-19T03:42:13.567 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.567 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-19T03:42:13.567 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.567 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.567 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.567 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.567 INFO:teuthology.orchestra.run.smithi117.stdout: "username": { 2024-03-19T03:42:13.567 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "username", 2024-03-19T03:42:13.567 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.568 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.568 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.568 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.568 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.568 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.568 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.568 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "username of InfluxDB server user", 2024-03-19T03:42:13.568 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.568 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.568 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.568 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.569 INFO:teuthology.orchestra.run.smithi117.stdout: "verify_ssl": { 2024-03-19T03:42:13.569 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "verify_ssl", 2024-03-19T03:42:13.569 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.569 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.569 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.569 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "true", 2024-03-19T03:42:13.569 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.569 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.569 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.569 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-19T03:42:13.569 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.570 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.570 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.570 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.570 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.570 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.570 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.570 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "insights", 2024-03-19T03:42:13.570 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.570 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.570 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.570 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.571 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.571 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.571 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.571 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.571 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.571 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.571 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.571 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.571 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.572 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.572 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.572 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.572 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.572 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.572 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.572 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.572 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.572 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.572 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.572 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.572 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.572 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.572 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.572 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.573 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.573 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.573 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.573 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.573 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.573 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.573 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.573 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.573 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.573 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.573 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.573 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.573 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.573 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.573 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.574 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.574 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.574 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.574 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.574 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.574 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.574 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.574 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.574 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.574 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.574 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.574 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.574 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.574 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.575 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.575 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.575 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.575 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.575 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.575 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.575 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.575 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.575 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.575 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.575 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.575 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.575 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.575 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.575 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.576 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.576 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.576 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.576 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.576 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "iostat", 2024-03-19T03:42:13.576 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.576 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.576 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.576 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.576 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.576 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.576 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.576 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.576 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.577 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.577 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.577 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.577 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.577 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.577 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.577 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.577 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.577 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.577 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.577 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.577 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.577 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.578 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.578 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.578 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.578 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.578 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.578 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.578 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.578 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.578 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.578 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.579 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.579 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.579 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.579 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.579 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.579 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.579 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.579 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.579 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.579 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.579 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.580 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.580 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.580 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.580 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.580 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.580 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.580 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.580 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.580 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.580 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.580 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.581 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.581 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.581 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.581 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.581 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.581 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.581 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.581 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.581 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.581 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.581 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.582 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.582 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.582 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.582 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.582 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.582 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.582 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.582 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.582 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.582 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.582 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.583 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "k8sevents", 2024-03-19T03:42:13.583 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.583 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.583 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.583 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_event_retention_days": { 2024-03-19T03:42:13.583 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "ceph_event_retention_days", 2024-03-19T03:42:13.583 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.583 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.583 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.583 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "7", 2024-03-19T03:42:13.584 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.584 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.584 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.584 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-19T03:42:13.584 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.584 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.584 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.584 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.584 INFO:teuthology.orchestra.run.smithi117.stdout: "config_check_secs": { 2024-03-19T03:42:13.585 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "config_check_secs", 2024-03-19T03:42:13.585 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.585 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.585 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.585 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10", 2024-03-19T03:42:13.585 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "10", 2024-03-19T03:42:13.585 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.585 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.586 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-19T03:42:13.586 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.586 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.586 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.586 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.586 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.586 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.586 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.587 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.587 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.587 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.587 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.587 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.587 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.587 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.587 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.587 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.587 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.587 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.587 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.587 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.587 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.588 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.588 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.588 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.588 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.588 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.588 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.588 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.588 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.588 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.588 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.588 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.588 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.588 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.588 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.589 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.589 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.589 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.589 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.589 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.589 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.589 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.589 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.589 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.589 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.589 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.589 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.589 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.590 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.590 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.590 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.590 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.590 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.590 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.590 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.590 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.590 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.590 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.590 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.590 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.590 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.591 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.591 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.591 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.591 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.591 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.591 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.591 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.591 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.591 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.591 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.591 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.591 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.591 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.591 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.592 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.592 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.592 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "localpool", 2024-03-19T03:42:13.592 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.592 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.592 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.592 INFO:teuthology.orchestra.run.smithi117.stdout: "failure_domain": { 2024-03-19T03:42:13.592 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "failure_domain", 2024-03-19T03:42:13.592 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.592 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.592 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.592 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "host", 2024-03-19T03:42:13.592 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.592 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.593 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.593 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "failure domain for any created local pool", 2024-03-19T03:42:13.593 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-19T03:42:13.593 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.593 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.593 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.593 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.593 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.593 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.593 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.593 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.594 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.594 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.594 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.594 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.594 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.594 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.594 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.594 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.594 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.595 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.595 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.595 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.595 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.595 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.595 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.595 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.595 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.595 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.595 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.596 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.596 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.596 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.596 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.596 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.596 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.596 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.596 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.596 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.596 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.597 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.597 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.597 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.597 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.597 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.597 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.597 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.597 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.597 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.597 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.598 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.598 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.598 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.598 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.598 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.598 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.598 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.598 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.598 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.598 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.598 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.599 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.599 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.599 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.599 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.599 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.599 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.599 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.599 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.599 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.599 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.599 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.599 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.599 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.599 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.599 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.600 INFO:teuthology.orchestra.run.smithi117.stdout: "min_size": { 2024-03-19T03:42:13.600 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "min_size", 2024-03-19T03:42:13.600 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.600 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.600 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.600 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.600 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.600 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.600 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.600 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "default min_size for any created local pool", 2024-03-19T03:42:13.600 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-19T03:42:13.600 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.600 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.600 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.600 INFO:teuthology.orchestra.run.smithi117.stdout: "num_rep": { 2024-03-19T03:42:13.601 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "num_rep", 2024-03-19T03:42:13.601 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.601 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.601 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.601 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "3", 2024-03-19T03:42:13.601 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.601 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.601 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.601 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "default replica count for any created local pool", 2024-03-19T03:42:13.601 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.601 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.601 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.601 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.601 INFO:teuthology.orchestra.run.smithi117.stdout: "pg_num": { 2024-03-19T03:42:13.601 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "pg_num", 2024-03-19T03:42:13.602 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.602 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.602 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.602 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "128", 2024-03-19T03:42:13.602 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.602 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.602 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.602 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "default pg_num for any created local pool", 2024-03-19T03:42:13.602 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.602 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.602 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.602 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.602 INFO:teuthology.orchestra.run.smithi117.stdout: "prefix": { 2024-03-19T03:42:13.602 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "prefix", 2024-03-19T03:42:13.602 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.603 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.603 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.603 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.603 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.603 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.603 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.603 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "name prefix for any created local pool", 2024-03-19T03:42:13.603 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.603 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.603 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.603 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.603 INFO:teuthology.orchestra.run.smithi117.stdout: "subtree": { 2024-03-19T03:42:13.603 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "subtree", 2024-03-19T03:42:13.603 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.603 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.604 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.604 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "rack", 2024-03-19T03:42:13.604 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.604 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.604 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.604 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-19T03:42:13.604 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-19T03:42:13.604 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.604 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.604 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.604 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.604 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.604 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.604 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "mds_autoscaler", 2024-03-19T03:42:13.604 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.604 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.605 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.605 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.605 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.605 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.605 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.605 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.605 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.605 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.605 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.605 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.605 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.605 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.605 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.606 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.606 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.606 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.606 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.606 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.606 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.606 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.606 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.606 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.606 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.606 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.606 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.606 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.607 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.607 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.607 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.607 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.607 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.607 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.607 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.607 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.607 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.607 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.608 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.608 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.608 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.608 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.608 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.608 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.608 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.608 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.608 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.608 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.608 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.608 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.608 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.609 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.609 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.609 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.609 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.609 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.609 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.609 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.609 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.609 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.609 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.609 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.610 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.610 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.610 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.610 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.610 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.610 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.610 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.610 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.610 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.610 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.610 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.610 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.610 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.611 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.611 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "mirroring", 2024-03-19T03:42:13.611 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.611 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.611 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.611 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.611 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.611 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.611 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.611 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.611 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.611 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.611 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.611 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.611 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.612 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.612 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.612 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.612 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.612 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.612 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.612 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.612 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.612 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.612 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.612 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.612 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.612 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.612 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.612 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.613 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.613 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.613 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.613 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.613 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.613 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.613 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.613 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.613 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.613 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.613 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.613 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.613 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.613 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.613 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.614 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.614 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.614 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.614 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.614 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.614 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.614 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.614 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.614 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.614 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.614 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.614 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.614 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.614 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.614 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.615 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.615 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.615 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.615 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.615 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.615 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.615 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.615 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.615 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.615 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.615 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.615 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.615 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.616 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.616 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.616 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.616 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.616 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.616 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "nfs", 2024-03-19T03:42:13.616 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.616 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.616 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.616 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.616 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.616 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.616 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.616 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.616 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.617 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.617 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.617 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.617 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.617 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.617 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.617 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.617 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.617 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.617 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.617 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.617 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.617 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.618 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.618 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.618 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.618 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.618 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.618 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.618 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.618 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.618 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.618 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.618 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.618 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.618 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.618 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.619 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.619 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.619 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.619 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.619 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.619 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.619 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.619 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.619 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.619 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.619 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.619 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.619 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.620 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.620 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.620 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.620 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.620 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.620 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.620 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.620 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.620 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.620 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.620 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.620 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.620 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.620 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.621 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.621 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.621 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.621 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.621 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.621 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.621 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.621 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.621 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.621 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.621 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.621 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.621 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.622 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "orchestrator", 2024-03-19T03:42:13.622 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.622 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.622 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.622 INFO:teuthology.orchestra.run.smithi117.stdout: "fail_fs": { 2024-03-19T03:42:13.622 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "fail_fs", 2024-03-19T03:42:13.622 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.622 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.622 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.622 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.622 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.622 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.622 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.623 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-19T03:42:13.623 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.623 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.623 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.623 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.623 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.623 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.623 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.623 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.623 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.623 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.623 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.623 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.623 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.624 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.624 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.624 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.624 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.624 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.624 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.624 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.624 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.624 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.624 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.624 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.624 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.624 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.625 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.625 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.625 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.625 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.625 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.625 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.625 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.625 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.625 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.625 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.625 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.626 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.626 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.626 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.626 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.626 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.626 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.626 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.626 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.626 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.626 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.626 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.626 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.626 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.626 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.627 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.627 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.627 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.627 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.627 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.627 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.627 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.627 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.627 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.627 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.627 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.627 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.627 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.627 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.627 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.628 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.628 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.628 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.628 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.628 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.628 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.628 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.628 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.628 INFO:teuthology.orchestra.run.smithi117.stdout: "orchestrator": { 2024-03-19T03:42:13.628 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "orchestrator", 2024-03-19T03:42:13.628 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.628 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.628 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.628 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.628 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.629 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.629 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.629 INFO:teuthology.orchestra.run.smithi117.stdout: "cephadm", 2024-03-19T03:42:13.629 INFO:teuthology.orchestra.run.smithi117.stdout: "rook", 2024-03-19T03:42:13.629 INFO:teuthology.orchestra.run.smithi117.stdout: "test_orchestrator" 2024-03-19T03:42:13.629 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.629 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Orchestrator backend", 2024-03-19T03:42:13.629 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.629 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.629 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.629 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.629 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.629 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.630 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.630 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "osd_perf_query", 2024-03-19T03:42:13.630 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.630 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.630 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.630 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.630 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.630 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.630 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.630 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.630 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.631 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.631 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.631 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.631 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.631 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.631 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.631 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.631 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.631 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.631 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.631 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.631 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.631 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.631 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.631 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.632 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.632 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.632 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.632 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.632 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.632 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.632 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.632 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.632 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.632 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.632 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.632 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.632 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.632 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.632 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.633 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.633 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.633 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.633 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.633 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.633 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.633 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.633 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.633 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.633 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.633 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.633 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.633 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.634 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.634 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.634 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.634 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.634 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.634 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.634 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.634 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.634 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.634 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.634 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.634 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.634 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.634 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.635 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.635 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.635 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.635 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.635 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.635 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.635 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.635 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.635 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.635 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.635 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "osd_support", 2024-03-19T03:42:13.635 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.635 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.635 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.635 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.636 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.636 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.636 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.636 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.636 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.636 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.636 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.636 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.636 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.636 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.636 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.636 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.636 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.637 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.637 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.637 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.637 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.637 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.637 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.637 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.637 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.637 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.637 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.637 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.637 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.637 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.637 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.637 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.638 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.638 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.638 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.638 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.638 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.638 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.638 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.638 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.638 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.638 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.638 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.638 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.639 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.639 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.639 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.639 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.639 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.639 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.639 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.639 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.639 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.639 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.639 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.639 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.639 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.639 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.639 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.640 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.640 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.640 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.640 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.640 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.640 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.640 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.640 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.640 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.640 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.640 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.640 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.640 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.640 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.641 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.641 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.641 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.641 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "pg_autoscaler", 2024-03-19T03:42:13.641 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.641 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.641 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.641 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.641 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.641 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.641 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.641 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.641 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.641 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.642 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.642 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.642 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.642 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.642 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.642 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.642 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.642 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.642 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.642 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.642 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.642 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.643 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.643 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.643 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.643 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.643 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.643 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.643 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.643 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.643 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.643 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.643 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.643 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.643 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.644 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.644 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.644 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.644 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.644 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.644 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.644 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.644 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.644 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.644 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.644 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.644 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.644 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.644 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.644 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.645 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.645 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.645 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.645 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.645 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.645 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.645 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.645 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.645 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.645 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.645 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.645 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.645 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.645 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.646 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.646 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.646 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.646 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.646 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.646 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.646 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.646 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.646 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.646 INFO:teuthology.orchestra.run.smithi117.stdout: "sleep_interval": { 2024-03-19T03:42:13.646 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "sleep_interval", 2024-03-19T03:42:13.646 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.646 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.647 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.647 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "60", 2024-03-19T03:42:13.647 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.647 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.647 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.647 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.647 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.647 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.647 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.647 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.647 INFO:teuthology.orchestra.run.smithi117.stdout: "threshold": { 2024-03-19T03:42:13.647 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "threshold", 2024-03-19T03:42:13.647 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "float", 2024-03-19T03:42:13.648 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.648 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.648 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "3.0", 2024-03-19T03:42:13.648 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "1.0", 2024-03-19T03:42:13.648 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.648 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.648 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "scaling threshold", 2024-03-19T03:42:13.648 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-19T03:42:13.648 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.648 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.648 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.648 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.648 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.648 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.649 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "progress", 2024-03-19T03:42:13.649 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.649 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.649 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.649 INFO:teuthology.orchestra.run.smithi117.stdout: "allow_pg_recovery_event": { 2024-03-19T03:42:13.649 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "allow_pg_recovery_event", 2024-03-19T03:42:13.649 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.649 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.649 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.649 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.649 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.649 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.649 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.649 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-19T03:42:13.649 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.650 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.650 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.650 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.650 INFO:teuthology.orchestra.run.smithi117.stdout: "enabled": { 2024-03-19T03:42:13.650 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "enabled", 2024-03-19T03:42:13.650 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.650 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.650 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.650 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.650 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.650 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.650 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.650 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.650 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.650 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.651 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.651 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.651 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.651 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.651 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.651 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.651 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.651 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.651 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.651 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.651 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.651 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.651 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.651 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.651 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.652 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.652 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.652 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.652 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.652 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.652 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.652 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.652 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.652 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.652 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.652 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.652 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.652 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.652 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.652 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.653 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.653 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.653 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.653 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.653 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.653 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.653 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.653 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.653 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.653 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.653 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.653 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.653 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.653 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.653 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.654 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.654 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.654 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.654 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.654 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.654 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.654 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.654 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.654 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.654 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.654 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.654 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.654 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.654 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.654 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.655 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.655 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.655 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.655 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.655 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.655 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.655 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.655 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.655 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.655 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.655 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.655 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.655 INFO:teuthology.orchestra.run.smithi117.stdout: "max_completed_events": { 2024-03-19T03:42:13.655 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "max_completed_events", 2024-03-19T03:42:13.655 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.655 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.656 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.656 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "50", 2024-03-19T03:42:13.656 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.656 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.656 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.656 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "number of past completed events to remember", 2024-03-19T03:42:13.656 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.656 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.656 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.656 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.656 INFO:teuthology.orchestra.run.smithi117.stdout: "sleep_interval": { 2024-03-19T03:42:13.656 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "sleep_interval", 2024-03-19T03:42:13.656 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.656 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.656 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.657 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "5", 2024-03-19T03:42:13.657 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.657 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.657 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.657 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "how long the module is going to sleep", 2024-03-19T03:42:13.657 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.657 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.657 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.657 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.657 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.657 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.657 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.657 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "prometheus", 2024-03-19T03:42:13.657 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.657 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.657 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.658 INFO:teuthology.orchestra.run.smithi117.stdout: "cache": { 2024-03-19T03:42:13.658 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "cache", 2024-03-19T03:42:13.658 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.658 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.658 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.658 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.658 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.658 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.658 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.658 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.658 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.658 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.658 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.658 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.658 INFO:teuthology.orchestra.run.smithi117.stdout: "exclude_perf_counters": { 2024-03-19T03:42:13.659 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "exclude_perf_counters", 2024-03-19T03:42:13.659 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.659 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.659 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.659 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.659 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.659 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.659 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.659 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-19T03:42:13.659 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-19T03:42:13.659 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.659 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.659 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.659 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.660 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.660 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.660 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.660 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.660 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.660 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.660 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.660 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.660 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.660 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.660 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.660 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.660 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.660 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.660 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.661 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.661 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.661 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.661 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.661 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.661 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.661 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.661 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.661 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.661 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.661 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.661 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.661 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.661 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.661 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.662 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.662 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.662 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.662 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.662 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.662 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.662 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.662 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.662 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.662 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.662 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.662 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.662 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.662 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.663 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.663 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.663 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.663 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.663 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.663 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.663 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.663 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.663 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.663 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.663 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.663 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.663 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.663 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.663 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.664 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.664 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.664 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.664 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.664 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.664 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.664 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.664 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.664 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.664 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.664 INFO:teuthology.orchestra.run.smithi117.stdout: "rbd_stats_pools": { 2024-03-19T03:42:13.664 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rbd_stats_pools", 2024-03-19T03:42:13.664 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.664 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.664 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.664 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.665 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.665 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.665 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.665 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.665 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.665 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.665 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.665 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.665 INFO:teuthology.orchestra.run.smithi117.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-19T03:42:13.665 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-19T03:42:13.665 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.665 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.665 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.665 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "300", 2024-03-19T03:42:13.665 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.666 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.666 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.666 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.666 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.666 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.666 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.666 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.666 INFO:teuthology.orchestra.run.smithi117.stdout: "scrape_interval": { 2024-03-19T03:42:13.666 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "scrape_interval", 2024-03-19T03:42:13.666 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "float", 2024-03-19T03:42:13.666 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.666 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.666 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "15.0", 2024-03-19T03:42:13.667 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.667 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.667 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.667 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.667 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.667 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.667 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.667 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.667 INFO:teuthology.orchestra.run.smithi117.stdout: "server_addr": { 2024-03-19T03:42:13.667 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "server_addr", 2024-03-19T03:42:13.667 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.667 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.667 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.667 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "::", 2024-03-19T03:42:13.668 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.668 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.668 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.668 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-19T03:42:13.668 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.668 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.668 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.668 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.668 INFO:teuthology.orchestra.run.smithi117.stdout: "server_port": { 2024-03-19T03:42:13.668 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "server_port", 2024-03-19T03:42:13.668 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.668 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.668 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.668 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "9283", 2024-03-19T03:42:13.668 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.669 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.669 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.669 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-19T03:42:13.669 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.669 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.669 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.669 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.669 INFO:teuthology.orchestra.run.smithi117.stdout: "stale_cache_strategy": { 2024-03-19T03:42:13.669 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "stale_cache_strategy", 2024-03-19T03:42:13.669 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.669 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.669 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.669 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "log", 2024-03-19T03:42:13.669 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.670 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.670 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.670 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.670 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.670 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.670 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.670 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.670 INFO:teuthology.orchestra.run.smithi117.stdout: "standby_behaviour": { 2024-03-19T03:42:13.670 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "standby_behaviour", 2024-03-19T03:42:13.670 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.670 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.670 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.670 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "default", 2024-03-19T03:42:13.670 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.671 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.671 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.671 INFO:teuthology.orchestra.run.smithi117.stdout: "default", 2024-03-19T03:42:13.671 INFO:teuthology.orchestra.run.smithi117.stdout: "error" 2024-03-19T03:42:13.671 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.671 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.671 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.671 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.671 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.671 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.671 INFO:teuthology.orchestra.run.smithi117.stdout: "standby_error_status_code": { 2024-03-19T03:42:13.671 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "standby_error_status_code", 2024-03-19T03:42:13.671 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.671 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.671 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.671 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "500", 2024-03-19T03:42:13.672 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "400", 2024-03-19T03:42:13.672 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "599", 2024-03-19T03:42:13.672 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.672 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.672 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.672 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.672 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.672 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.672 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.672 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.672 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.672 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rbd_support", 2024-03-19T03:42:13.672 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.672 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.672 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.672 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.673 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.673 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.673 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.673 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.673 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.673 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.673 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.673 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.673 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.673 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.673 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.673 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.673 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.673 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.673 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.673 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.674 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.674 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.674 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.674 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.674 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.674 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.674 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.674 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.674 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.674 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.674 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.674 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.675 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.675 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.675 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.675 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.675 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.675 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.675 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.675 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.675 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.675 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.675 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.675 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.675 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.675 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.675 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.676 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.676 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.676 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.676 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.676 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.676 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.676 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.676 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.676 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.676 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.676 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.676 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.676 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.677 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.677 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.677 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.677 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.677 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.677 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.677 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.677 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.677 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.677 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.677 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.677 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.677 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.677 INFO:teuthology.orchestra.run.smithi117.stdout: "max_concurrent_snap_create": { 2024-03-19T03:42:13.677 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "max_concurrent_snap_create", 2024-03-19T03:42:13.678 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.678 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.678 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.678 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10", 2024-03-19T03:42:13.678 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.678 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.678 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.678 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.678 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.678 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.678 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.678 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.678 INFO:teuthology.orchestra.run.smithi117.stdout: "mirror_snapshot_schedule": { 2024-03-19T03:42:13.679 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "mirror_snapshot_schedule", 2024-03-19T03:42:13.679 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.679 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.679 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.679 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.679 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.679 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.679 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.679 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.679 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.679 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.679 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.679 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.679 INFO:teuthology.orchestra.run.smithi117.stdout: "trash_purge_schedule": { 2024-03-19T03:42:13.680 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "trash_purge_schedule", 2024-03-19T03:42:13.680 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.680 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.680 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.680 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.680 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.680 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.680 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.680 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.680 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.680 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.680 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.680 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.680 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.681 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.681 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.681 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "restful", 2024-03-19T03:42:13.681 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.681 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.681 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.681 INFO:teuthology.orchestra.run.smithi117.stdout: "enable_auth": { 2024-03-19T03:42:13.681 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "enable_auth", 2024-03-19T03:42:13.681 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.681 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.681 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.681 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.681 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.681 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.681 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.682 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.682 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.682 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.682 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.682 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.682 INFO:teuthology.orchestra.run.smithi117.stdout: "key_file": { 2024-03-19T03:42:13.682 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "key_file", 2024-03-19T03:42:13.682 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.682 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.682 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.682 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.682 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.682 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.682 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.683 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.683 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.683 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.683 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.683 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.683 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.683 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.683 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.683 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.683 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.683 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.683 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.683 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.683 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.684 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.684 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.684 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.684 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.684 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.684 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.684 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.684 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.684 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.684 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.684 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.684 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.684 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.684 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.684 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.685 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.685 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.685 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.685 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.685 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.685 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.685 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.685 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.685 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.685 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.685 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.685 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.685 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.685 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.686 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.686 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.686 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.686 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.686 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.686 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.686 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.686 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.686 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.686 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.686 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.686 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.686 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.687 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.687 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.687 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.687 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.687 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.687 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.687 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.687 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.687 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.687 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.687 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.687 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.687 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.687 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.687 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.687 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.688 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.688 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.688 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.688 INFO:teuthology.orchestra.run.smithi117.stdout: "server_addr": { 2024-03-19T03:42:13.688 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "server_addr", 2024-03-19T03:42:13.688 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.688 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.688 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.688 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.688 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.688 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.688 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.688 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.688 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.688 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.688 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.689 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.689 INFO:teuthology.orchestra.run.smithi117.stdout: "server_port": { 2024-03-19T03:42:13.689 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "server_port", 2024-03-19T03:42:13.689 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.689 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.689 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.689 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.689 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.689 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.689 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.689 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.689 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.689 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.689 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.689 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.689 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.690 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.690 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.690 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rgw", 2024-03-19T03:42:13.690 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.690 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.690 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.690 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.690 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.690 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.690 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.690 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.690 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.691 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.691 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.691 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.691 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.691 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.691 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.691 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.691 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.691 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.691 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.691 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.691 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.691 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.691 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.691 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.691 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.692 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.692 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.692 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.692 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.692 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.692 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.692 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.692 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.692 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.692 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.692 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.692 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.692 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.692 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.692 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.692 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.693 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.693 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.693 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.693 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.693 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.693 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.693 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.693 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.693 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.693 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.693 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.693 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.693 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.693 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.693 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.693 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.693 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.694 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.694 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.694 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.694 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.694 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.694 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.694 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.694 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.694 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.694 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.694 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.694 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.694 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.694 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.695 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.695 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.695 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.695 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.695 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rook", 2024-03-19T03:42:13.695 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.695 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.695 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.695 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.695 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.695 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.695 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.695 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.695 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.695 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.696 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.696 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.696 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.696 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.696 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.696 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.696 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.696 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.696 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.696 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.696 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.696 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.696 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.696 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.696 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.697 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.697 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.697 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.697 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.697 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.697 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.697 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.697 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.697 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.697 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.697 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.697 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.697 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.697 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.698 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.698 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.698 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.698 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.698 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.698 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.698 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.698 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.698 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.698 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.698 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.698 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.698 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.699 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.699 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.699 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.699 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.699 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.699 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.699 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.699 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.699 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.699 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.699 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.699 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.699 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.699 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.699 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.700 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.700 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.700 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.700 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.700 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.700 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.700 INFO:teuthology.orchestra.run.smithi117.stdout: "storage_class": { 2024-03-19T03:42:13.700 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "storage_class", 2024-03-19T03:42:13.700 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.700 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.700 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.700 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "local", 2024-03-19T03:42:13.700 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.700 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.701 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.701 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-19T03:42:13.701 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.701 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.701 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.701 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.701 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.701 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.701 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.701 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "selftest", 2024-03-19T03:42:13.701 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.701 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.701 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.701 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.702 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.702 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.702 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.702 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.702 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.702 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.702 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.702 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.702 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.702 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.702 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.702 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.702 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.702 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.703 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.703 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.703 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.703 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.703 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.703 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.703 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.703 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.703 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.703 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.703 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.703 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.703 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.703 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.704 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.704 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.704 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.704 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.704 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.704 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.704 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.704 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.704 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.704 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.704 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.704 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.704 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.704 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.705 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.705 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.705 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.705 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.705 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.705 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.705 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.705 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.705 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.705 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.705 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.705 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.705 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.705 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.705 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.706 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.706 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.706 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.706 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.706 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.706 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.706 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.706 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.706 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.706 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.706 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.706 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.706 INFO:teuthology.orchestra.run.smithi117.stdout: "roption1": { 2024-03-19T03:42:13.706 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "roption1", 2024-03-19T03:42:13.707 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.707 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.707 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.707 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.707 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.707 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.707 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.707 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.707 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.707 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.707 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.707 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.707 INFO:teuthology.orchestra.run.smithi117.stdout: "roption2": { 2024-03-19T03:42:13.707 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "roption2", 2024-03-19T03:42:13.707 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.708 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.708 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.708 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "xyz", 2024-03-19T03:42:13.708 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.708 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.708 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.708 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.708 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.708 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.708 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.708 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.708 INFO:teuthology.orchestra.run.smithi117.stdout: "rwoption1": { 2024-03-19T03:42:13.708 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rwoption1", 2024-03-19T03:42:13.708 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.708 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.709 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.709 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.709 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.709 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.709 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.709 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.709 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.709 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.709 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.709 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.709 INFO:teuthology.orchestra.run.smithi117.stdout: "rwoption2": { 2024-03-19T03:42:13.709 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rwoption2", 2024-03-19T03:42:13.709 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.709 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.710 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.710 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.710 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.710 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.710 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.710 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.710 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.710 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.710 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.710 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.710 INFO:teuthology.orchestra.run.smithi117.stdout: "rwoption3": { 2024-03-19T03:42:13.710 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rwoption3", 2024-03-19T03:42:13.710 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "float", 2024-03-19T03:42:13.710 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.711 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.711 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.711 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.711 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.711 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.711 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.711 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.711 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.711 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.711 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.711 INFO:teuthology.orchestra.run.smithi117.stdout: "rwoption4": { 2024-03-19T03:42:13.711 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rwoption4", 2024-03-19T03:42:13.711 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.711 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.711 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.711 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.712 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.712 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.712 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.712 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.712 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.712 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.712 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.712 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.712 INFO:teuthology.orchestra.run.smithi117.stdout: "rwoption5": { 2024-03-19T03:42:13.712 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rwoption5", 2024-03-19T03:42:13.712 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.712 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.712 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.712 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.712 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.713 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.713 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.713 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.713 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.713 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.713 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.713 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.713 INFO:teuthology.orchestra.run.smithi117.stdout: "rwoption6": { 2024-03-19T03:42:13.713 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rwoption6", 2024-03-19T03:42:13.713 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.713 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.713 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.713 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.713 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.713 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.713 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.714 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.714 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.714 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.714 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.714 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.714 INFO:teuthology.orchestra.run.smithi117.stdout: "rwoption7": { 2024-03-19T03:42:13.714 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rwoption7", 2024-03-19T03:42:13.714 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.714 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.714 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.714 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.714 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "1", 2024-03-19T03:42:13.714 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "42", 2024-03-19T03:42:13.714 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.715 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.715 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.715 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.715 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.715 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.715 INFO:teuthology.orchestra.run.smithi117.stdout: "testkey": { 2024-03-19T03:42:13.715 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "testkey", 2024-03-19T03:42:13.715 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.715 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.715 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.715 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.715 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.715 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.715 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.716 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.716 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.716 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.716 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.716 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.716 INFO:teuthology.orchestra.run.smithi117.stdout: "testlkey": { 2024-03-19T03:42:13.716 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "testlkey", 2024-03-19T03:42:13.716 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.716 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.716 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.716 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.716 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.716 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.716 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.716 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.716 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.717 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.717 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.717 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.717 INFO:teuthology.orchestra.run.smithi117.stdout: "testnewline": { 2024-03-19T03:42:13.717 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "testnewline", 2024-03-19T03:42:13.717 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.717 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.717 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.717 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.717 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.717 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.717 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.717 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.717 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.717 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.717 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.717 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.718 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.718 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.718 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.718 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "snap_schedule", 2024-03-19T03:42:13.718 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.718 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.718 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.718 INFO:teuthology.orchestra.run.smithi117.stdout: "allow_m_granularity": { 2024-03-19T03:42:13.718 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "allow_m_granularity", 2024-03-19T03:42:13.718 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.718 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.718 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.718 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.718 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.719 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.719 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.719 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "allow minute scheduled snapshots", 2024-03-19T03:42:13.719 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.719 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.719 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.719 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.719 INFO:teuthology.orchestra.run.smithi117.stdout: "dump_on_update": { 2024-03-19T03:42:13.719 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "dump_on_update", 2024-03-19T03:42:13.719 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.719 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.719 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.719 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.719 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.719 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.719 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.720 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "dump database to debug log on update", 2024-03-19T03:42:13.720 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.720 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.720 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.720 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.720 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.720 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.720 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.720 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.720 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.720 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.720 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.720 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.720 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.721 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.721 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.721 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.721 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.721 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.721 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.721 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.721 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.721 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.721 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.721 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.721 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.721 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.721 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.721 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.721 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.722 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.722 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.722 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.722 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.722 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.722 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.722 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.722 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.722 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.722 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.722 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.722 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.722 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.722 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.722 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.722 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.723 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.723 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.723 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.723 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.723 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.723 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.723 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.723 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.723 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.723 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.723 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.723 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.723 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.723 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.723 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.724 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.724 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.724 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.724 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.724 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.724 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.724 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.724 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.724 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.724 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.724 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.724 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.724 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.724 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.724 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.724 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.724 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.725 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "stats", 2024-03-19T03:42:13.725 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.725 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.725 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.725 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.725 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.725 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.725 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.725 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.725 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.725 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.725 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.725 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.725 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.725 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.725 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.726 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.726 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.726 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.726 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.726 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.726 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.726 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.726 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.726 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.726 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.726 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.726 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.726 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.726 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.726 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.727 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.727 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.727 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.727 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.727 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.727 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.727 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.727 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.727 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.727 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.727 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.727 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.727 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.728 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.728 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.728 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.728 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.728 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.728 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.728 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.728 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.728 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.728 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.728 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.728 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.728 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.728 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.729 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.729 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.729 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.729 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.729 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.729 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.729 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.729 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.729 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.729 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.729 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.729 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.729 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.729 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.729 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.730 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.730 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.730 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.730 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.730 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "status", 2024-03-19T03:42:13.730 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.730 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.730 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.730 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.730 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.730 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.730 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.730 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.730 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.731 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.731 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.731 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.731 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.731 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.731 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.731 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.731 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.731 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.731 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.731 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.731 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.732 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.732 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.732 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.732 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.732 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.732 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.732 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.732 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.732 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.732 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.732 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.732 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.732 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.732 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.732 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.733 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.733 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.733 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.733 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.733 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.733 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.733 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.733 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.733 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.733 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.733 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.733 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.733 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.734 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.734 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.734 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.734 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.734 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.734 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.734 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.734 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.734 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.734 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.734 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.734 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.734 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.734 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.735 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.735 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.735 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.735 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.735 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.735 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.735 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.735 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.735 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.735 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.735 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.735 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.735 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.735 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "telegraf", 2024-03-19T03:42:13.735 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.736 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.736 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.736 INFO:teuthology.orchestra.run.smithi117.stdout: "address": { 2024-03-19T03:42:13.736 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "address", 2024-03-19T03:42:13.736 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.736 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.736 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.736 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-19T03:42:13.736 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.736 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.736 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.736 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.736 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.736 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.736 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.737 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.737 INFO:teuthology.orchestra.run.smithi117.stdout: "interval": { 2024-03-19T03:42:13.737 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "interval", 2024-03-19T03:42:13.737 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.737 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.737 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.737 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "15", 2024-03-19T03:42:13.737 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.737 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.737 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.737 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.737 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.737 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.737 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.738 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.738 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.738 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.738 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.738 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.738 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.738 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.738 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.738 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.738 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.738 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.738 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.738 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.738 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.739 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.739 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.739 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.739 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.739 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.739 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.739 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.739 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.739 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.739 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.739 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.739 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.739 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.739 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.739 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.739 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.740 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.740 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.740 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.740 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.740 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.740 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.740 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.740 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.740 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.740 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.740 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.740 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.740 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.740 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.740 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.740 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.741 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.741 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.741 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.741 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.741 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.741 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.741 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.741 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.741 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.741 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.741 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.741 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.741 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.741 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.741 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.741 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.741 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.742 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.742 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.742 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.742 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.742 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.742 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.742 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.742 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.742 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.742 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.742 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.742 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "telemetry", 2024-03-19T03:42:13.742 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.742 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.743 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.743 INFO:teuthology.orchestra.run.smithi117.stdout: "channel_basic": { 2024-03-19T03:42:13.743 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "channel_basic", 2024-03-19T03:42:13.743 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.743 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.743 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.743 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.743 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.743 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.743 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.743 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-19T03:42:13.743 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.743 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.743 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.743 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.743 INFO:teuthology.orchestra.run.smithi117.stdout: "channel_crash": { 2024-03-19T03:42:13.743 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "channel_crash", 2024-03-19T03:42:13.744 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.744 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.744 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.744 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.744 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.744 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.744 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.744 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-19T03:42:13.744 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.744 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.744 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.744 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.744 INFO:teuthology.orchestra.run.smithi117.stdout: "channel_device": { 2024-03-19T03:42:13.744 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "channel_device", 2024-03-19T03:42:13.744 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.744 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.745 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.745 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.745 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.745 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.745 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.745 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-19T03:42:13.745 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.745 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.745 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.745 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.745 INFO:teuthology.orchestra.run.smithi117.stdout: "channel_ident": { 2024-03-19T03:42:13.745 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "channel_ident", 2024-03-19T03:42:13.745 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.745 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.745 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.745 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.746 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.746 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.746 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.746 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-19T03:42:13.746 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.746 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.746 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.746 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.746 INFO:teuthology.orchestra.run.smithi117.stdout: "channel_perf": { 2024-03-19T03:42:13.746 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "channel_perf", 2024-03-19T03:42:13.746 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.746 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.746 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.746 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.746 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.746 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.747 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.747 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-19T03:42:13.747 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.747 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.747 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.747 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.747 INFO:teuthology.orchestra.run.smithi117.stdout: "contact": { 2024-03-19T03:42:13.747 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "contact", 2024-03-19T03:42:13.747 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.747 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.747 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.747 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.747 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.747 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.747 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.747 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.748 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.748 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.748 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.748 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.748 INFO:teuthology.orchestra.run.smithi117.stdout: "description": { 2024-03-19T03:42:13.748 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "description", 2024-03-19T03:42:13.748 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.748 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.748 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.748 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.748 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.748 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.748 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.748 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.748 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.748 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.748 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.749 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.749 INFO:teuthology.orchestra.run.smithi117.stdout: "device_url": { 2024-03-19T03:42:13.749 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "device_url", 2024-03-19T03:42:13.749 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.749 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.749 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.749 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-19T03:42:13.749 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.749 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.749 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.749 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.749 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.749 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.749 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.749 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.749 INFO:teuthology.orchestra.run.smithi117.stdout: "enabled": { 2024-03-19T03:42:13.749 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "enabled", 2024-03-19T03:42:13.750 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.750 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.750 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.750 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.750 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.750 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.750 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.750 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.750 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.750 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.750 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.750 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.750 INFO:teuthology.orchestra.run.smithi117.stdout: "interval": { 2024-03-19T03:42:13.750 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "interval", 2024-03-19T03:42:13.751 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.751 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.751 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.751 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "24", 2024-03-19T03:42:13.751 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "8", 2024-03-19T03:42:13.751 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.751 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.751 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.751 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.751 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.751 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.751 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.751 INFO:teuthology.orchestra.run.smithi117.stdout: "last_opt_revision": { 2024-03-19T03:42:13.751 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "last_opt_revision", 2024-03-19T03:42:13.751 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.751 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.752 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.752 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "1", 2024-03-19T03:42:13.752 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.752 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.752 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.752 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.752 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.752 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.752 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.752 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.752 INFO:teuthology.orchestra.run.smithi117.stdout: "leaderboard": { 2024-03-19T03:42:13.752 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "leaderboard", 2024-03-19T03:42:13.752 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.752 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.752 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.752 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.752 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.753 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.753 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.753 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.753 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.753 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.753 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.753 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.753 INFO:teuthology.orchestra.run.smithi117.stdout: "leaderboard_description": { 2024-03-19T03:42:13.753 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "leaderboard_description", 2024-03-19T03:42:13.753 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.753 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.753 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.753 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.753 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.753 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.753 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.753 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.754 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.754 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.754 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.754 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.754 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.754 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.754 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.754 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.754 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.754 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.754 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.754 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.754 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.755 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.755 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.755 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.755 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.755 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.755 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.755 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.755 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.755 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.755 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.755 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.755 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.755 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.755 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.756 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.756 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.756 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.756 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.756 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.756 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.756 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.756 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.756 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.756 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.756 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.756 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.757 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.757 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.757 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.757 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.757 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.757 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.757 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.757 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.757 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.757 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.757 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.757 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.757 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.758 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.758 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.758 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.758 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.758 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.758 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.758 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.758 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.758 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.758 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.758 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.758 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.758 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.758 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.759 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.759 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.759 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.759 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.759 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.759 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.759 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.759 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.759 INFO:teuthology.orchestra.run.smithi117.stdout: "organization": { 2024-03-19T03:42:13.759 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "organization", 2024-03-19T03:42:13.759 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.759 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.759 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.759 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.759 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.760 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.760 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.760 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.760 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.760 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.760 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.760 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.760 INFO:teuthology.orchestra.run.smithi117.stdout: "proxy": { 2024-03-19T03:42:13.760 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "proxy", 2024-03-19T03:42:13.760 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.760 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.760 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.760 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.760 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.761 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.761 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.761 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.761 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.761 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.761 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.761 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.761 INFO:teuthology.orchestra.run.smithi117.stdout: "url": { 2024-03-19T03:42:13.761 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "url", 2024-03-19T03:42:13.761 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.761 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.761 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.761 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-19T03:42:13.761 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.761 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.762 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.762 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.762 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.762 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.762 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.762 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.762 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.762 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.762 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.762 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "test_orchestrator", 2024-03-19T03:42:13.762 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.763 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.763 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.763 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.763 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.763 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.763 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.763 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.763 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.763 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.763 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.763 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.763 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.764 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.764 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.764 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.764 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.764 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.764 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.764 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.764 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.764 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.764 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.764 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.764 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.764 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.764 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.764 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.765 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.765 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.765 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.765 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.765 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.765 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.765 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.765 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.765 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.765 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.765 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.765 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.765 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.765 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.765 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.765 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.766 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.766 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.766 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.766 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.766 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.766 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.766 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.766 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.766 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.766 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.766 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.766 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.766 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.766 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.766 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.767 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.767 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.767 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.767 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.767 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.767 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.767 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.767 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.767 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.767 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.767 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.767 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.767 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.767 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.767 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.768 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.768 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.768 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "volumes", 2024-03-19T03:42:13.768 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.768 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.768 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.768 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.768 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.768 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.768 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.768 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.768 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.768 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.768 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.768 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.769 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.769 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.769 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.769 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.769 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.769 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.769 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.769 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.769 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.769 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.769 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.769 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.769 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.769 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.769 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.769 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.770 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.770 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.770 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.770 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.770 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.770 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.770 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.770 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.770 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.770 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.770 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.770 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.770 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.770 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.771 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.771 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.771 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.771 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.771 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.771 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.771 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.771 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.771 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.771 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.771 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.771 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.771 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.772 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.772 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.772 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.772 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.772 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.772 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.772 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.772 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.772 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.772 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.772 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.772 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.772 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.772 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.772 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.773 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.773 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.773 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.773 INFO:teuthology.orchestra.run.smithi117.stdout: "max_concurrent_clones": { 2024-03-19T03:42:13.773 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "max_concurrent_clones", 2024-03-19T03:42:13.773 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.773 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.773 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.773 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "4", 2024-03-19T03:42:13.773 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.773 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.773 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.773 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-19T03:42:13.773 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.773 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.774 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.774 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.774 INFO:teuthology.orchestra.run.smithi117.stdout: "periodic_async_work": { 2024-03-19T03:42:13.774 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "periodic_async_work", 2024-03-19T03:42:13.774 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.774 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.774 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.774 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.774 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.774 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.774 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.774 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Periodically check for async work", 2024-03-19T03:42:13.774 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.775 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.775 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.775 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.775 INFO:teuthology.orchestra.run.smithi117.stdout: "snapshot_clone_delay": { 2024-03-19T03:42:13.775 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "snapshot_clone_delay", 2024-03-19T03:42:13.775 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.775 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.775 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.775 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "0", 2024-03-19T03:42:13.775 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.775 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.775 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.775 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-19T03:42:13.775 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.775 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.776 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.776 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.776 INFO:teuthology.orchestra.run.smithi117.stdout: "snapshot_clone_no_wait": { 2024-03-19T03:42:13.776 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "snapshot_clone_no_wait", 2024-03-19T03:42:13.776 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.776 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.776 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.776 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "True", 2024-03-19T03:42:13.776 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.776 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.776 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.776 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-19T03:42:13.776 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.776 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.776 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.776 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.777 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.777 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.777 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.777 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "zabbix", 2024-03-19T03:42:13.777 INFO:teuthology.orchestra.run.smithi117.stdout: "can_run": true, 2024-03-19T03:42:13.777 INFO:teuthology.orchestra.run.smithi117.stdout: "error_string": "", 2024-03-19T03:42:13.777 INFO:teuthology.orchestra.run.smithi117.stdout: "module_options": { 2024-03-19T03:42:13.777 INFO:teuthology.orchestra.run.smithi117.stdout: "discovery_interval": { 2024-03-19T03:42:13.777 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "discovery_interval", 2024-03-19T03:42:13.777 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "uint", 2024-03-19T03:42:13.777 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.777 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.777 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "100", 2024-03-19T03:42:13.777 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.777 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.777 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.778 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.778 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.778 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.778 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.778 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.778 INFO:teuthology.orchestra.run.smithi117.stdout: "identifier": { 2024-03-19T03:42:13.778 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "identifier", 2024-03-19T03:42:13.778 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.778 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.778 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.778 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.778 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.778 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.778 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.778 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.778 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.779 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.779 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.779 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.779 INFO:teuthology.orchestra.run.smithi117.stdout: "interval": { 2024-03-19T03:42:13.779 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "interval", 2024-03-19T03:42:13.779 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "secs", 2024-03-19T03:42:13.779 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.779 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.779 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "60", 2024-03-19T03:42:13.779 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.779 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.779 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.779 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.780 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.780 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.780 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.780 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.780 INFO:teuthology.orchestra.run.smithi117.stdout: "log_level": { 2024-03-19T03:42:13.780 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_level", 2024-03-19T03:42:13.780 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.780 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.780 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.780 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.780 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.780 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.780 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.780 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.781 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.781 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.781 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.781 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.781 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.781 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.781 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.781 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.781 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.781 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.781 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.781 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster": { 2024-03-19T03:42:13.781 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster", 2024-03-19T03:42:13.782 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.782 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.782 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.782 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.782 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.782 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.782 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.782 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.782 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.782 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.782 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.782 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.782 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_cluster_level": { 2024-03-19T03:42:13.782 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_cluster_level", 2024-03-19T03:42:13.782 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.783 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.783 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.783 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "info", 2024-03-19T03:42:13.783 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.783 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.783 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [ 2024-03-19T03:42:13.783 INFO:teuthology.orchestra.run.smithi117.stdout: "", 2024-03-19T03:42:13.783 INFO:teuthology.orchestra.run.smithi117.stdout: "critical", 2024-03-19T03:42:13.783 INFO:teuthology.orchestra.run.smithi117.stdout: "debug", 2024-03-19T03:42:13.783 INFO:teuthology.orchestra.run.smithi117.stdout: "error", 2024-03-19T03:42:13.783 INFO:teuthology.orchestra.run.smithi117.stdout: "info", 2024-03-19T03:42:13.783 INFO:teuthology.orchestra.run.smithi117.stdout: "warning" 2024-03-19T03:42:13.783 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.783 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.783 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.783 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.784 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.784 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.784 INFO:teuthology.orchestra.run.smithi117.stdout: "log_to_file": { 2024-03-19T03:42:13.784 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "log_to_file", 2024-03-19T03:42:13.784 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "bool", 2024-03-19T03:42:13.784 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.784 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 1, 2024-03-19T03:42:13.784 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "False", 2024-03-19T03:42:13.784 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.784 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.784 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.784 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.784 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.785 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.785 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.785 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.785 INFO:teuthology.orchestra.run.smithi117.stdout: "zabbix_host": { 2024-03-19T03:42:13.785 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "zabbix_host", 2024-03-19T03:42:13.785 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.785 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.785 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.785 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "", 2024-03-19T03:42:13.785 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.785 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.785 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.785 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.785 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.785 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.785 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.786 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.786 INFO:teuthology.orchestra.run.smithi117.stdout: "zabbix_port": { 2024-03-19T03:42:13.786 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "zabbix_port", 2024-03-19T03:42:13.786 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "int", 2024-03-19T03:42:13.786 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.786 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.786 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "10051", 2024-03-19T03:42:13.786 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.786 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.786 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.786 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.786 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.786 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.786 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.786 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.787 INFO:teuthology.orchestra.run.smithi117.stdout: "zabbix_sender": { 2024-03-19T03:42:13.787 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "zabbix_sender", 2024-03-19T03:42:13.787 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "str", 2024-03-19T03:42:13.787 INFO:teuthology.orchestra.run.smithi117.stdout: "level": "advanced", 2024-03-19T03:42:13.787 INFO:teuthology.orchestra.run.smithi117.stdout: "flags": 0, 2024-03-19T03:42:13.787 INFO:teuthology.orchestra.run.smithi117.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-19T03:42:13.787 INFO:teuthology.orchestra.run.smithi117.stdout: "min": "", 2024-03-19T03:42:13.787 INFO:teuthology.orchestra.run.smithi117.stdout: "max": "", 2024-03-19T03:42:13.787 INFO:teuthology.orchestra.run.smithi117.stdout: "enum_allowed": [], 2024-03-19T03:42:13.787 INFO:teuthology.orchestra.run.smithi117.stdout: "desc": "", 2024-03-19T03:42:13.787 INFO:teuthology.orchestra.run.smithi117.stdout: "long_desc": "", 2024-03-19T03:42:13.787 INFO:teuthology.orchestra.run.smithi117.stdout: "tags": [], 2024-03-19T03:42:13.787 INFO:teuthology.orchestra.run.smithi117.stdout: "see_also": [] 2024-03-19T03:42:13.787 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.787 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.787 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.788 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.788 INFO:teuthology.orchestra.run.smithi117.stdout: "services": { 2024-03-19T03:42:13.788 INFO:teuthology.orchestra.run.smithi117.stdout: "dashboard": "https://172.21.15.117:8443/" 2024-03-19T03:42:13.788 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.788 INFO:teuthology.orchestra.run.smithi117.stdout: "always_on_modules": { 2024-03-19T03:42:13.788 INFO:teuthology.orchestra.run.smithi117.stdout: "octopus": [ 2024-03-19T03:42:13.788 INFO:teuthology.orchestra.run.smithi117.stdout: "balancer", 2024-03-19T03:42:13.788 INFO:teuthology.orchestra.run.smithi117.stdout: "crash", 2024-03-19T03:42:13.788 INFO:teuthology.orchestra.run.smithi117.stdout: "devicehealth", 2024-03-19T03:42:13.788 INFO:teuthology.orchestra.run.smithi117.stdout: "orchestrator", 2024-03-19T03:42:13.788 INFO:teuthology.orchestra.run.smithi117.stdout: "pg_autoscaler", 2024-03-19T03:42:13.788 INFO:teuthology.orchestra.run.smithi117.stdout: "progress", 2024-03-19T03:42:13.788 INFO:teuthology.orchestra.run.smithi117.stdout: "rbd_support", 2024-03-19T03:42:13.788 INFO:teuthology.orchestra.run.smithi117.stdout: "status", 2024-03-19T03:42:13.788 INFO:teuthology.orchestra.run.smithi117.stdout: "telemetry", 2024-03-19T03:42:13.788 INFO:teuthology.orchestra.run.smithi117.stdout: "volumes" 2024-03-19T03:42:13.789 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.789 INFO:teuthology.orchestra.run.smithi117.stdout: "pacific": [ 2024-03-19T03:42:13.789 INFO:teuthology.orchestra.run.smithi117.stdout: "balancer", 2024-03-19T03:42:13.789 INFO:teuthology.orchestra.run.smithi117.stdout: "crash", 2024-03-19T03:42:13.789 INFO:teuthology.orchestra.run.smithi117.stdout: "devicehealth", 2024-03-19T03:42:13.789 INFO:teuthology.orchestra.run.smithi117.stdout: "orchestrator", 2024-03-19T03:42:13.789 INFO:teuthology.orchestra.run.smithi117.stdout: "pg_autoscaler", 2024-03-19T03:42:13.789 INFO:teuthology.orchestra.run.smithi117.stdout: "progress", 2024-03-19T03:42:13.789 INFO:teuthology.orchestra.run.smithi117.stdout: "rbd_support", 2024-03-19T03:42:13.789 INFO:teuthology.orchestra.run.smithi117.stdout: "status", 2024-03-19T03:42:13.789 INFO:teuthology.orchestra.run.smithi117.stdout: "telemetry", 2024-03-19T03:42:13.789 INFO:teuthology.orchestra.run.smithi117.stdout: "volumes" 2024-03-19T03:42:13.789 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.789 INFO:teuthology.orchestra.run.smithi117.stdout: "quincy": [ 2024-03-19T03:42:13.789 INFO:teuthology.orchestra.run.smithi117.stdout: "balancer", 2024-03-19T03:42:13.789 INFO:teuthology.orchestra.run.smithi117.stdout: "crash", 2024-03-19T03:42:13.790 INFO:teuthology.orchestra.run.smithi117.stdout: "devicehealth", 2024-03-19T03:42:13.790 INFO:teuthology.orchestra.run.smithi117.stdout: "orchestrator", 2024-03-19T03:42:13.790 INFO:teuthology.orchestra.run.smithi117.stdout: "pg_autoscaler", 2024-03-19T03:42:13.790 INFO:teuthology.orchestra.run.smithi117.stdout: "progress", 2024-03-19T03:42:13.790 INFO:teuthology.orchestra.run.smithi117.stdout: "rbd_support", 2024-03-19T03:42:13.790 INFO:teuthology.orchestra.run.smithi117.stdout: "status", 2024-03-19T03:42:13.790 INFO:teuthology.orchestra.run.smithi117.stdout: "telemetry", 2024-03-19T03:42:13.790 INFO:teuthology.orchestra.run.smithi117.stdout: "volumes" 2024-03-19T03:42:13.790 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.790 INFO:teuthology.orchestra.run.smithi117.stdout: "reef": [ 2024-03-19T03:42:13.790 INFO:teuthology.orchestra.run.smithi117.stdout: "balancer", 2024-03-19T03:42:13.790 INFO:teuthology.orchestra.run.smithi117.stdout: "crash", 2024-03-19T03:42:13.790 INFO:teuthology.orchestra.run.smithi117.stdout: "devicehealth", 2024-03-19T03:42:13.790 INFO:teuthology.orchestra.run.smithi117.stdout: "orchestrator", 2024-03-19T03:42:13.791 INFO:teuthology.orchestra.run.smithi117.stdout: "pg_autoscaler", 2024-03-19T03:42:13.791 INFO:teuthology.orchestra.run.smithi117.stdout: "progress", 2024-03-19T03:42:13.791 INFO:teuthology.orchestra.run.smithi117.stdout: "rbd_support", 2024-03-19T03:42:13.791 INFO:teuthology.orchestra.run.smithi117.stdout: "status", 2024-03-19T03:42:13.791 INFO:teuthology.orchestra.run.smithi117.stdout: "telemetry", 2024-03-19T03:42:13.791 INFO:teuthology.orchestra.run.smithi117.stdout: "volumes" 2024-03-19T03:42:13.791 INFO:teuthology.orchestra.run.smithi117.stdout: ], 2024-03-19T03:42:13.791 INFO:teuthology.orchestra.run.smithi117.stdout: "squid": [ 2024-03-19T03:42:13.791 INFO:teuthology.orchestra.run.smithi117.stdout: "balancer", 2024-03-19T03:42:13.791 INFO:teuthology.orchestra.run.smithi117.stdout: "crash", 2024-03-19T03:42:13.791 INFO:teuthology.orchestra.run.smithi117.stdout: "devicehealth", 2024-03-19T03:42:13.791 INFO:teuthology.orchestra.run.smithi117.stdout: "orchestrator", 2024-03-19T03:42:13.791 INFO:teuthology.orchestra.run.smithi117.stdout: "pg_autoscaler", 2024-03-19T03:42:13.791 INFO:teuthology.orchestra.run.smithi117.stdout: "progress", 2024-03-19T03:42:13.791 INFO:teuthology.orchestra.run.smithi117.stdout: "rbd_support", 2024-03-19T03:42:13.791 INFO:teuthology.orchestra.run.smithi117.stdout: "status", 2024-03-19T03:42:13.791 INFO:teuthology.orchestra.run.smithi117.stdout: "telemetry", 2024-03-19T03:42:13.792 INFO:teuthology.orchestra.run.smithi117.stdout: "volumes" 2024-03-19T03:42:13.792 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-03-19T03:42:13.792 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.792 INFO:teuthology.orchestra.run.smithi117.stdout: "last_failure_osd_epoch": 3, 2024-03-19T03:42:13.792 INFO:teuthology.orchestra.run.smithi117.stdout: "active_clients": [ 2024-03-19T03:42:13.792 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.792 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "devicehealth", 2024-03-19T03:42:13.792 INFO:teuthology.orchestra.run.smithi117.stdout: "addrvec": [ 2024-03-19T03:42:13.792 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.792 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "v2", 2024-03-19T03:42:13.792 INFO:teuthology.orchestra.run.smithi117.stdout: "addr": "172.21.15.117:0", 2024-03-19T03:42:13.792 INFO:teuthology.orchestra.run.smithi117.stdout: "nonce": 622505007 2024-03-19T03:42:13.792 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.792 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-03-19T03:42:13.792 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.793 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.793 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "libcephsqlite", 2024-03-19T03:42:13.793 INFO:teuthology.orchestra.run.smithi117.stdout: "addrvec": [ 2024-03-19T03:42:13.793 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.793 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "v2", 2024-03-19T03:42:13.793 INFO:teuthology.orchestra.run.smithi117.stdout: "addr": "172.21.15.117:0", 2024-03-19T03:42:13.793 INFO:teuthology.orchestra.run.smithi117.stdout: "nonce": 2721065064 2024-03-19T03:42:13.793 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.793 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-03-19T03:42:13.793 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.793 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.793 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "rbd_support", 2024-03-19T03:42:13.793 INFO:teuthology.orchestra.run.smithi117.stdout: "addrvec": [ 2024-03-19T03:42:13.793 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.793 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "v2", 2024-03-19T03:42:13.793 INFO:teuthology.orchestra.run.smithi117.stdout: "addr": "172.21.15.117:0", 2024-03-19T03:42:13.794 INFO:teuthology.orchestra.run.smithi117.stdout: "nonce": 1074989880 2024-03-19T03:42:13.794 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.794 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-03-19T03:42:13.794 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:13.794 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.794 INFO:teuthology.orchestra.run.smithi117.stdout: "name": "volumes", 2024-03-19T03:42:13.794 INFO:teuthology.orchestra.run.smithi117.stdout: "addrvec": [ 2024-03-19T03:42:13.794 INFO:teuthology.orchestra.run.smithi117.stdout: { 2024-03-19T03:42:13.794 INFO:teuthology.orchestra.run.smithi117.stdout: "type": "v2", 2024-03-19T03:42:13.794 INFO:teuthology.orchestra.run.smithi117.stdout: "addr": "172.21.15.117:0", 2024-03-19T03:42:13.794 INFO:teuthology.orchestra.run.smithi117.stdout: "nonce": 1939214638 2024-03-19T03:42:13.794 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.794 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-03-19T03:42:13.794 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:13.794 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-03-19T03:42:13.795 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-19T03:42:13.795 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:13 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2125650409' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-19T03:42:13.796 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:13 smithi119 ceph-mon[66861]: pgmap v670: 129 pgs: 129 active+clean; 247 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 878 KiB/s wr, 418 op/s 2024-03-19T03:42:13.796 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:13 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2125650409' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-19T03:42:13.796 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:13 smithi178 ceph-mon[67578]: pgmap v670: 129 pgs: 129 active+clean; 247 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 878 KiB/s wr, 418 op/s 2024-03-19T03:42:13.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.510+0000 7f0a29000640 1 -- 172.21.15.117:0/2125650409 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0a080a2880 msgr2=0x7f0a080a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:13.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.510+0000 7f0a29000640 1 --2- 172.21.15.117:0/2125650409 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0a080a2880 0x7f0a080a4d40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0a2419b200 tx=0x7f0a14005f90 comp rx=0 tx=0).stop 2024-03-19T03:42:13.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.510+0000 7f0a29000640 1 -- 172.21.15.117:0/2125650409 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a24105610 msgr2=0x7f0a241a12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:13.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.510+0000 7f0a29000640 1 --2- 172.21.15.117:0/2125650409 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a24105610 0x7f0a241a12a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c037040 tx=0x7f0a1c035d10 comp rx=0 tx=0).stop 2024-03-19T03:42:13.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.510+0000 7f0a29000640 1 -- 172.21.15.117:0/2125650409 shutdown_connections 2024-03-19T03:42:13.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.510+0000 7f0a29000640 1 --2- 172.21.15.117:0/2125650409 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f0a080a2880 0x7f0a080a4d40 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:13.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.510+0000 7f0a29000640 1 --2- 172.21.15.117:0/2125650409 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a24105610 0x7f0a241a12a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:13.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.510+0000 7f0a29000640 1 --2- 172.21.15.117:0/2125650409 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a24104c50 0x7f0a2419a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:13.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.510+0000 7f0a29000640 1 --2- 172.21.15.117:0/2125650409 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a24103a60 0x7f0a24199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:13.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.510+0000 7f0a29000640 1 -- 172.21.15.117:0/2125650409 >> 172.21.15.117:0/2125650409 conn(0x7f0a240ff010 msgr2=0x7f0a24107f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:13.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.510+0000 7f0a29000640 1 -- 172.21.15.117:0/2125650409 shutdown_connections 2024-03-19T03:42:13.798 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:12.510+0000 7f0a29000640 1 -- 172.21.15.117:0/2125650409 wait complete. 2024-03-19T03:42:13.804 DEBUG:tasks.check_counter:Getting stats from a 2024-03-19T03:42:13.804 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-19T03:42:13.804 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:13 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2125650409' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-19T03:42:13.804 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:13 smithi117 ceph-mon[31838]: pgmap v670: 129 pgs: 129 active+clean; 247 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 878 KiB/s wr, 418 op/s 2024-03-19T03:42:14.257 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:42:14.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:14 smithi119 ceph-mon[66861]: pgmap v671: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 546 KiB/s wr, 348 op/s 2024-03-19T03:42:15.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:14 smithi117 ceph-mon[31838]: pgmap v671: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 546 KiB/s wr, 348 op/s 2024-03-19T03:42:15.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:14 smithi178 ceph-mon[67578]: pgmap v671: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 546 KiB/s wr, 348 op/s 2024-03-19T03:42:16.570 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-19T03:42:16.571 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T03:42:16.571 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 15091, 2024-03-19T03:42:16.571 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 11262, 2024-03-19T03:42:16.571 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 7164889, 2024-03-19T03:42:16.571 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 7590682, 2024-03-19T03:42:16.571 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 9, 2024-03-19T03:42:16.571 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 5, 2024-03-19T03:42:16.571 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 2.319740469, 2024-03-19T03:42:16.571 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.747012078, 2024-03-19T03:42:16.571 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 1.620940899, 2024-03-19T03:42:16.571 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.083026218, 2024-03-19T03:42:16.571 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:16.571 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 11261, 2024-03-19T03:42:16.571 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 3.440059434, 2024-03-19T03:42:16.571 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000305484 2024-03-19T03:42:16.572 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.572 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:16.572 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 24023, 2024-03-19T03:42:16.572 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.013551596, 2024-03-19T03:42:16.572 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000564 2024-03-19T03:42:16.572 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.572 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 912672, 2024-03-19T03:42:16.572 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 124480 2024-03-19T03:42:16.572 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.572 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T03:42:16.572 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 1029, 2024-03-19T03:42:16.572 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 1048, 2024-03-19T03:42:16.572 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 430890, 2024-03-19T03:42:16.572 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 862234, 2024-03-19T03:42:16.573 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 16, 2024-03-19T03:42:16.573 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 10, 2024-03-19T03:42:16.573 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.269790063, 2024-03-19T03:42:16.573 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.140635042, 2024-03-19T03:42:16.573 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.094850535, 2024-03-19T03:42:16.573 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.029523677, 2024-03-19T03:42:16.573 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:16.573 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1046, 2024-03-19T03:42:16.573 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.523605074, 2024-03-19T03:42:16.573 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000500578 2024-03-19T03:42:16.573 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.573 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:16.574 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1123, 2024-03-19T03:42:16.574 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.001572674, 2024-03-19T03:42:16.574 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000001400 2024-03-19T03:42:16.574 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.574 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 4608, 2024-03-19T03:42:16.574 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 334368 2024-03-19T03:42:16.574 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.574 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T03:42:16.574 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 10801, 2024-03-19T03:42:16.574 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 13421, 2024-03-19T03:42:16.574 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 3693588, 2024-03-19T03:42:16.575 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 6119239, 2024-03-19T03:42:16.575 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 8, 2024-03-19T03:42:16.575 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 6, 2024-03-19T03:42:16.575 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 3.863285377, 2024-03-19T03:42:16.575 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 2.983505310, 2024-03-19T03:42:16.575 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 1.131704402, 2024-03-19T03:42:16.575 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.061961199, 2024-03-19T03:42:16.575 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:16.575 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 13420, 2024-03-19T03:42:16.575 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 7.261831538, 2024-03-19T03:42:16.575 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000541120 2024-03-19T03:42:16.575 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.575 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:16.576 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 9073, 2024-03-19T03:42:16.576 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.015465792, 2024-03-19T03:42:16.576 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000001704 2024-03-19T03:42:16.576 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.576 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-19T03:42:16.576 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 448 2024-03-19T03:42:16.576 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.576 INFO:teuthology.orchestra.run.smithi117.stdout: "cct": { 2024-03-19T03:42:16.576 INFO:teuthology.orchestra.run.smithi117.stdout: "total_workers": 1, 2024-03-19T03:42:16.576 INFO:teuthology.orchestra.run.smithi117.stdout: "unhealthy_workers": 0 2024-03-19T03:42:16.576 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.576 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-MDSRank": { 2024-03-19T03:42:16.577 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-19T03:42:16.577 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-19T03:42:16.577 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 534, 2024-03-19T03:42:16.577 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 4.134770949, 2024-03-19T03:42:16.577 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.007743016 2024-03-19T03:42:16.577 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:16.577 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.577 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-PurgeQueue": { 2024-03-19T03:42:16.577 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-19T03:42:16.577 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-19T03:42:16.577 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1390, 2024-03-19T03:42:16.577 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.304326041, 2024-03-19T03:42:16.578 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000218939 2024-03-19T03:42:16.578 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:16.578 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.578 INFO:teuthology.orchestra.run.smithi117.stdout: "mds": { 2024-03-19T03:42:16.578 INFO:teuthology.orchestra.run.smithi117.stdout: "request": 4547, 2024-03-19T03:42:16.578 INFO:teuthology.orchestra.run.smithi117.stdout: "reply": 4041, 2024-03-19T03:42:16.578 INFO:teuthology.orchestra.run.smithi117.stdout: "reply_latency": { 2024-03-19T03:42:16.578 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 4041, 2024-03-19T03:42:16.578 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 44.320264735, 2024-03-19T03:42:16.578 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.010967647 2024-03-19T03:42:16.578 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.578 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_reply": 0, 2024-03-19T03:42:16.578 INFO:teuthology.orchestra.run.smithi117.stdout: "forward": 506, 2024-03-19T03:42:16.578 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_complete": 0, 2024-03-19T03:42:16.579 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_keys": 0, 2024-03-19T03:42:16.579 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_commit": 29, 2024-03-19T03:42:16.579 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_split": 2, 2024-03-19T03:42:16.579 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_merge": 0, 2024-03-19T03:42:16.579 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes": 24, 2024-03-19T03:42:16.579 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_top": 6, 2024-03-19T03:42:16.579 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_bottom": 8, 2024-03-19T03:42:16.579 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pin_tail": 10, 2024-03-19T03:42:16.579 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pinned": 13, 2024-03-19T03:42:16.579 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_expired": 354, 2024-03-19T03:42:16.579 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_with_caps": 0, 2024-03-19T03:42:16.579 INFO:teuthology.orchestra.run.smithi117.stdout: "caps": 0, 2024-03-19T03:42:16.579 INFO:teuthology.orchestra.run.smithi117.stdout: "subtrees": 3, 2024-03-19T03:42:16.579 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse": 12336, 2024-03-19T03:42:16.580 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_hit": 9886, 2024-03-19T03:42:16.580 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_forward": 502, 2024-03-19T03:42:16.580 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_discover": 193, 2024-03-19T03:42:16.580 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_dir_fetch": 0, 2024-03-19T03:42:16.580 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_remote_ino": 0, 2024-03-19T03:42:16.580 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_lock": 96, 2024-03-19T03:42:16.580 INFO:teuthology.orchestra.run.smithi117.stdout: "load_cent": 0, 2024-03-19T03:42:16.580 INFO:teuthology.orchestra.run.smithi117.stdout: "q": 0, 2024-03-19T03:42:16.580 INFO:teuthology.orchestra.run.smithi117.stdout: "exported": 16, 2024-03-19T03:42:16.580 INFO:teuthology.orchestra.run.smithi117.stdout: "exported_inodes": 2, 2024-03-19T03:42:16.580 INFO:teuthology.orchestra.run.smithi117.stdout: "imported": 2, 2024-03-19T03:42:16.580 INFO:teuthology.orchestra.run.smithi117.stdout: "imported_inodes": 483, 2024-03-19T03:42:16.580 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_dir_fetch": 0, 2024-03-19T03:42:16.580 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_backtrace_fetch": 0, 2024-03-19T03:42:16.581 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_peer_discover": 0, 2024-03-19T03:42:16.581 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rfiles": 3308, 2024-03-19T03:42:16.581 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rbytes": 68731964, 2024-03-19T03:42:16.581 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rsnaps": 0, 2024-03-19T03:42:16.581 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T03:42:16.581 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_set_tag": 0, 2024-03-19T03:42:16.581 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T03:42:16.581 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_inotable_repaired": 0, 2024-03-19T03:42:16.581 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_inodes": 0, 2024-03-19T03:42:16.581 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T03:42:16.581 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T03:42:16.581 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_file_inodes": 0, 2024-03-19T03:42:16.581 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_inode_file_caps": 15, 2024-03-19T03:42:16.581 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_revoke": 642, 2024-03-19T03:42:16.582 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_grant": 1254, 2024-03-19T03:42:16.582 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T03:42:16.582 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T03:42:16.582 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T03:42:16.582 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps": 1376, 2024-03-19T03:42:16.582 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps_dirty": 681, 2024-03-19T03:42:16.582 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_cap_release": 86, 2024-03-19T03:42:16.582 INFO:teuthology.orchestra.run.smithi117.stdout: "process_request_cap_release": 2417 2024-03-19T03:42:16.582 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.582 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_cache": { 2024-03-19T03:42:16.582 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update": 288, 2024-03-19T03:42:16.582 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update_receipt": 4472, 2024-03-19T03:42:16.582 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_try_discover": 3014, 2024-03-19T03:42:16.583 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_send_discover": 3229, 2024-03-19T03:42:16.583 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_handle_discover": 79, 2024-03-19T03:42:16.583 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays": 0, 2024-03-19T03:42:16.583 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_delayed": 0, 2024-03-19T03:42:16.583 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_enqueuing": 0, 2024-03-19T03:42:16.583 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_created": 679, 2024-03-19T03:42:16.583 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_enqueued": 679, 2024-03-19T03:42:16.583 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_reintegrated": 0, 2024-03-19T03:42:16.583 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_migrated": 0, 2024-03-19T03:42:16.583 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_processing": 0, 2024-03-19T03:42:16.583 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_enqueued": 0, 2024-03-19T03:42:16.583 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_prioritized": 0, 2024-03-19T03:42:16.583 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_started": 0, 2024-03-19T03:42:16.583 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_completed": 0, 2024-03-19T03:42:16.583 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_quiesce_path": 0, 2024-03-19T03:42:16.583 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_quiesce_inode": 0, 2024-03-19T03:42:16.584 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T03:42:16.584 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_exportdir": 16, 2024-03-19T03:42:16.584 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_flush": 0, 2024-03-19T03:42:16.584 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragmentdir": 2, 2024-03-19T03:42:16.584 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragstats": 0, 2024-03-19T03:42:16.584 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_inodestats": 0 2024-03-19T03:42:16.584 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.584 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_log": { 2024-03-19T03:42:16.584 INFO:teuthology.orchestra.run.smithi117.stdout: "evlrg": 0, 2024-03-19T03:42:16.584 INFO:teuthology.orchestra.run.smithi117.stdout: "evadd": 3520, 2024-03-19T03:42:16.584 INFO:teuthology.orchestra.run.smithi117.stdout: "evex": 0, 2024-03-19T03:42:16.584 INFO:teuthology.orchestra.run.smithi117.stdout: "evtrm": 0, 2024-03-19T03:42:16.585 INFO:teuthology.orchestra.run.smithi117.stdout: "ev": 3520, 2024-03-19T03:42:16.585 INFO:teuthology.orchestra.run.smithi117.stdout: "evexg": 0, 2024-03-19T03:42:16.585 INFO:teuthology.orchestra.run.smithi117.stdout: "evexd": 0, 2024-03-19T03:42:16.585 INFO:teuthology.orchestra.run.smithi117.stdout: "segadd": 5, 2024-03-19T03:42:16.585 INFO:teuthology.orchestra.run.smithi117.stdout: "segex": 0, 2024-03-19T03:42:16.585 INFO:teuthology.orchestra.run.smithi117.stdout: "segtrm": 0, 2024-03-19T03:42:16.585 INFO:teuthology.orchestra.run.smithi117.stdout: "seg": 5, 2024-03-19T03:42:16.585 INFO:teuthology.orchestra.run.smithi117.stdout: "segmjr": 2, 2024-03-19T03:42:16.585 INFO:teuthology.orchestra.run.smithi117.stdout: "segexg": 0, 2024-03-19T03:42:16.585 INFO:teuthology.orchestra.run.smithi117.stdout: "segexd": 0, 2024-03-19T03:42:16.585 INFO:teuthology.orchestra.run.smithi117.stdout: "expos": 4194304, 2024-03-19T03:42:16.585 INFO:teuthology.orchestra.run.smithi117.stdout: "wrpos": 9524560, 2024-03-19T03:42:16.585 INFO:teuthology.orchestra.run.smithi117.stdout: "rdpos": 0, 2024-03-19T03:42:16.585 INFO:teuthology.orchestra.run.smithi117.stdout: "jlat": { 2024-03-19T03:42:16.585 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 178, 2024-03-19T03:42:16.586 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 2.432578991, 2024-03-19T03:42:16.586 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.013666174 2024-03-19T03:42:16.586 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.586 INFO:teuthology.orchestra.run.smithi117.stdout: "replayed": 0 2024-03-19T03:42:16.586 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.586 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_mem": { 2024-03-19T03:42:16.586 INFO:teuthology.orchestra.run.smithi117.stdout: "ino": 28, 2024-03-19T03:42:16.586 INFO:teuthology.orchestra.run.smithi117.stdout: "ino+": 884, 2024-03-19T03:42:16.586 INFO:teuthology.orchestra.run.smithi117.stdout: "ino-": 856, 2024-03-19T03:42:16.586 INFO:teuthology.orchestra.run.smithi117.stdout: "dir": 27, 2024-03-19T03:42:16.586 INFO:teuthology.orchestra.run.smithi117.stdout: "dir+": 1371, 2024-03-19T03:42:16.586 INFO:teuthology.orchestra.run.smithi117.stdout: "dir-": 1344, 2024-03-19T03:42:16.586 INFO:teuthology.orchestra.run.smithi117.stdout: "dn": 24, 2024-03-19T03:42:16.586 INFO:teuthology.orchestra.run.smithi117.stdout: "dn+": 1833, 2024-03-19T03:42:16.586 INFO:teuthology.orchestra.run.smithi117.stdout: "dn-": 1809, 2024-03-19T03:42:16.587 INFO:teuthology.orchestra.run.smithi117.stdout: "cap": 0, 2024-03-19T03:42:16.587 INFO:teuthology.orchestra.run.smithi117.stdout: "cap+": 1073, 2024-03-19T03:42:16.587 INFO:teuthology.orchestra.run.smithi117.stdout: "cap-": 1073, 2024-03-19T03:42:16.587 INFO:teuthology.orchestra.run.smithi117.stdout: "rss": 62904, 2024-03-19T03:42:16.587 INFO:teuthology.orchestra.run.smithi117.stdout: "heap": 223612 2024-03-19T03:42:16.587 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.587 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_server": { 2024-03-19T03:42:16.587 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_client_request": 6085, 2024-03-19T03:42:16.587 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_server_request": 85, 2024-03-19T03:42:16.587 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_request": 4547, 2024-03-19T03:42:16.587 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_session": 15, 2024-03-19T03:42:16.587 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_peer_request": 1473, 2024-03-19T03:42:16.588 INFO:teuthology.orchestra.run.smithi117.stdout: "req_create_latency": { 2024-03-19T03:42:16.588 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 447, 2024-03-19T03:42:16.588 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 5.944772597, 2024-03-19T03:42:16.588 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.013299267 2024-03-19T03:42:16.588 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.588 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getattr_latency": { 2024-03-19T03:42:16.588 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2, 2024-03-19T03:42:16.589 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.002004116, 2024-03-19T03:42:16.589 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.001002058 2024-03-19T03:42:16.589 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.589 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getfilelock_latency": { 2024-03-19T03:42:16.589 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.589 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.589 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.589 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.589 INFO:teuthology.orchestra.run.smithi117.stdout: "req_link_latency": { 2024-03-19T03:42:16.590 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.590 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.590 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.590 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.590 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookup_latency": { 2024-03-19T03:42:16.590 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2388, 2024-03-19T03:42:16.590 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 30.051507263, 2024-03-19T03:42:16.590 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.012584383 2024-03-19T03:42:16.590 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.590 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookuphash_latency": { 2024-03-19T03:42:16.590 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.590 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.591 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.591 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.591 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupino_latency": { 2024-03-19T03:42:16.591 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.591 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.591 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.591 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.591 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupname_latency": { 2024-03-19T03:42:16.591 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.591 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.591 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.591 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.591 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupparent_latency": { 2024-03-19T03:42:16.591 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.592 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.592 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.592 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.592 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupsnap_latency": { 2024-03-19T03:42:16.592 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.592 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.592 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.592 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.592 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lssnap_latency": { 2024-03-19T03:42:16.592 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.593 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.593 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.593 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.593 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mkdir_latency": { 2024-03-19T03:42:16.593 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.593 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.593 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.593 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.593 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mknod_latency": { 2024-03-19T03:42:16.593 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.594 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.594 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.594 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.594 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mksnap_latency": { 2024-03-19T03:42:16.594 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.594 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.594 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.594 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.594 INFO:teuthology.orchestra.run.smithi117.stdout: "req_open_latency": { 2024-03-19T03:42:16.594 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 361, 2024-03-19T03:42:16.594 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.544373083, 2024-03-19T03:42:16.594 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.001507958 2024-03-19T03:42:16.594 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.595 INFO:teuthology.orchestra.run.smithi117.stdout: "req_readdir_latency": { 2024-03-19T03:42:16.595 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 16, 2024-03-19T03:42:16.595 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.052604071, 2024-03-19T03:42:16.595 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.003287754 2024-03-19T03:42:16.595 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.595 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rename_latency": { 2024-03-19T03:42:16.595 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 447, 2024-03-19T03:42:16.595 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 6.789922221, 2024-03-19T03:42:16.595 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.015189982 2024-03-19T03:42:16.595 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.595 INFO:teuthology.orchestra.run.smithi117.stdout: "req_renamesnap_latency": { 2024-03-19T03:42:16.595 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.595 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.595 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.595 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.595 INFO:teuthology.orchestra.run.smithi117.stdout: "req_snapdiff_latency": { 2024-03-19T03:42:16.596 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.596 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.596 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.596 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.596 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmdir_latency": { 2024-03-19T03:42:16.596 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.596 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.596 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.596 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.596 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmsnap_latency": { 2024-03-19T03:42:16.596 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.596 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.596 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.596 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.596 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmxattr_latency": { 2024-03-19T03:42:16.597 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.597 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.597 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.597 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.597 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setattr_latency": { 2024-03-19T03:42:16.597 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.597 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.597 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.597 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.597 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setdirlayout_latency": { 2024-03-19T03:42:16.598 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.598 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.598 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.598 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.598 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setfilelock_latency": { 2024-03-19T03:42:16.598 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.598 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.598 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.598 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.598 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setlayout_latency": { 2024-03-19T03:42:16.598 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.598 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.598 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.599 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.599 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setxattr_latency": { 2024-03-19T03:42:16.599 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.599 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.599 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.599 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.599 INFO:teuthology.orchestra.run.smithi117.stdout: "req_symlink_latency": { 2024-03-19T03:42:16.599 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.599 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.599 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.599 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.599 INFO:teuthology.orchestra.run.smithi117.stdout: "req_unlink_latency": { 2024-03-19T03:42:16.599 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 380, 2024-03-19T03:42:16.599 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.935081384, 2024-03-19T03:42:16.600 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.002460740 2024-03-19T03:42:16.600 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.600 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_revoke_eviction": 0, 2024-03-19T03:42:16.600 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_acquisition_throttle": 0, 2024-03-19T03:42:16.600 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getvxattr_latency": { 2024-03-19T03:42:16.600 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.600 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.600 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.600 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:16.600 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.600 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_sessions": { 2024-03-19T03:42:16.600 INFO:teuthology.orchestra.run.smithi117.stdout: "session_count": 1, 2024-03-19T03:42:16.600 INFO:teuthology.orchestra.run.smithi117.stdout: "session_add": 3, 2024-03-19T03:42:16.601 INFO:teuthology.orchestra.run.smithi117.stdout: "session_remove": 2, 2024-03-19T03:42:16.601 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_open": 1, 2024-03-19T03:42:16.601 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_stale": 0, 2024-03-19T03:42:16.601 INFO:teuthology.orchestra.run.smithi117.stdout: "total_load": 844, 2024-03-19T03:42:16.601 INFO:teuthology.orchestra.run.smithi117.stdout: "average_load": 844, 2024-03-19T03:42:16.601 INFO:teuthology.orchestra.run.smithi117.stdout: "avg_session_uptime": 457, 2024-03-19T03:42:16.601 INFO:teuthology.orchestra.run.smithi117.stdout: "mdthresh_evicted": 0 2024-03-19T03:42:16.601 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.601 INFO:teuthology.orchestra.run.smithi117.stdout: "mempool": { 2024-03-19T03:42:16.601 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_bytes": 0, 2024-03-19T03:42:16.601 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_items": 0, 2024-03-19T03:42:16.601 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T03:42:16.602 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_items": 0, 2024-03-19T03:42:16.602 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T03:42:16.602 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_items": 0, 2024-03-19T03:42:16.602 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T03:42:16.602 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T03:42:16.602 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T03:42:16.602 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T03:42:16.602 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T03:42:16.602 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_items": 0, 2024-03-19T03:42:16.602 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T03:42:16.602 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T03:42:16.602 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_bytes": 0, 2024-03-19T03:42:16.602 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_items": 0, 2024-03-19T03:42:16.603 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_bytes": 0, 2024-03-19T03:42:16.603 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_items": 0, 2024-03-19T03:42:16.603 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T03:42:16.603 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T03:42:16.603 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T03:42:16.603 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T03:42:16.603 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T03:42:16.603 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_items": 0, 2024-03-19T03:42:16.603 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_bytes": 0, 2024-03-19T03:42:16.603 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_items": 0, 2024-03-19T03:42:16.603 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T03:42:16.603 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T03:42:16.603 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_bytes": 0, 2024-03-19T03:42:16.603 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_items": 0, 2024-03-19T03:42:16.603 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_bytes": 0, 2024-03-19T03:42:16.603 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_items": 0, 2024-03-19T03:42:16.603 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T03:42:16.604 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_items": 0, 2024-03-19T03:42:16.604 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T03:42:16.604 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_items": 0, 2024-03-19T03:42:16.604 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_bytes": 299986, 2024-03-19T03:42:16.604 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_items": 130, 2024-03-19T03:42:16.604 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_bytes": 528, 2024-03-19T03:42:16.604 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_items": 6, 2024-03-19T03:42:16.604 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_bytes": 0, 2024-03-19T03:42:16.604 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_items": 0, 2024-03-19T03:42:16.604 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_bytes": 0, 2024-03-19T03:42:16.604 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_items": 0, 2024-03-19T03:42:16.604 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_bytes": 0, 2024-03-19T03:42:16.604 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_items": 0, 2024-03-19T03:42:16.604 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_bytes": 9184, 2024-03-19T03:42:16.604 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_items": 177, 2024-03-19T03:42:16.605 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T03:42:16.605 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_items": 0, 2024-03-19T03:42:16.605 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_bytes": 0, 2024-03-19T03:42:16.605 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_items": 0, 2024-03-19T03:42:16.605 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_bytes": 114957, 2024-03-19T03:42:16.605 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_items": 488, 2024-03-19T03:42:16.605 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_bytes": 0, 2024-03-19T03:42:16.605 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_items": 0, 2024-03-19T03:42:16.605 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_bytes": 0, 2024-03-19T03:42:16.605 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_items": 0 2024-03-19T03:42:16.605 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.605 INFO:teuthology.orchestra.run.smithi117.stdout: "objecter": { 2024-03-19T03:42:16.605 INFO:teuthology.orchestra.run.smithi117.stdout: "op_active": 0, 2024-03-19T03:42:16.605 INFO:teuthology.orchestra.run.smithi117.stdout: "op_laggy": 0, 2024-03-19T03:42:16.606 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send": 1895, 2024-03-19T03:42:16.606 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send_bytes": 5866125, 2024-03-19T03:42:16.606 INFO:teuthology.orchestra.run.smithi117.stdout: "op_resend": 33, 2024-03-19T03:42:16.606 INFO:teuthology.orchestra.run.smithi117.stdout: "op_reply": 1895, 2024-03-19T03:42:16.606 INFO:teuthology.orchestra.run.smithi117.stdout: "op_latency": { 2024-03-19T03:42:16.606 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1895, 2024-03-19T03:42:16.606 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 31.055396454, 2024-03-19T03:42:16.606 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.016388072 2024-03-19T03:42:16.606 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.606 INFO:teuthology.orchestra.run.smithi117.stdout: "op_inflight": 0, 2024-03-19T03:42:16.606 INFO:teuthology.orchestra.run.smithi117.stdout: "oplen_avg": { 2024-03-19T03:42:16.606 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1895, 2024-03-19T03:42:16.606 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 2077 2024-03-19T03:42:16.607 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.607 INFO:teuthology.orchestra.run.smithi117.stdout: "op": 1895, 2024-03-19T03:42:16.607 INFO:teuthology.orchestra.run.smithi117.stdout: "op_r": 60, 2024-03-19T03:42:16.607 INFO:teuthology.orchestra.run.smithi117.stdout: "op_w": 1835, 2024-03-19T03:42:16.607 INFO:teuthology.orchestra.run.smithi117.stdout: "op_rmw": 0, 2024-03-19T03:42:16.607 INFO:teuthology.orchestra.run.smithi117.stdout: "op_pg": 0, 2024-03-19T03:42:16.607 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_stat": 0, 2024-03-19T03:42:16.607 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_create": 10, 2024-03-19T03:42:16.607 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_read": 60, 2024-03-19T03:42:16.607 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_write": 238, 2024-03-19T03:42:16.607 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writefull": 63, 2024-03-19T03:42:16.607 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writesame": 0, 2024-03-19T03:42:16.607 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_append": 0, 2024-03-19T03:42:16.607 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_zero": 0, 2024-03-19T03:42:16.607 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_truncate": 2, 2024-03-19T03:42:16.608 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_delete": 1371, 2024-03-19T03:42:16.608 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_mapext": 0, 2024-03-19T03:42:16.608 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_sparse_read": 0, 2024-03-19T03:42:16.608 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_clonerange": 0, 2024-03-19T03:42:16.608 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_getxattr": 0, 2024-03-19T03:42:16.608 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_setxattr": 20, 2024-03-19T03:42:16.608 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_cmpxattr": 0, 2024-03-19T03:42:16.608 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_rmxattr": 0, 2024-03-19T03:42:16.608 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_resetxattrs": 0, 2024-03-19T03:42:16.608 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_call": 0, 2024-03-19T03:42:16.608 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_watch": 0, 2024-03-19T03:42:16.609 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_notify": 0, 2024-03-19T03:42:16.609 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T03:42:16.609 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls": 0, 2024-03-19T03:42:16.609 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls_filter": 0, 2024-03-19T03:42:16.609 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_other": 0, 2024-03-19T03:42:16.609 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_active": 0, 2024-03-19T03:42:16.609 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_send": 0, 2024-03-19T03:42:16.609 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_resend": 0, 2024-03-19T03:42:16.609 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_ping": 0, 2024-03-19T03:42:16.609 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_active": 0, 2024-03-19T03:42:16.609 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_send": 0, 2024-03-19T03:42:16.609 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_resend": 0, 2024-03-19T03:42:16.609 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_active": 0, 2024-03-19T03:42:16.609 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_send": 0, 2024-03-19T03:42:16.610 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_resend": 0, 2024-03-19T03:42:16.610 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_active": 0, 2024-03-19T03:42:16.610 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_send": 0, 2024-03-19T03:42:16.610 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_resend": 0, 2024-03-19T03:42:16.610 INFO:teuthology.orchestra.run.smithi117.stdout: "command_active": 0, 2024-03-19T03:42:16.610 INFO:teuthology.orchestra.run.smithi117.stdout: "command_send": 0, 2024-03-19T03:42:16.610 INFO:teuthology.orchestra.run.smithi117.stdout: "command_resend": 0, 2024-03-19T03:42:16.610 INFO:teuthology.orchestra.run.smithi117.stdout: "map_epoch": 93, 2024-03-19T03:42:16.610 INFO:teuthology.orchestra.run.smithi117.stdout: "map_full": 0, 2024-03-19T03:42:16.610 INFO:teuthology.orchestra.run.smithi117.stdout: "map_inc": 4, 2024-03-19T03:42:16.610 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_sessions": 12, 2024-03-19T03:42:16.610 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_open": 12, 2024-03-19T03:42:16.610 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_close": 0, 2024-03-19T03:42:16.611 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_laggy": 0, 2024-03-19T03:42:16.611 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_wr": 257, 2024-03-19T03:42:16.611 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_rd": 0, 2024-03-19T03:42:16.611 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_del": 56 2024-03-19T03:42:16.611 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.611 INFO:teuthology.orchestra.run.smithi117.stdout: "oft": { 2024-03-19T03:42:16.611 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_objs": 1, 2024-03-19T03:42:16.611 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_kv_pairs": 0, 2024-03-19T03:42:16.611 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_updates": 0, 2024-03-19T03:42:16.611 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_removes": 0 2024-03-19T03:42:16.611 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.611 INFO:teuthology.orchestra.run.smithi117.stdout: "osd-slow-ops": { 2024-03-19T03:42:16.611 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_ops_count": 0 2024-03-19T03:42:16.611 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.611 INFO:teuthology.orchestra.run.smithi117.stdout: "purge_queue": { 2024-03-19T03:42:16.612 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops": 0, 2024-03-19T03:42:16.612 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops_high_water": 10, 2024-03-19T03:42:16.612 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing": 0, 2024-03-19T03:42:16.612 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_high_water": 5, 2024-03-19T03:42:16.612 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed_ops": 1358, 2024-03-19T03:42:16.612 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed": 679, 2024-03-19T03:42:16.612 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_item_in_journal": 0 2024-03-19T03:42:16.612 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.612 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T03:42:16.612 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-19T03:42:16.612 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-19T03:42:16.612 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-19T03:42:16.612 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 26917, 2024-03-19T03:42:16.612 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 9007767, 2024-03-19T03:42:16.612 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:16.613 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 26917, 2024-03-19T03:42:16.613 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-19T03:42:16.613 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-19T03:42:16.613 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 26917, 2024-03-19T03:42:16.613 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 9007767, 2024-03-19T03:42:16.613 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-19T03:42:16.613 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.613 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.613 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.613 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:16.613 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.613 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_bytes": { 2024-03-19T03:42:16.613 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-19T03:42:16.613 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-19T03:42:16.614 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-19T03:42:16.614 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-19T03:42:16.614 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-19T03:42:16.614 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:16.614 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:16.614 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 1895, 2024-03-19T03:42:16.614 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 5940136, 2024-03-19T03:42:16.614 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 1895, 2024-03-19T03:42:16.614 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 5940136, 2024-03-19T03:42:16.614 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-19T03:42:16.614 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.614 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.614 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.614 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:16.615 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.615 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_ops": { 2024-03-19T03:42:16.615 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-19T03:42:16.615 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 1024, 2024-03-19T03:42:16.615 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-19T03:42:16.615 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-19T03:42:16.615 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-19T03:42:16.615 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:16.615 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:16.615 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 1895, 2024-03-19T03:42:16.615 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 1895, 2024-03-19T03:42:16.615 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 1895, 2024-03-19T03:42:16.615 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 1895, 2024-03-19T03:42:16.615 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-19T03:42:16.615 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.615 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.616 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.616 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:16.616 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.616 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle": { 2024-03-19T03:42:16.616 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-19T03:42:16.616 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-19T03:42:16.616 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-19T03:42:16.616 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 679, 2024-03-19T03:42:16.616 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 74011, 2024-03-19T03:42:16.616 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:16.616 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 679, 2024-03-19T03:42:16.616 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-19T03:42:16.616 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-19T03:42:16.616 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 60, 2024-03-19T03:42:16.616 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 74011, 2024-03-19T03:42:16.616 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-19T03:42:16.617 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.617 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.617 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.617 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:16.617 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:16.617 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle-0x557e0301c500": { 2024-03-19T03:42:16.617 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-19T03:42:16.617 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-19T03:42:16.617 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-19T03:42:16.617 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 3520, 2024-03-19T03:42:16.617 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 5330256, 2024-03-19T03:42:16.617 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:16.617 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 3520, 2024-03-19T03:42:16.617 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-19T03:42:16.617 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-19T03:42:16.617 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 178, 2024-03-19T03:42:16.617 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 5330256, 2024-03-19T03:42:16.618 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-19T03:42:16.618 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:16.618 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:16.618 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:16.618 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:16.618 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:16.618 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-19T03:42:16.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:16 smithi119 ceph-mon[66861]: pgmap v672: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 416 KiB/s wr, 261 op/s 2024-03-19T03:42:17.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:16 smithi117 ceph-mon[31838]: pgmap v672: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 416 KiB/s wr, 261 op/s 2024-03-19T03:42:17.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:16 smithi178 ceph-mon[67578]: pgmap v672: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 416 KiB/s wr, 261 op/s 2024-03-19T03:42:17.474 INFO:tasks.check_counter:Daemon mds.a mds.exported=16 2024-03-19T03:42:17.474 INFO:tasks.check_counter:Daemon mds.a mds.imported=2 2024-03-19T03:42:17.474 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update=288 2024-03-19T03:42:17.474 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update_receipt=4472 2024-03-19T03:42:17.474 INFO:tasks.check_counter:Daemon mds.a mds.dir_split=2 2024-03-19T03:42:17.474 DEBUG:tasks.check_counter:Getting stats from d 2024-03-19T03:42:17.474 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-19T03:42:17.907 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:42:18.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:18 smithi119 ceph-mon[66861]: pgmap v673: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 418 KiB/s wr, 207 op/s 2024-03-19T03:42:19.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:18 smithi117 ceph-mon[31838]: pgmap v673: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 418 KiB/s wr, 207 op/s 2024-03-19T03:42:19.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:18 smithi178 ceph-mon[67578]: pgmap v673: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 418 KiB/s wr, 207 op/s 2024-03-19T03:42:20.120 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-19T03:42:20.120 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T03:42:20.121 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 0, 2024-03-19T03:42:20.121 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 2, 2024-03-19T03:42:20.121 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 0, 2024-03-19T03:42:20.121 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 0, 2024-03-19T03:42:20.121 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 4, 2024-03-19T03:42:20.121 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 0, 2024-03-19T03:42:20.121 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.003558361, 2024-03-19T03:42:20.121 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.000000000, 2024-03-19T03:42:20.121 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.000732299, 2024-03-19T03:42:20.121 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-19T03:42:20.121 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:20.121 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:20.121 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:20.122 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:20.122 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:20.122 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:20.122 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:20.122 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:20.122 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:20.122 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:20.122 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T03:42:20.122 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T03:42:20.122 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:20.122 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T03:42:20.122 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 5, 2024-03-19T03:42:20.123 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 153, 2024-03-19T03:42:20.123 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 3232, 2024-03-19T03:42:20.123 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 104816, 2024-03-19T03:42:20.123 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 3, 2024-03-19T03:42:20.123 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 2, 2024-03-19T03:42:20.123 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.022360817, 2024-03-19T03:42:20.123 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.017086789, 2024-03-19T03:42:20.123 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.003122886, 2024-03-19T03:42:20.123 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-19T03:42:20.123 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:20.123 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 152, 2024-03-19T03:42:20.123 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.022289074, 2024-03-19T03:42:20.123 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000146638 2024-03-19T03:42:20.123 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:20.124 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:20.124 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:20.124 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:20.124 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:20.124 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:20.124 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 3232, 2024-03-19T03:42:20.124 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 104816 2024-03-19T03:42:20.124 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:20.124 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T03:42:20.124 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 204, 2024-03-19T03:42:20.124 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 199, 2024-03-19T03:42:20.124 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 746896, 2024-03-19T03:42:20.124 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 122144, 2024-03-19T03:42:20.124 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 2, 2024-03-19T03:42:20.125 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 1, 2024-03-19T03:42:20.125 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.059416373, 2024-03-19T03:42:20.125 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.027228083, 2024-03-19T03:42:20.125 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.026900717, 2024-03-19T03:42:20.125 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.006926566, 2024-03-19T03:42:20.125 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:20.125 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 198, 2024-03-19T03:42:20.125 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.030238671, 2024-03-19T03:42:20.125 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000152720 2024-03-19T03:42:20.125 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:20.125 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:20.125 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:20.125 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:20.125 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:20.126 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:20.126 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 746896, 2024-03-19T03:42:20.126 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 122144 2024-03-19T03:42:20.126 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:20.126 INFO:teuthology.orchestra.run.smithi117.stdout: "cct": { 2024-03-19T03:42:20.126 INFO:teuthology.orchestra.run.smithi117.stdout: "total_workers": 0, 2024-03-19T03:42:20.126 INFO:teuthology.orchestra.run.smithi117.stdout: "unhealthy_workers": 0 2024-03-19T03:42:20.126 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:20.126 INFO:teuthology.orchestra.run.smithi117.stdout: "mempool": { 2024-03-19T03:42:20.126 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_bytes": 0, 2024-03-19T03:42:20.126 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_items": 0, 2024-03-19T03:42:20.126 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T03:42:20.126 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_items": 0, 2024-03-19T03:42:20.127 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T03:42:20.127 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_items": 0, 2024-03-19T03:42:20.127 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T03:42:20.127 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T03:42:20.127 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T03:42:20.127 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T03:42:20.127 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T03:42:20.127 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_items": 0, 2024-03-19T03:42:20.127 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T03:42:20.127 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T03:42:20.127 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_bytes": 0, 2024-03-19T03:42:20.127 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_items": 0, 2024-03-19T03:42:20.127 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_bytes": 0, 2024-03-19T03:42:20.127 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_items": 0, 2024-03-19T03:42:20.128 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T03:42:20.128 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T03:42:20.128 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T03:42:20.128 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T03:42:20.128 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T03:42:20.128 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_items": 0, 2024-03-19T03:42:20.128 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_bytes": 0, 2024-03-19T03:42:20.128 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_items": 0, 2024-03-19T03:42:20.128 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T03:42:20.128 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T03:42:20.128 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_bytes": 0, 2024-03-19T03:42:20.128 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_items": 0, 2024-03-19T03:42:20.128 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_bytes": 0, 2024-03-19T03:42:20.128 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_items": 0, 2024-03-19T03:42:20.129 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T03:42:20.129 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_items": 0, 2024-03-19T03:42:20.129 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T03:42:20.129 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_items": 0, 2024-03-19T03:42:20.129 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_bytes": 149998, 2024-03-19T03:42:20.129 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_items": 30, 2024-03-19T03:42:20.129 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_bytes": 0, 2024-03-19T03:42:20.129 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_items": 0, 2024-03-19T03:42:20.129 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_bytes": 0, 2024-03-19T03:42:20.129 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_items": 0, 2024-03-19T03:42:20.129 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_bytes": 0, 2024-03-19T03:42:20.129 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_items": 0, 2024-03-19T03:42:20.129 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_bytes": 0, 2024-03-19T03:42:20.130 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_items": 0, 2024-03-19T03:42:20.130 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_bytes": 0, 2024-03-19T03:42:20.130 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_items": 0, 2024-03-19T03:42:20.130 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T03:42:20.130 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_items": 0, 2024-03-19T03:42:20.130 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_bytes": 0, 2024-03-19T03:42:20.130 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_items": 0, 2024-03-19T03:42:20.130 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_bytes": 928, 2024-03-19T03:42:20.130 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_items": 2, 2024-03-19T03:42:20.130 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_bytes": 0, 2024-03-19T03:42:20.130 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_items": 0, 2024-03-19T03:42:20.130 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_bytes": 0, 2024-03-19T03:42:20.130 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_items": 0 2024-03-19T03:42:20.130 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:20.131 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T03:42:20.131 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-19T03:42:20.131 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-19T03:42:20.131 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-19T03:42:20.131 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 205, 2024-03-19T03:42:20.131 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 718111, 2024-03-19T03:42:20.131 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:20.131 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 205, 2024-03-19T03:42:20.131 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-19T03:42:20.131 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-19T03:42:20.131 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 205, 2024-03-19T03:42:20.131 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 718111, 2024-03-19T03:42:20.131 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-19T03:42:20.132 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:20.132 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:20.132 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:20.132 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:20.132 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:20.132 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-19T03:42:20.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:20 smithi117 ceph-mon[31838]: pgmap v674: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 42 KiB/s wr, 91 op/s 2024-03-19T03:42:20.920 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-19T03:42:20.920 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-19T03:42:20.920 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-03-19T03:42:20.920 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-03-19T03:42:20.920 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-03-19T03:42:20.920 DEBUG:tasks.check_counter:Getting stats from g 2024-03-19T03:42:20.920 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-19T03:42:20.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:20 smithi119 ceph-mon[66861]: pgmap v674: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 42 KiB/s wr, 91 op/s 2024-03-19T03:42:21.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:20 smithi178 ceph-mon[67578]: pgmap v674: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 42 KiB/s wr, 91 op/s 2024-03-19T03:42:21.369 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:42:22.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:22 smithi119 ceph-mon[66861]: pgmap v675: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 42 KiB/s wr, 75 op/s 2024-03-19T03:42:23.022 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:22 smithi117 ceph-mon[31838]: pgmap v675: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 42 KiB/s wr, 75 op/s 2024-03-19T03:42:23.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:22 smithi178 ceph-mon[67578]: pgmap v675: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 42 KiB/s wr, 75 op/s 2024-03-19T03:42:23.543 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-19T03:42:23.543 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T03:42:23.543 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 16796, 2024-03-19T03:42:23.543 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 14779, 2024-03-19T03:42:23.543 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 7780722, 2024-03-19T03:42:23.543 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 7610733, 2024-03-19T03:42:23.544 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 9, 2024-03-19T03:42:23.544 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 4, 2024-03-19T03:42:23.544 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 2.692319628, 2024-03-19T03:42:23.544 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.865221254, 2024-03-19T03:42:23.544 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 1.769921642, 2024-03-19T03:42:23.544 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.109738330, 2024-03-19T03:42:23.544 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:23.544 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 14781, 2024-03-19T03:42:23.544 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 4.537347002, 2024-03-19T03:42:23.544 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000306971 2024-03-19T03:42:23.544 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.544 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:23.544 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 28351, 2024-03-19T03:42:23.544 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.012555983, 2024-03-19T03:42:23.545 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000442 2024-03-19T03:42:23.545 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.545 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 913472, 2024-03-19T03:42:23.545 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 127360 2024-03-19T03:42:23.545 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.545 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T03:42:23.545 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 1665, 2024-03-19T03:42:23.545 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 1889, 2024-03-19T03:42:23.545 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 489660, 2024-03-19T03:42:23.545 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 5504238, 2024-03-19T03:42:23.545 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 9, 2024-03-19T03:42:23.545 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 7, 2024-03-19T03:42:23.545 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.425986996, 2024-03-19T03:42:23.545 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.186410570, 2024-03-19T03:42:23.545 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.086431483, 2024-03-19T03:42:23.546 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.118960783, 2024-03-19T03:42:23.546 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:23.546 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1887, 2024-03-19T03:42:23.546 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.843392446, 2024-03-19T03:42:23.546 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000446948 2024-03-19T03:42:23.546 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.546 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:23.546 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 559, 2024-03-19T03:42:23.546 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000121905, 2024-03-19T03:42:23.546 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000218 2024-03-19T03:42:23.546 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.546 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-19T03:42:23.546 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 331280 2024-03-19T03:42:23.547 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.547 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T03:42:23.547 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 16176, 2024-03-19T03:42:23.547 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 21003, 2024-03-19T03:42:23.547 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 5585416, 2024-03-19T03:42:23.547 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 11835747, 2024-03-19T03:42:23.547 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 14, 2024-03-19T03:42:23.547 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 10, 2024-03-19T03:42:23.547 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 6.019313566, 2024-03-19T03:42:23.547 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 4.684495179, 2024-03-19T03:42:23.547 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 3.349810958, 2024-03-19T03:42:23.547 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.085094777, 2024-03-19T03:42:23.547 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:23.547 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 20999, 2024-03-19T03:42:23.548 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 9.325468967, 2024-03-19T03:42:23.548 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000444091 2024-03-19T03:42:23.548 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.548 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:23.548 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 16207, 2024-03-19T03:42:23.548 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.009215635, 2024-03-19T03:42:23.548 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000568 2024-03-19T03:42:23.548 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.548 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 928, 2024-03-19T03:42:23.548 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 736 2024-03-19T03:42:23.548 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.548 INFO:teuthology.orchestra.run.smithi117.stdout: "cct": { 2024-03-19T03:42:23.548 INFO:teuthology.orchestra.run.smithi117.stdout: "total_workers": 1, 2024-03-19T03:42:23.548 INFO:teuthology.orchestra.run.smithi117.stdout: "unhealthy_workers": 0 2024-03-19T03:42:23.548 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.549 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-MDSRank": { 2024-03-19T03:42:23.549 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-19T03:42:23.549 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-19T03:42:23.549 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 732, 2024-03-19T03:42:23.549 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 8.814394943, 2024-03-19T03:42:23.549 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.012041523 2024-03-19T03:42:23.549 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:23.549 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.549 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-PurgeQueue": { 2024-03-19T03:42:23.549 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-19T03:42:23.549 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-19T03:42:23.549 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2691, 2024-03-19T03:42:23.549 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.721520568, 2024-03-19T03:42:23.549 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000268123 2024-03-19T03:42:23.550 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:23.550 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.550 INFO:teuthology.orchestra.run.smithi117.stdout: "mds": { 2024-03-19T03:42:23.550 INFO:teuthology.orchestra.run.smithi117.stdout: "request": 8473, 2024-03-19T03:42:23.550 INFO:teuthology.orchestra.run.smithi117.stdout: "reply": 7759, 2024-03-19T03:42:23.550 INFO:teuthology.orchestra.run.smithi117.stdout: "reply_latency": { 2024-03-19T03:42:23.550 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 7759, 2024-03-19T03:42:23.550 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 159.615701468, 2024-03-19T03:42:23.550 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.020571684 2024-03-19T03:42:23.550 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.550 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_reply": 0, 2024-03-19T03:42:23.550 INFO:teuthology.orchestra.run.smithi117.stdout: "forward": 714, 2024-03-19T03:42:23.550 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_complete": 0, 2024-03-19T03:42:23.551 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_keys": 0, 2024-03-19T03:42:23.551 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_commit": 29, 2024-03-19T03:42:23.551 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_split": 2, 2024-03-19T03:42:23.551 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_merge": 0, 2024-03-19T03:42:23.551 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes": 24, 2024-03-19T03:42:23.551 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_top": 6, 2024-03-19T03:42:23.551 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_bottom": 8, 2024-03-19T03:42:23.551 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pin_tail": 10, 2024-03-19T03:42:23.551 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pinned": 13, 2024-03-19T03:42:23.551 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_expired": 386, 2024-03-19T03:42:23.551 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_with_caps": 0, 2024-03-19T03:42:23.551 INFO:teuthology.orchestra.run.smithi117.stdout: "caps": 0, 2024-03-19T03:42:23.551 INFO:teuthology.orchestra.run.smithi117.stdout: "subtrees": 3, 2024-03-19T03:42:23.551 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse": 21188, 2024-03-19T03:42:23.552 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_hit": 15377, 2024-03-19T03:42:23.552 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_forward": 714, 2024-03-19T03:42:23.552 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_discover": 199, 2024-03-19T03:42:23.552 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_dir_fetch": 0, 2024-03-19T03:42:23.552 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_remote_ino": 0, 2024-03-19T03:42:23.552 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_lock": 150, 2024-03-19T03:42:23.552 INFO:teuthology.orchestra.run.smithi117.stdout: "load_cent": 0, 2024-03-19T03:42:23.552 INFO:teuthology.orchestra.run.smithi117.stdout: "q": 0, 2024-03-19T03:42:23.552 INFO:teuthology.orchestra.run.smithi117.stdout: "exported": 25, 2024-03-19T03:42:23.552 INFO:teuthology.orchestra.run.smithi117.stdout: "exported_inodes": 19, 2024-03-19T03:42:23.552 INFO:teuthology.orchestra.run.smithi117.stdout: "imported": 11, 2024-03-19T03:42:23.553 INFO:teuthology.orchestra.run.smithi117.stdout: "imported_inodes": 752, 2024-03-19T03:42:23.553 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_dir_fetch": 0, 2024-03-19T03:42:23.553 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_backtrace_fetch": 0, 2024-03-19T03:42:23.553 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_peer_discover": 0, 2024-03-19T03:42:23.553 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rfiles": 3308, 2024-03-19T03:42:23.553 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rbytes": 68731964, 2024-03-19T03:42:23.553 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rsnaps": 0, 2024-03-19T03:42:23.553 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T03:42:23.553 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_set_tag": 0, 2024-03-19T03:42:23.553 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T03:42:23.553 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_inotable_repaired": 0, 2024-03-19T03:42:23.553 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_inodes": 0, 2024-03-19T03:42:23.553 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T03:42:23.554 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T03:42:23.554 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_file_inodes": 0, 2024-03-19T03:42:23.554 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_inode_file_caps": 5, 2024-03-19T03:42:23.554 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_revoke": 742, 2024-03-19T03:42:23.554 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_grant": 2575, 2024-03-19T03:42:23.554 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T03:42:23.554 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T03:42:23.554 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T03:42:23.554 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps": 2690, 2024-03-19T03:42:23.554 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps_dirty": 1261, 2024-03-19T03:42:23.554 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_cap_release": 110, 2024-03-19T03:42:23.554 INFO:teuthology.orchestra.run.smithi117.stdout: "process_request_cap_release": 5217 2024-03-19T03:42:23.554 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.555 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_cache": { 2024-03-19T03:42:23.555 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update": 591, 2024-03-19T03:42:23.555 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update_receipt": 4428, 2024-03-19T03:42:23.555 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_try_discover": 3019, 2024-03-19T03:42:23.555 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_send_discover": 3121, 2024-03-19T03:42:23.555 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_handle_discover": 303, 2024-03-19T03:42:23.555 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays": 0, 2024-03-19T03:42:23.555 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_delayed": 0, 2024-03-19T03:42:23.555 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_enqueuing": 0, 2024-03-19T03:42:23.555 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_created": 1413, 2024-03-19T03:42:23.556 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_enqueued": 1413, 2024-03-19T03:42:23.556 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_reintegrated": 0, 2024-03-19T03:42:23.556 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_migrated": 0, 2024-03-19T03:42:23.556 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_processing": 0, 2024-03-19T03:42:23.556 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_enqueued": 0, 2024-03-19T03:42:23.556 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_prioritized": 0, 2024-03-19T03:42:23.556 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_started": 0, 2024-03-19T03:42:23.556 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_completed": 0, 2024-03-19T03:42:23.556 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_quiesce_path": 0, 2024-03-19T03:42:23.556 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_quiesce_inode": 0, 2024-03-19T03:42:23.556 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T03:42:23.556 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_exportdir": 25, 2024-03-19T03:42:23.556 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_flush": 0, 2024-03-19T03:42:23.556 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragmentdir": 2, 2024-03-19T03:42:23.557 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragstats": 0, 2024-03-19T03:42:23.557 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_inodestats": 0 2024-03-19T03:42:23.557 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.557 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_log": { 2024-03-19T03:42:23.557 INFO:teuthology.orchestra.run.smithi117.stdout: "evlrg": 0, 2024-03-19T03:42:23.557 INFO:teuthology.orchestra.run.smithi117.stdout: "evadd": 6429, 2024-03-19T03:42:23.557 INFO:teuthology.orchestra.run.smithi117.stdout: "evex": 0, 2024-03-19T03:42:23.557 INFO:teuthology.orchestra.run.smithi117.stdout: "evtrm": 0, 2024-03-19T03:42:23.557 INFO:teuthology.orchestra.run.smithi117.stdout: "ev": 6429, 2024-03-19T03:42:23.557 INFO:teuthology.orchestra.run.smithi117.stdout: "evexg": 0, 2024-03-19T03:42:23.557 INFO:teuthology.orchestra.run.smithi117.stdout: "evexd": 0, 2024-03-19T03:42:23.558 INFO:teuthology.orchestra.run.smithi117.stdout: "segadd": 9, 2024-03-19T03:42:23.558 INFO:teuthology.orchestra.run.smithi117.stdout: "segex": 0, 2024-03-19T03:42:23.558 INFO:teuthology.orchestra.run.smithi117.stdout: "segtrm": 0, 2024-03-19T03:42:23.558 INFO:teuthology.orchestra.run.smithi117.stdout: "seg": 9, 2024-03-19T03:42:23.558 INFO:teuthology.orchestra.run.smithi117.stdout: "segmjr": 2, 2024-03-19T03:42:23.558 INFO:teuthology.orchestra.run.smithi117.stdout: "segexg": 0, 2024-03-19T03:42:23.558 INFO:teuthology.orchestra.run.smithi117.stdout: "segexd": 0, 2024-03-19T03:42:23.558 INFO:teuthology.orchestra.run.smithi117.stdout: "expos": 4194304, 2024-03-19T03:42:23.558 INFO:teuthology.orchestra.run.smithi117.stdout: "wrpos": 14178433, 2024-03-19T03:42:23.558 INFO:teuthology.orchestra.run.smithi117.stdout: "rdpos": 0, 2024-03-19T03:42:23.559 INFO:teuthology.orchestra.run.smithi117.stdout: "jlat": { 2024-03-19T03:42:23.559 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 269, 2024-03-19T03:42:23.559 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 5.707667213, 2024-03-19T03:42:23.559 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.021218093 2024-03-19T03:42:23.559 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.559 INFO:teuthology.orchestra.run.smithi117.stdout: "replayed": 0 2024-03-19T03:42:23.559 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.560 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_mem": { 2024-03-19T03:42:23.560 INFO:teuthology.orchestra.run.smithi117.stdout: "ino": 28, 2024-03-19T03:42:23.560 INFO:teuthology.orchestra.run.smithi117.stdout: "ino+": 1623, 2024-03-19T03:42:23.560 INFO:teuthology.orchestra.run.smithi117.stdout: "ino-": 1595, 2024-03-19T03:42:23.560 INFO:teuthology.orchestra.run.smithi117.stdout: "dir": 27, 2024-03-19T03:42:23.560 INFO:teuthology.orchestra.run.smithi117.stdout: "dir+": 1300, 2024-03-19T03:42:23.560 INFO:teuthology.orchestra.run.smithi117.stdout: "dir-": 1273, 2024-03-19T03:42:23.560 INFO:teuthology.orchestra.run.smithi117.stdout: "dn": 24, 2024-03-19T03:42:23.560 INFO:teuthology.orchestra.run.smithi117.stdout: "dn+": 2989, 2024-03-19T03:42:23.560 INFO:teuthology.orchestra.run.smithi117.stdout: "dn-": 2965, 2024-03-19T03:42:23.560 INFO:teuthology.orchestra.run.smithi117.stdout: "cap": 0, 2024-03-19T03:42:23.560 INFO:teuthology.orchestra.run.smithi117.stdout: "cap+": 2075, 2024-03-19T03:42:23.561 INFO:teuthology.orchestra.run.smithi117.stdout: "cap-": 2075, 2024-03-19T03:42:23.561 INFO:teuthology.orchestra.run.smithi117.stdout: "rss": 65208, 2024-03-19T03:42:23.561 INFO:teuthology.orchestra.run.smithi117.stdout: "heap": 223612 2024-03-19T03:42:23.561 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.561 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_server": { 2024-03-19T03:42:23.561 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_client_request": 11589, 2024-03-19T03:42:23.561 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_server_request": 33, 2024-03-19T03:42:23.561 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_request": 8473, 2024-03-19T03:42:23.561 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_session": 11, 2024-03-19T03:42:23.561 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_peer_request": 2793, 2024-03-19T03:42:23.561 INFO:teuthology.orchestra.run.smithi117.stdout: "req_create_latency": { 2024-03-19T03:42:23.561 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1061, 2024-03-19T03:42:23.561 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 11.958452956, 2024-03-19T03:42:23.561 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.011270926 2024-03-19T03:42:23.562 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.562 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getattr_latency": { 2024-03-19T03:42:23.562 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.562 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.562 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.562 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.562 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getfilelock_latency": { 2024-03-19T03:42:23.562 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.562 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.562 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.562 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.562 INFO:teuthology.orchestra.run.smithi117.stdout: "req_link_latency": { 2024-03-19T03:42:23.562 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.563 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.563 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.563 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.563 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookup_latency": { 2024-03-19T03:42:23.563 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 4554, 2024-03-19T03:42:23.563 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 132.573581804, 2024-03-19T03:42:23.563 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.029111458 2024-03-19T03:42:23.563 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.563 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookuphash_latency": { 2024-03-19T03:42:23.563 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.563 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.563 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.563 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.563 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupino_latency": { 2024-03-19T03:42:23.563 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.564 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.564 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.564 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.564 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupname_latency": { 2024-03-19T03:42:23.564 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.564 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.564 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.564 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.564 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupparent_latency": { 2024-03-19T03:42:23.564 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.564 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.564 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.564 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.564 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupsnap_latency": { 2024-03-19T03:42:23.564 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.565 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.565 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.565 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.565 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lssnap_latency": { 2024-03-19T03:42:23.565 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.565 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.566 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.566 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.566 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mkdir_latency": { 2024-03-19T03:42:23.566 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.566 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.566 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.566 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.566 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mknod_latency": { 2024-03-19T03:42:23.566 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.566 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.566 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.566 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.566 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mksnap_latency": { 2024-03-19T03:42:23.566 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.567 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.567 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.567 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.567 INFO:teuthology.orchestra.run.smithi117.stdout: "req_open_latency": { 2024-03-19T03:42:23.567 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 474, 2024-03-19T03:42:23.567 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.948100175, 2024-03-19T03:42:23.567 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.002000211 2024-03-19T03:42:23.567 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.567 INFO:teuthology.orchestra.run.smithi117.stdout: "req_readdir_latency": { 2024-03-19T03:42:23.567 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 33, 2024-03-19T03:42:23.567 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.036550823, 2024-03-19T03:42:23.567 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.001107600 2024-03-19T03:42:23.567 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.567 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rename_latency": { 2024-03-19T03:42:23.568 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1062, 2024-03-19T03:42:23.568 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 12.807521928, 2024-03-19T03:42:23.568 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.012059813 2024-03-19T03:42:23.568 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.568 INFO:teuthology.orchestra.run.smithi117.stdout: "req_renamesnap_latency": { 2024-03-19T03:42:23.568 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.568 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.568 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.568 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.568 INFO:teuthology.orchestra.run.smithi117.stdout: "req_snapdiff_latency": { 2024-03-19T03:42:23.568 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.568 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.568 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.568 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.569 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmdir_latency": { 2024-03-19T03:42:23.569 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.569 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.569 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.569 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.569 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmsnap_latency": { 2024-03-19T03:42:23.569 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.569 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.569 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.569 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.569 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmxattr_latency": { 2024-03-19T03:42:23.569 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.569 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.570 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.570 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.570 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setattr_latency": { 2024-03-19T03:42:23.570 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.570 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.570 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.570 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.570 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setdirlayout_latency": { 2024-03-19T03:42:23.570 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.570 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.571 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.571 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.571 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setfilelock_latency": { 2024-03-19T03:42:23.571 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.571 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.571 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.571 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.571 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setlayout_latency": { 2024-03-19T03:42:23.571 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.571 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.571 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.571 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.571 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setxattr_latency": { 2024-03-19T03:42:23.572 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.572 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.572 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.572 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.572 INFO:teuthology.orchestra.run.smithi117.stdout: "req_symlink_latency": { 2024-03-19T03:42:23.572 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.572 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.572 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.572 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.572 INFO:teuthology.orchestra.run.smithi117.stdout: "req_unlink_latency": { 2024-03-19T03:42:23.572 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 575, 2024-03-19T03:42:23.572 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 1.291493782, 2024-03-19T03:42:23.572 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.002246076 2024-03-19T03:42:23.572 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.573 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_revoke_eviction": 0, 2024-03-19T03:42:23.573 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_acquisition_throttle": 0, 2024-03-19T03:42:23.573 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getvxattr_latency": { 2024-03-19T03:42:23.573 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.573 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.573 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.573 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:23.573 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.573 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_sessions": { 2024-03-19T03:42:23.573 INFO:teuthology.orchestra.run.smithi117.stdout: "session_count": 1, 2024-03-19T03:42:23.573 INFO:teuthology.orchestra.run.smithi117.stdout: "session_add": 2, 2024-03-19T03:42:23.573 INFO:teuthology.orchestra.run.smithi117.stdout: "session_remove": 1, 2024-03-19T03:42:23.573 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_open": 1, 2024-03-19T03:42:23.574 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_stale": 0, 2024-03-19T03:42:23.574 INFO:teuthology.orchestra.run.smithi117.stdout: "total_load": 1322, 2024-03-19T03:42:23.574 INFO:teuthology.orchestra.run.smithi117.stdout: "average_load": 1322, 2024-03-19T03:42:23.574 INFO:teuthology.orchestra.run.smithi117.stdout: "avg_session_uptime": 463, 2024-03-19T03:42:23.574 INFO:teuthology.orchestra.run.smithi117.stdout: "mdthresh_evicted": 0 2024-03-19T03:42:23.574 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.574 INFO:teuthology.orchestra.run.smithi117.stdout: "mempool": { 2024-03-19T03:42:23.574 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_bytes": 0, 2024-03-19T03:42:23.574 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_items": 0, 2024-03-19T03:42:23.574 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T03:42:23.574 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_items": 0, 2024-03-19T03:42:23.574 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T03:42:23.574 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_items": 0, 2024-03-19T03:42:23.574 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T03:42:23.574 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T03:42:23.575 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T03:42:23.575 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T03:42:23.575 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T03:42:23.575 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_items": 0, 2024-03-19T03:42:23.575 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T03:42:23.575 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T03:42:23.575 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_bytes": 0, 2024-03-19T03:42:23.575 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_items": 0, 2024-03-19T03:42:23.575 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_bytes": 0, 2024-03-19T03:42:23.575 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_items": 0, 2024-03-19T03:42:23.575 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T03:42:23.575 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T03:42:23.575 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T03:42:23.576 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T03:42:23.576 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T03:42:23.576 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_items": 0, 2024-03-19T03:42:23.576 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_bytes": 0, 2024-03-19T03:42:23.576 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_items": 0, 2024-03-19T03:42:23.576 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T03:42:23.576 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T03:42:23.576 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_bytes": 0, 2024-03-19T03:42:23.576 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_items": 0, 2024-03-19T03:42:23.576 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_bytes": 0, 2024-03-19T03:42:23.576 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_items": 0, 2024-03-19T03:42:23.576 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T03:42:23.576 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_items": 0, 2024-03-19T03:42:23.576 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T03:42:23.577 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_items": 0, 2024-03-19T03:42:23.577 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_bytes": 310847, 2024-03-19T03:42:23.577 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_items": 139, 2024-03-19T03:42:23.577 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_bytes": 880, 2024-03-19T03:42:23.577 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_items": 10, 2024-03-19T03:42:23.577 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_bytes": 0, 2024-03-19T03:42:23.577 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_items": 0, 2024-03-19T03:42:23.577 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_bytes": 0, 2024-03-19T03:42:23.577 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_items": 0, 2024-03-19T03:42:23.577 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_bytes": 0, 2024-03-19T03:42:23.577 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_items": 0, 2024-03-19T03:42:23.577 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_bytes": 9184, 2024-03-19T03:42:23.577 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_items": 177, 2024-03-19T03:42:23.578 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T03:42:23.578 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_items": 0, 2024-03-19T03:42:23.578 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_bytes": 0, 2024-03-19T03:42:23.578 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_items": 0, 2024-03-19T03:42:23.578 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_bytes": 114557, 2024-03-19T03:42:23.578 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_items": 462, 2024-03-19T03:42:23.578 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_bytes": 0, 2024-03-19T03:42:23.578 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_items": 0, 2024-03-19T03:42:23.578 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_bytes": 0, 2024-03-19T03:42:23.578 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_items": 0 2024-03-19T03:42:23.578 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.578 INFO:teuthology.orchestra.run.smithi117.stdout: "objecter": { 2024-03-19T03:42:23.578 INFO:teuthology.orchestra.run.smithi117.stdout: "op_active": 0, 2024-03-19T03:42:23.579 INFO:teuthology.orchestra.run.smithi117.stdout: "op_laggy": 0, 2024-03-19T03:42:23.579 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send": 3523, 2024-03-19T03:42:23.579 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send_bytes": 10653660, 2024-03-19T03:42:23.579 INFO:teuthology.orchestra.run.smithi117.stdout: "op_resend": 33, 2024-03-19T03:42:23.579 INFO:teuthology.orchestra.run.smithi117.stdout: "op_reply": 3523, 2024-03-19T03:42:23.579 INFO:teuthology.orchestra.run.smithi117.stdout: "op_latency": { 2024-03-19T03:42:23.579 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 3523, 2024-03-19T03:42:23.579 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 68.527666117, 2024-03-19T03:42:23.579 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.019451508 2024-03-19T03:42:23.579 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.579 INFO:teuthology.orchestra.run.smithi117.stdout: "op_inflight": 0, 2024-03-19T03:42:23.579 INFO:teuthology.orchestra.run.smithi117.stdout: "oplen_avg": { 2024-03-19T03:42:23.579 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 3523, 2024-03-19T03:42:23.580 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 3739 2024-03-19T03:42:23.580 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.580 INFO:teuthology.orchestra.run.smithi117.stdout: "op": 3523, 2024-03-19T03:42:23.580 INFO:teuthology.orchestra.run.smithi117.stdout: "op_r": 75, 2024-03-19T03:42:23.580 INFO:teuthology.orchestra.run.smithi117.stdout: "op_w": 3448, 2024-03-19T03:42:23.580 INFO:teuthology.orchestra.run.smithi117.stdout: "op_rmw": 0, 2024-03-19T03:42:23.580 INFO:teuthology.orchestra.run.smithi117.stdout: "op_pg": 0, 2024-03-19T03:42:23.580 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_stat": 0, 2024-03-19T03:42:23.580 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_create": 10, 2024-03-19T03:42:23.580 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_read": 75, 2024-03-19T03:42:23.580 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_write": 345, 2024-03-19T03:42:23.580 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writefull": 73, 2024-03-19T03:42:23.580 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writesame": 0, 2024-03-19T03:42:23.580 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_append": 0, 2024-03-19T03:42:23.580 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_zero": 0, 2024-03-19T03:42:23.581 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_truncate": 2, 2024-03-19T03:42:23.581 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_delete": 2840, 2024-03-19T03:42:23.581 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_mapext": 0, 2024-03-19T03:42:23.581 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_sparse_read": 0, 2024-03-19T03:42:23.581 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_clonerange": 0, 2024-03-19T03:42:23.581 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_getxattr": 0, 2024-03-19T03:42:23.581 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_setxattr": 20, 2024-03-19T03:42:23.581 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_cmpxattr": 0, 2024-03-19T03:42:23.581 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_rmxattr": 0, 2024-03-19T03:42:23.581 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_resetxattrs": 0, 2024-03-19T03:42:23.581 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_call": 0, 2024-03-19T03:42:23.581 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_watch": 0, 2024-03-19T03:42:23.581 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_notify": 0, 2024-03-19T03:42:23.581 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T03:42:23.582 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls": 0, 2024-03-19T03:42:23.582 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls_filter": 0, 2024-03-19T03:42:23.582 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_other": 0, 2024-03-19T03:42:23.582 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_active": 0, 2024-03-19T03:42:23.582 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_send": 0, 2024-03-19T03:42:23.582 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_resend": 0, 2024-03-19T03:42:23.582 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_ping": 0, 2024-03-19T03:42:23.582 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_active": 0, 2024-03-19T03:42:23.582 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_send": 0, 2024-03-19T03:42:23.582 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_resend": 0, 2024-03-19T03:42:23.582 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_active": 0, 2024-03-19T03:42:23.582 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_send": 0, 2024-03-19T03:42:23.582 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_resend": 0, 2024-03-19T03:42:23.582 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_active": 0, 2024-03-19T03:42:23.583 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_send": 0, 2024-03-19T03:42:23.583 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_resend": 0, 2024-03-19T03:42:23.583 INFO:teuthology.orchestra.run.smithi117.stdout: "command_active": 0, 2024-03-19T03:42:23.583 INFO:teuthology.orchestra.run.smithi117.stdout: "command_send": 0, 2024-03-19T03:42:23.583 INFO:teuthology.orchestra.run.smithi117.stdout: "command_resend": 0, 2024-03-19T03:42:23.583 INFO:teuthology.orchestra.run.smithi117.stdout: "map_epoch": 93, 2024-03-19T03:42:23.583 INFO:teuthology.orchestra.run.smithi117.stdout: "map_full": 0, 2024-03-19T03:42:23.583 INFO:teuthology.orchestra.run.smithi117.stdout: "map_inc": 4, 2024-03-19T03:42:23.583 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_sessions": 12, 2024-03-19T03:42:23.583 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_open": 12, 2024-03-19T03:42:23.583 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_close": 0, 2024-03-19T03:42:23.583 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_laggy": 0, 2024-03-19T03:42:23.583 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_wr": 301, 2024-03-19T03:42:23.583 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_rd": 0, 2024-03-19T03:42:23.583 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_del": 73 2024-03-19T03:42:23.583 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.584 INFO:teuthology.orchestra.run.smithi117.stdout: "oft": { 2024-03-19T03:42:23.584 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_objs": 1, 2024-03-19T03:42:23.584 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_kv_pairs": 0, 2024-03-19T03:42:23.584 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_updates": 0, 2024-03-19T03:42:23.584 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_removes": 0 2024-03-19T03:42:23.584 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.584 INFO:teuthology.orchestra.run.smithi117.stdout: "osd-slow-ops": { 2024-03-19T03:42:23.584 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_ops_count": 0 2024-03-19T03:42:23.584 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.584 INFO:teuthology.orchestra.run.smithi117.stdout: "purge_queue": { 2024-03-19T03:42:23.584 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops": 0, 2024-03-19T03:42:23.584 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops_high_water": 10, 2024-03-19T03:42:23.584 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing": 0, 2024-03-19T03:42:23.584 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_high_water": 5, 2024-03-19T03:42:23.585 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed_ops": 2826, 2024-03-19T03:42:23.585 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed": 1413, 2024-03-19T03:42:23.585 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_item_in_journal": 0 2024-03-19T03:42:23.585 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.585 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T03:42:23.585 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-19T03:42:23.585 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-19T03:42:23.585 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-19T03:42:23.585 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 34633, 2024-03-19T03:42:23.585 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 10956406, 2024-03-19T03:42:23.585 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:23.585 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 34633, 2024-03-19T03:42:23.585 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-19T03:42:23.585 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-19T03:42:23.586 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 34633, 2024-03-19T03:42:23.586 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 10956406, 2024-03-19T03:42:23.586 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-19T03:42:23.586 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.586 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.586 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.586 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:23.586 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.586 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_bytes": { 2024-03-19T03:42:23.586 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-19T03:42:23.586 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-19T03:42:23.586 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-19T03:42:23.586 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-19T03:42:23.586 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-19T03:42:23.586 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:23.587 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:23.587 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 3523, 2024-03-19T03:42:23.587 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 10807677, 2024-03-19T03:42:23.587 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 3523, 2024-03-19T03:42:23.587 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 10807677, 2024-03-19T03:42:23.587 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-19T03:42:23.587 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.587 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.587 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.587 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:23.587 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.587 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_ops": { 2024-03-19T03:42:23.587 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-19T03:42:23.587 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 1024, 2024-03-19T03:42:23.587 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-19T03:42:23.588 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-19T03:42:23.588 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-19T03:42:23.588 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:23.588 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:23.588 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 3523, 2024-03-19T03:42:23.588 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 3523, 2024-03-19T03:42:23.588 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 3523, 2024-03-19T03:42:23.588 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 3523, 2024-03-19T03:42:23.588 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-19T03:42:23.588 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.588 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.588 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.588 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:23.588 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.588 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle": { 2024-03-19T03:42:23.589 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-19T03:42:23.589 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-19T03:42:23.589 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-19T03:42:23.589 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 1413, 2024-03-19T03:42:23.589 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 154017, 2024-03-19T03:42:23.589 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:23.589 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 1413, 2024-03-19T03:42:23.589 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-19T03:42:23.589 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-19T03:42:23.589 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 76, 2024-03-19T03:42:23.589 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 154017, 2024-03-19T03:42:23.589 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-19T03:42:23.589 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.589 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.589 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.589 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:23.590 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:23.590 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle-0x55906caa6500": { 2024-03-19T03:42:23.590 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-19T03:42:23.590 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-19T03:42:23.590 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-19T03:42:23.590 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 6429, 2024-03-19T03:42:23.590 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 9984129, 2024-03-19T03:42:23.590 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:23.590 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 6429, 2024-03-19T03:42:23.591 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-19T03:42:23.591 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-19T03:42:23.591 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 269, 2024-03-19T03:42:23.591 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 9984129, 2024-03-19T03:42:23.591 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-19T03:42:23.591 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:23.591 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:23.591 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:23.591 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:23.591 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:23.591 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-19T03:42:24.372 INFO:tasks.check_counter:Daemon mds.g mds.exported=25 2024-03-19T03:42:24.372 INFO:tasks.check_counter:Daemon mds.g mds.imported=11 2024-03-19T03:42:24.372 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update=591 2024-03-19T03:42:24.372 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update_receipt=4428 2024-03-19T03:42:24.372 INFO:tasks.check_counter:Daemon mds.g mds.dir_split=2 2024-03-19T03:42:24.372 DEBUG:tasks.check_counter:Getting stats from j 2024-03-19T03:42:24.372 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-19T03:42:24.696 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:24 smithi117 ceph-mon[31838]: pgmap v676: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 20 KiB/s wr, 27 op/s 2024-03-19T03:42:24.844 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T03:42:24.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:24 smithi119 ceph-mon[66861]: pgmap v676: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 20 KiB/s wr, 27 op/s 2024-03-19T03:42:25.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:24 smithi178 ceph-mon[67578]: pgmap v676: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 20 KiB/s wr, 27 op/s 2024-03-19T03:42:26.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:26 smithi119 ceph-mon[66861]: pgmap v677: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.0 KiB/s wr, 12 op/s 2024-03-19T03:42:27.015 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:26 smithi117 ceph-mon[31838]: pgmap v677: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.0 KiB/s wr, 12 op/s 2024-03-19T03:42:27.017 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-19T03:42:27.017 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T03:42:27.017 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 0, 2024-03-19T03:42:27.018 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 2, 2024-03-19T03:42:27.018 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 0, 2024-03-19T03:42:27.018 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 0, 2024-03-19T03:42:27.018 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 4, 2024-03-19T03:42:27.018 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 0, 2024-03-19T03:42:27.018 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.003510733, 2024-03-19T03:42:27.018 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.000000000, 2024-03-19T03:42:27.018 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.000804605, 2024-03-19T03:42:27.018 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-19T03:42:27.018 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:27.018 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:27.018 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:27.018 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:27.018 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:27.019 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:27.019 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:27.019 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:27.019 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:27.019 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:27.019 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T03:42:27.019 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T03:42:27.019 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:27.019 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T03:42:27.019 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 207, 2024-03-19T03:42:27.019 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 202, 2024-03-19T03:42:27.019 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 747712, 2024-03-19T03:42:27.019 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 124048, 2024-03-19T03:42:27.019 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 2, 2024-03-19T03:42:27.019 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 1, 2024-03-19T03:42:27.019 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.073376613, 2024-03-19T03:42:27.020 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.032669943, 2024-03-19T03:42:27.020 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.037931728, 2024-03-19T03:42:27.020 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.007821489, 2024-03-19T03:42:27.020 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:27.020 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 201, 2024-03-19T03:42:27.020 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.030237816, 2024-03-19T03:42:27.020 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000150436 2024-03-19T03:42:27.020 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:27.020 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:27.020 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:27.020 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:27.020 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:27.020 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:27.020 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 747712, 2024-03-19T03:42:27.021 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 124048 2024-03-19T03:42:27.021 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:27.021 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T03:42:27.021 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 5, 2024-03-19T03:42:27.021 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 155, 2024-03-19T03:42:27.021 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 3232, 2024-03-19T03:42:27.021 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 105168, 2024-03-19T03:42:27.021 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 3, 2024-03-19T03:42:27.021 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 2, 2024-03-19T03:42:27.021 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.022980381, 2024-03-19T03:42:27.021 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.017887296, 2024-03-19T03:42:27.021 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.002836722, 2024-03-19T03:42:27.021 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-19T03:42:27.021 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:27.022 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 154, 2024-03-19T03:42:27.022 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.025094423, 2024-03-19T03:42:27.022 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000162950 2024-03-19T03:42:27.022 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:27.022 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:27.022 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:27.022 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:27.022 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:27.022 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:27.022 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 3232, 2024-03-19T03:42:27.022 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 105168 2024-03-19T03:42:27.022 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:27.022 INFO:teuthology.orchestra.run.smithi117.stdout: "cct": { 2024-03-19T03:42:27.022 INFO:teuthology.orchestra.run.smithi117.stdout: "total_workers": 0, 2024-03-19T03:42:27.023 INFO:teuthology.orchestra.run.smithi117.stdout: "unhealthy_workers": 0 2024-03-19T03:42:27.023 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:27.023 INFO:teuthology.orchestra.run.smithi117.stdout: "mempool": { 2024-03-19T03:42:27.023 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_bytes": 0, 2024-03-19T03:42:27.023 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_items": 0, 2024-03-19T03:42:27.023 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T03:42:27.023 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_items": 0, 2024-03-19T03:42:27.023 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T03:42:27.023 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_items": 0, 2024-03-19T03:42:27.023 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T03:42:27.023 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T03:42:27.023 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T03:42:27.023 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T03:42:27.023 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T03:42:27.024 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_items": 0, 2024-03-19T03:42:27.024 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T03:42:27.024 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T03:42:27.024 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_bytes": 0, 2024-03-19T03:42:27.024 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_items": 0, 2024-03-19T03:42:27.024 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_bytes": 0, 2024-03-19T03:42:27.024 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_items": 0, 2024-03-19T03:42:27.024 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T03:42:27.024 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T03:42:27.024 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T03:42:27.024 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T03:42:27.024 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T03:42:27.024 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_items": 0, 2024-03-19T03:42:27.024 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_bytes": 0, 2024-03-19T03:42:27.025 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_items": 0, 2024-03-19T03:42:27.025 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T03:42:27.025 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T03:42:27.025 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_bytes": 0, 2024-03-19T03:42:27.025 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_items": 0, 2024-03-19T03:42:27.025 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_bytes": 0, 2024-03-19T03:42:27.025 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_items": 0, 2024-03-19T03:42:27.025 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T03:42:27.025 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_items": 0, 2024-03-19T03:42:27.025 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T03:42:27.025 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_items": 0, 2024-03-19T03:42:27.025 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_bytes": 149998, 2024-03-19T03:42:27.025 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_items": 30, 2024-03-19T03:42:27.025 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_bytes": 0, 2024-03-19T03:42:27.025 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_items": 0, 2024-03-19T03:42:27.025 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_bytes": 0, 2024-03-19T03:42:27.026 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_items": 0, 2024-03-19T03:42:27.026 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_bytes": 0, 2024-03-19T03:42:27.026 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_items": 0, 2024-03-19T03:42:27.026 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_bytes": 0, 2024-03-19T03:42:27.026 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_items": 0, 2024-03-19T03:42:27.026 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_bytes": 0, 2024-03-19T03:42:27.026 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_items": 0, 2024-03-19T03:42:27.026 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T03:42:27.026 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_items": 0, 2024-03-19T03:42:27.026 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_bytes": 0, 2024-03-19T03:42:27.026 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_items": 0, 2024-03-19T03:42:27.026 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_bytes": 928, 2024-03-19T03:42:27.026 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_items": 2, 2024-03-19T03:42:27.026 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_bytes": 0, 2024-03-19T03:42:27.027 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_items": 0, 2024-03-19T03:42:27.027 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_bytes": 0, 2024-03-19T03:42:27.027 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_items": 0 2024-03-19T03:42:27.027 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-19T03:42:27.027 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T03:42:27.027 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-19T03:42:27.027 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-19T03:42:27.027 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-19T03:42:27.027 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 208, 2024-03-19T03:42:27.027 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 718501, 2024-03-19T03:42:27.027 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:27.027 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 208, 2024-03-19T03:42:27.027 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-19T03:42:27.027 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-19T03:42:27.028 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 208, 2024-03-19T03:42:27.028 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 718501, 2024-03-19T03:42:27.028 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-19T03:42:27.028 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-19T03:42:27.028 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-19T03:42:27.028 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-19T03:42:27.028 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:27.028 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-19T03:42:27.028 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-19T03:42:27.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:26 smithi178 ceph-mon[67578]: pgmap v677: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.0 KiB/s wr, 12 op/s 2024-03-19T03:42:27.821 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-19T03:42:27.822 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-19T03:42:27.822 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-03-19T03:42:27.822 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-03-19T03:42:27.822 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-03-19T03:42:27.822 DEBUG:tasks.check_counter:Getting stats from b 2024-03-19T03:42:27.822 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-19T03:42:28.315 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T03:42:28.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:28 smithi119 ceph-mon[66861]: pgmap v678: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.3 KiB/s wr, 13 op/s 2024-03-19T03:42:29.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:28 smithi117 ceph-mon[31838]: pgmap v678: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.3 KiB/s wr, 13 op/s 2024-03-19T03:42:29.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:28 smithi178 ceph-mon[67578]: pgmap v678: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.3 KiB/s wr, 13 op/s 2024-03-19T03:42:30.497 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-19T03:42:30.497 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T03:42:30.497 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 62015, 2024-03-19T03:42:30.497 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 68534, 2024-03-19T03:42:30.497 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 16146499, 2024-03-19T03:42:30.497 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 252682395, 2024-03-19T03:42:30.497 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 10, 2024-03-19T03:42:30.497 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 3, 2024-03-19T03:42:30.497 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 7.680986358, 2024-03-19T03:42:30.497 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 3.520418673, 2024-03-19T03:42:30.497 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 2.028139286, 2024-03-19T03:42:30.497 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 2.006660754, 2024-03-19T03:42:30.497 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 68539, 2024-03-19T03:42:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 41.566211261, 2024-03-19T03:42:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000606460 2024-03-19T03:42:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 48615, 2024-03-19T03:42:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.010249665, 2024-03-19T03:42:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000210 2024-03-19T03:42:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-19T03:42:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 448 2024-03-19T03:42:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T03:42:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 300268, 2024-03-19T03:42:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 484477, 2024-03-19T03:42:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 102079496, 2024-03-19T03:42:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 424602679, 2024-03-19T03:42:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 17, 2024-03-19T03:42:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 7, 2024-03-19T03:42:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 19.738103443, 2024-03-19T03:42:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 9.324154944, 2024-03-19T03:42:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 32.632597043, 2024-03-19T03:42:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 2.278411366, 2024-03-19T03:42:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 484472, 2024-03-19T03:42:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 85.363826781, 2024-03-19T03:42:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000176199 2024-03-19T03:42:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 370015, 2024-03-19T03:42:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.095828963, 2024-03-19T03:42:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000258 2024-03-19T03:42:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 57856, 2024-03-19T03:42:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 454688 2024-03-19T03:42:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T03:42:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 59620, 2024-03-19T03:42:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 63659, 2024-03-19T03:42:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 15893737, 2024-03-19T03:42:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 161550561, 2024-03-19T03:42:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 13, 2024-03-19T03:42:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 7, 2024-03-19T03:42:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 8.285256283, 2024-03-19T03:42:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 4.576359452, 2024-03-19T03:42:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 2.098423084, 2024-03-19T03:42:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 1.637165242, 2024-03-19T03:42:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 63655, 2024-03-19T03:42:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 32.132142977, 2024-03-19T03:42:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000504785 2024-03-19T03:42:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 57326, 2024-03-19T03:42:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.011628991, 2024-03-19T03:42:30.502 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000202 2024-03-19T03:42:30.502 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.502 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 958048, 2024-03-19T03:42:30.502 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 130640 2024-03-19T03:42:30.502 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.502 INFO:teuthology.orchestra.run.smithi119.stdout: "cct": { 2024-03-19T03:42:30.502 INFO:teuthology.orchestra.run.smithi119.stdout: "total_workers": 1, 2024-03-19T03:42:30.502 INFO:teuthology.orchestra.run.smithi119.stdout: "unhealthy_workers": 0 2024-03-19T03:42:30.502 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.502 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-MDSRank": { 2024-03-19T03:42:30.502 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-03-19T03:42:30.502 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-03-19T03:42:30.502 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 3213, 2024-03-19T03:42:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 293.309004997, 2024-03-19T03:42:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.091288205 2024-03-19T03:42:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-PurgeQueue": { 2024-03-19T03:42:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-03-19T03:42:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-03-19T03:42:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 79699, 2024-03-19T03:42:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 40.494274816, 2024-03-19T03:42:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000508090 2024-03-19T03:42:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": { 2024-03-19T03:42:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: "request": 151093, 2024-03-19T03:42:30.504 INFO:teuthology.orchestra.run.smithi119.stdout: "reply": 148060, 2024-03-19T03:42:30.504 INFO:teuthology.orchestra.run.smithi119.stdout: "reply_latency": { 2024-03-19T03:42:30.504 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 148060, 2024-03-19T03:42:30.504 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 3875.609104458, 2024-03-19T03:42:30.504 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.026175936 2024-03-19T03:42:30.504 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.504 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_reply": 0, 2024-03-19T03:42:30.504 INFO:teuthology.orchestra.run.smithi119.stdout: "forward": 3033, 2024-03-19T03:42:30.504 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_complete": 0, 2024-03-19T03:42:30.504 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_keys": 0, 2024-03-19T03:42:30.504 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_commit": 1845, 2024-03-19T03:42:30.504 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_split": 137, 2024-03-19T03:42:30.504 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_merge": 9, 2024-03-19T03:42:30.504 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes": 62, 2024-03-19T03:42:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_top": 29, 2024-03-19T03:42:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_bottom": 19, 2024-03-19T03:42:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pin_tail": 14, 2024-03-19T03:42:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pinned": 20, 2024-03-19T03:42:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_expired": 4060, 2024-03-19T03:42:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_with_caps": 1, 2024-03-19T03:42:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": 1, 2024-03-19T03:42:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: "subtrees": 6, 2024-03-19T03:42:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse": 221032, 2024-03-19T03:42:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_hit": 208647, 2024-03-19T03:42:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_forward": 3013, 2024-03-19T03:42:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_discover": 27, 2024-03-19T03:42:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_dir_fetch": 0, 2024-03-19T03:42:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_remote_ino": 0, 2024-03-19T03:42:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_lock": 168, 2024-03-19T03:42:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: "load_cent": 0, 2024-03-19T03:42:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: "q": 0, 2024-03-19T03:42:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: "exported": 21, 2024-03-19T03:42:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: "exported_inodes": 2525, 2024-03-19T03:42:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: "imported": 98, 2024-03-19T03:42:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: "imported_inodes": 367, 2024-03-19T03:42:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_dir_fetch": 0, 2024-03-19T03:42:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_backtrace_fetch": 0, 2024-03-19T03:42:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_peer_discover": 0, 2024-03-19T03:42:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rfiles": 4, 2024-03-19T03:42:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rbytes": 234, 2024-03-19T03:42:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rsnaps": 0, 2024-03-19T03:42:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T03:42:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_set_tag": 0, 2024-03-19T03:42:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T03:42:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_inotable_repaired": 0, 2024-03-19T03:42:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_inodes": 0, 2024-03-19T03:42:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T03:42:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T03:42:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_file_inodes": 0, 2024-03-19T03:42:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_inode_file_caps": 28, 2024-03-19T03:42:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_revoke": 35865, 2024-03-19T03:42:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_grant": 89806, 2024-03-19T03:42:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_trunc": 6, 2024-03-19T03:42:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T03:42:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T03:42:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps": 94363, 2024-03-19T03:42:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps_dirty": 43081, 2024-03-19T03:42:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_cap_release": 3233, 2024-03-19T03:42:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: "process_request_cap_release": 88349 2024-03-19T03:42:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache": { 2024-03-19T03:42:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update": 4120, 2024-03-19T03:42:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update_receipt": 0, 2024-03-19T03:42:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_try_discover": 0, 2024-03-19T03:42:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_send_discover": 25, 2024-03-19T03:42:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_handle_discover": 12031, 2024-03-19T03:42:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays": 5, 2024-03-19T03:42:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_delayed": 0, 2024-03-19T03:42:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_enqueuing": 0, 2024-03-19T03:42:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_created": 42461, 2024-03-19T03:42:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_enqueued": 42456, 2024-03-19T03:42:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_reintegrated": 0, 2024-03-19T03:42:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_migrated": 0, 2024-03-19T03:42:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_processing": 0, 2024-03-19T03:42:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_enqueued": 0, 2024-03-19T03:42:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_prioritized": 0, 2024-03-19T03:42:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_started": 0, 2024-03-19T03:42:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_completed": 0, 2024-03-19T03:42:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_path": 0, 2024-03-19T03:42:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_inode": 0, 2024-03-19T03:42:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T03:42:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_exportdir": 21, 2024-03-19T03:42:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_flush": 0, 2024-03-19T03:42:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragmentdir": 146, 2024-03-19T03:42:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragstats": 0, 2024-03-19T03:42:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_inodestats": 0 2024-03-19T03:42:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log": { 2024-03-19T03:42:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: "evlrg": 0, 2024-03-19T03:42:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: "evadd": 232022, 2024-03-19T03:42:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: "evex": 133643, 2024-03-19T03:42:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: "evtrm": 122901, 2024-03-19T03:42:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: "ev": 109121, 2024-03-19T03:42:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: "evexg": 0, 2024-03-19T03:42:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: "evexd": 10742, 2024-03-19T03:42:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: "segadd": 299, 2024-03-19T03:42:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: "segex": 171, 2024-03-19T03:42:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: "segtrm": 158, 2024-03-19T03:42:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: "seg": 141, 2024-03-19T03:42:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: "segmjr": 9, 2024-03-19T03:42:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: "segexg": 0, 2024-03-19T03:42:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: "segexd": 13, 2024-03-19T03:42:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: "expos": 305387685, 2024-03-19T03:42:30.511 INFO:teuthology.orchestra.run.smithi119.stdout: "wrpos": 526284850, 2024-03-19T03:42:30.511 INFO:teuthology.orchestra.run.smithi119.stdout: "rdpos": 0, 2024-03-19T03:42:30.511 INFO:teuthology.orchestra.run.smithi119.stdout: "jlat": { 2024-03-19T03:42:30.511 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2451, 2024-03-19T03:42:30.511 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 813.379997909, 2024-03-19T03:42:30.511 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.331856384 2024-03-19T03:42:30.511 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.511 INFO:teuthology.orchestra.run.smithi119.stdout: "replayed": 0 2024-03-19T03:42:30.511 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.511 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_mem": { 2024-03-19T03:42:30.511 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 41, 2024-03-19T03:42:30.511 INFO:teuthology.orchestra.run.smithi119.stdout: "ino+": 43669, 2024-03-19T03:42:30.511 INFO:teuthology.orchestra.run.smithi119.stdout: "ino-": 43628, 2024-03-19T03:42:30.511 INFO:teuthology.orchestra.run.smithi119.stdout: "dir": 50, 2024-03-19T03:42:30.511 INFO:teuthology.orchestra.run.smithi119.stdout: "dir+": 1405, 2024-03-19T03:42:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: "dir-": 1355, 2024-03-19T03:42:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: "dn": 62, 2024-03-19T03:42:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: "dn+": 98764, 2024-03-19T03:42:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: "dn-": 98702, 2024-03-19T03:42:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: "cap": 1, 2024-03-19T03:42:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: "cap+": 44200, 2024-03-19T03:42:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: "cap-": 44199, 2024-03-19T03:42:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: "rss": 146840, 2024-03-19T03:42:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: "heap": 231804 2024-03-19T03:42:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_server": { 2024-03-19T03:42:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_client_request": 157972, 2024-03-19T03:42:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_server_request": 8648, 2024-03-19T03:42:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_request": 151093, 2024-03-19T03:42:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_session": 42, 2024-03-19T03:42:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_peer_request": 17662, 2024-03-19T03:42:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: "req_create_latency": { 2024-03-19T03:42:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 43433, 2024-03-19T03:42:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 887.226044901, 2024-03-19T03:42:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.020427464 2024-03-19T03:42:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getattr_latency": { 2024-03-19T03:42:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2731, 2024-03-19T03:42:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 965.493228892, 2024-03-19T03:42:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.353531024 2024-03-19T03:42:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getfilelock_latency": { 2024-03-19T03:42:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: "req_link_latency": { 2024-03-19T03:42:30.514 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:30.514 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:30.514 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:30.514 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.514 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookup_latency": { 2024-03-19T03:42:30.514 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 22972, 2024-03-19T03:42:30.514 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 532.363479479, 2024-03-19T03:42:30.514 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.023174450 2024-03-19T03:42:30.514 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.514 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookuphash_latency": { 2024-03-19T03:42:30.514 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:30.514 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:30.514 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:30.514 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.514 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupino_latency": { 2024-03-19T03:42:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupname_latency": { 2024-03-19T03:42:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupparent_latency": { 2024-03-19T03:42:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupsnap_latency": { 2024-03-19T03:42:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lssnap_latency": { 2024-03-19T03:42:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mkdir_latency": { 2024-03-19T03:42:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 182, 2024-03-19T03:42:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 10.948454171, 2024-03-19T03:42:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.060156341 2024-03-19T03:42:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mknod_latency": { 2024-03-19T03:42:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mksnap_latency": { 2024-03-19T03:42:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: "req_open_latency": { 2024-03-19T03:42:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 8963, 2024-03-19T03:42:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 363.899874815, 2024-03-19T03:42:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.040600231 2024-03-19T03:42:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: "req_readdir_latency": { 2024-03-19T03:42:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1144, 2024-03-19T03:42:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 4.977463615, 2024-03-19T03:42:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.004350929 2024-03-19T03:42:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rename_latency": { 2024-03-19T03:42:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 43051, 2024-03-19T03:42:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 1015.408127529, 2024-03-19T03:42:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.023586168 2024-03-19T03:42:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: "req_renamesnap_latency": { 2024-03-19T03:42:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: "req_snapdiff_latency": { 2024-03-19T03:42:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmdir_latency": { 2024-03-19T03:42:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 176, 2024-03-19T03:42:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 31.853471590, 2024-03-19T03:42:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.180985634 2024-03-19T03:42:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmsnap_latency": { 2024-03-19T03:42:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmxattr_latency": { 2024-03-19T03:42:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setattr_latency": { 2024-03-19T03:42:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 8, 2024-03-19T03:42:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.518702875, 2024-03-19T03:42:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.064837859 2024-03-19T03:42:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setdirlayout_latency": { 2024-03-19T03:42:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setfilelock_latency": { 2024-03-19T03:42:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setlayout_latency": { 2024-03-19T03:42:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setxattr_latency": { 2024-03-19T03:42:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 9, 2024-03-19T03:42:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.025660157, 2024-03-19T03:42:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.002851128 2024-03-19T03:42:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: "req_symlink_latency": { 2024-03-19T03:42:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 3, 2024-03-19T03:42:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.003166172, 2024-03-19T03:42:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.001055390 2024-03-19T03:42:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: "req_unlink_latency": { 2024-03-19T03:42:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 25388, 2024-03-19T03:42:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 62.891430262, 2024-03-19T03:42:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.002477210 2024-03-19T03:42:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_revoke_eviction": 0, 2024-03-19T03:42:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_acquisition_throttle": 0, 2024-03-19T03:42:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getvxattr_latency": { 2024-03-19T03:42:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_sessions": { 2024-03-19T03:42:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: "session_count": 1, 2024-03-19T03:42:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: "session_add": 4, 2024-03-19T03:42:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: "session_remove": 3, 2024-03-19T03:42:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_open": 1, 2024-03-19T03:42:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_stale": 0, 2024-03-19T03:42:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: "total_load": 27201, 2024-03-19T03:42:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: "average_load": 27201, 2024-03-19T03:42:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: "avg_session_uptime": 539, 2024-03-19T03:42:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: "mdthresh_evicted": 0 2024-03-19T03:42:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "mempool": { 2024-03-19T03:42:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_bytes": 360, 2024-03-19T03:42:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_items": 360, 2024-03-19T03:42:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T03:42:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_items": 0, 2024-03-19T03:42:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T03:42:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_items": 0, 2024-03-19T03:42:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T03:42:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T03:42:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T03:42:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T03:42:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T03:42:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_items": 0, 2024-03-19T03:42:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T03:42:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T03:42:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_bytes": 0, 2024-03-19T03:42:30.525 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_items": 0, 2024-03-19T03:42:30.525 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_bytes": 0, 2024-03-19T03:42:30.525 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_items": 0, 2024-03-19T03:42:30.525 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T03:42:30.525 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T03:42:30.525 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T03:42:30.525 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T03:42:30.525 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T03:42:30.525 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_items": 0, 2024-03-19T03:42:30.525 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_bytes": 0, 2024-03-19T03:42:30.525 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_items": 0, 2024-03-19T03:42:30.525 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T03:42:30.525 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T03:42:30.525 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_bytes": 0, 2024-03-19T03:42:30.525 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_items": 0, 2024-03-19T03:42:30.525 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_bytes": 0, 2024-03-19T03:42:30.525 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_items": 0, 2024-03-19T03:42:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T03:42:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_items": 0, 2024-03-19T03:42:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T03:42:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_items": 0, 2024-03-19T03:42:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_bytes": 296168, 2024-03-19T03:42:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_items": 110, 2024-03-19T03:42:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_bytes": 176, 2024-03-19T03:42:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_items": 2, 2024-03-19T03:42:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_bytes": 0, 2024-03-19T03:42:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_items": 0, 2024-03-19T03:42:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_bytes": 0, 2024-03-19T03:42:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_items": 0, 2024-03-19T03:42:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_bytes": 0, 2024-03-19T03:42:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_items": 0, 2024-03-19T03:42:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_bytes": 9184, 2024-03-19T03:42:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_items": 177, 2024-03-19T03:42:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T03:42:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_items": 0, 2024-03-19T03:42:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_bytes": 0, 2024-03-19T03:42:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_items": 0, 2024-03-19T03:42:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_bytes": 212110, 2024-03-19T03:42:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_items": 1865, 2024-03-19T03:42:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_bytes": 0, 2024-03-19T03:42:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_items": 0, 2024-03-19T03:42:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_bytes": 0, 2024-03-19T03:42:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_items": 0 2024-03-19T03:42:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: "objecter": { 2024-03-19T03:42:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: "op_active": 0, 2024-03-19T03:42:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: "op_laggy": 0, 2024-03-19T03:42:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send": 119403, 2024-03-19T03:42:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send_bytes": 575824759, 2024-03-19T03:42:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: "op_resend": 37, 2024-03-19T03:42:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: "op_reply": 119403, 2024-03-19T03:42:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: "op_latency": { 2024-03-19T03:42:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 119403, 2024-03-19T03:42:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 6456.645993139, 2024-03-19T03:42:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.054074403 2024-03-19T03:42:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: "op_inflight": 0, 2024-03-19T03:42:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: "oplen_avg": { 2024-03-19T03:42:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 119403, 2024-03-19T03:42:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 161368 2024-03-19T03:42:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: "op": 119403, 2024-03-19T03:42:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: "op_r": 1037, 2024-03-19T03:42:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: "op_w": 118366, 2024-03-19T03:42:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: "op_rmw": 0, 2024-03-19T03:42:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: "op_pg": 0, 2024-03-19T03:42:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_stat": 577, 2024-03-19T03:42:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_create": 25323, 2024-03-19T03:42:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_read": 1037, 2024-03-19T03:42:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_write": 3488, 2024-03-19T03:42:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writefull": 483, 2024-03-19T03:42:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writesame": 0, 2024-03-19T03:42:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_append": 0, 2024-03-19T03:42:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_zero": 0, 2024-03-19T03:42:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_truncate": 137, 2024-03-19T03:42:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_delete": 86624, 2024-03-19T03:42:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_mapext": 0, 2024-03-19T03:42:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_sparse_read": 0, 2024-03-19T03:42:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_clonerange": 0, 2024-03-19T03:42:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_getxattr": 0, 2024-03-19T03:42:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_setxattr": 38058, 2024-03-19T03:42:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_cmpxattr": 0, 2024-03-19T03:42:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_rmxattr": 0, 2024-03-19T03:42:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_resetxattrs": 0, 2024-03-19T03:42:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_call": 0, 2024-03-19T03:42:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_watch": 0, 2024-03-19T03:42:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_notify": 0, 2024-03-19T03:42:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T03:42:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls": 0, 2024-03-19T03:42:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls_filter": 0, 2024-03-19T03:42:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_other": 6, 2024-03-19T03:42:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_active": 0, 2024-03-19T03:42:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_send": 0, 2024-03-19T03:42:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_resend": 0, 2024-03-19T03:42:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_ping": 0, 2024-03-19T03:42:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_active": 0, 2024-03-19T03:42:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_send": 0, 2024-03-19T03:42:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_resend": 0, 2024-03-19T03:42:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_active": 0, 2024-03-19T03:42:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_send": 0, 2024-03-19T03:42:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_resend": 0, 2024-03-19T03:42:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_active": 0, 2024-03-19T03:42:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_send": 0, 2024-03-19T03:42:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_resend": 0, 2024-03-19T03:42:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "command_active": 0, 2024-03-19T03:42:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "command_send": 0, 2024-03-19T03:42:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "command_resend": 0, 2024-03-19T03:42:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: "map_epoch": 93, 2024-03-19T03:42:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: "map_full": 0, 2024-03-19T03:42:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: "map_inc": 9, 2024-03-19T03:42:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_sessions": 12, 2024-03-19T03:42:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_open": 12, 2024-03-19T03:42:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_close": 0, 2024-03-19T03:42:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_laggy": 0, 2024-03-19T03:42:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_wr": 4541, 2024-03-19T03:42:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_rd": 0, 2024-03-19T03:42:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_del": 1094 2024-03-19T03:42:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: "oft": { 2024-03-19T03:42:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_objs": 1, 2024-03-19T03:42:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_kv_pairs": 0, 2024-03-19T03:42:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_updates": 0, 2024-03-19T03:42:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_removes": 0 2024-03-19T03:42:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: "osd-slow-ops": { 2024-03-19T03:42:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_ops_count": 0 2024-03-19T03:42:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: "purge_queue": { 2024-03-19T03:42:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops": 0, 2024-03-19T03:42:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops_high_water": 17, 2024-03-19T03:42:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing": 0, 2024-03-19T03:42:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_high_water": 6, 2024-03-19T03:42:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed_ops": 85874, 2024-03-19T03:42:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed": 42456, 2024-03-19T03:42:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_item_in_journal": 0 2024-03-19T03:42:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T03:42:30.534 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-19T03:42:30.534 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-03-19T03:42:30.534 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-19T03:42:30.534 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 421899, 2024-03-19T03:42:30.534 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 100163406, 2024-03-19T03:42:30.534 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:30.534 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 421899, 2024-03-19T03:42:30.534 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-19T03:42:30.534 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-19T03:42:30.534 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 421899, 2024-03-19T03:42:30.534 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 100163406, 2024-03-19T03:42:30.534 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-19T03:42:30.534 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:30.534 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:30.535 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:30.535 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:30.535 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.535 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_bytes": { 2024-03-19T03:42:30.535 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-19T03:42:30.535 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-03-19T03:42:30.535 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-19T03:42:30.535 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-19T03:42:30.535 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-19T03:42:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 119403, 2024-03-19T03:42:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 580452207, 2024-03-19T03:42:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 119403, 2024-03-19T03:42:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 580452207, 2024-03-19T03:42:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-19T03:42:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_ops": { 2024-03-19T03:42:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-19T03:42:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 1024, 2024-03-19T03:42:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-19T03:42:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-19T03:42:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-19T03:42:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 119403, 2024-03-19T03:42:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 119403, 2024-03-19T03:42:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 119403, 2024-03-19T03:42:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 119403, 2024-03-19T03:42:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-19T03:42:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle": { 2024-03-19T03:42:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-19T03:42:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-03-19T03:42:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-19T03:42:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 42456, 2024-03-19T03:42:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 4627448, 2024-03-19T03:42:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 42456, 2024-03-19T03:42:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-19T03:42:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-19T03:42:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 1037, 2024-03-19T03:42:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 4627448, 2024-03-19T03:42:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-19T03:42:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:30.539 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:30.539 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:30.539 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle-0x55fabdcbe500": { 2024-03-19T03:42:30.539 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-19T03:42:30.539 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-03-19T03:42:30.539 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-19T03:42:30.539 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 232022, 2024-03-19T03:42:30.539 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 522090546, 2024-03-19T03:42:30.539 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:30.539 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 232022, 2024-03-19T03:42:30.539 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-19T03:42:30.539 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-19T03:42:30.539 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 2451, 2024-03-19T03:42:30.539 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 522090546, 2024-03-19T03:42:30.540 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-19T03:42:30.540 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:30.540 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:30.540 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:30.540 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:30.540 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:30.540 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-19T03:42:30.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:30 smithi119 ceph-mon[66861]: pgmap v679: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-19T03:42:31.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:30 smithi117 ceph-mon[31838]: pgmap v679: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-19T03:42:31.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:30 smithi178 ceph-mon[67578]: pgmap v679: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-19T03:42:31.427 INFO:tasks.check_counter:Daemon mds.b mds.exported=21 2024-03-19T03:42:31.427 INFO:tasks.check_counter:Daemon mds.b mds.imported=98 2024-03-19T03:42:31.428 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=4120 2024-03-19T03:42:31.428 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=0 2024-03-19T03:42:31.428 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=137 2024-03-19T03:42:31.428 DEBUG:tasks.check_counter:Getting stats from e 2024-03-19T03:42:31.428 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-19T03:42:31.699 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T03:42:32.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:32 smithi119 ceph-mon[66861]: pgmap v680: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-19T03:42:33.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:32 smithi117 ceph-mon[31838]: pgmap v680: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-19T03:42:33.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:32 smithi178 ceph-mon[67578]: pgmap v680: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-19T03:42:34.006 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-19T03:42:34.007 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T03:42:34.007 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 399, 2024-03-19T03:42:34.007 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 401, 2024-03-19T03:42:34.007 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 4299129, 2024-03-19T03:42:34.007 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 126946, 2024-03-19T03:42:34.007 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 6, 2024-03-19T03:42:34.007 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 2, 2024-03-19T03:42:34.007 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.029785202, 2024-03-19T03:42:34.007 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.009538821, 2024-03-19T03:42:34.007 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.009343170, 2024-03-19T03:42:34.007 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.007962931, 2024-03-19T03:42:34.008 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:34.008 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 399, 2024-03-19T03:42:34.008 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.012346764, 2024-03-19T03:42:34.008 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000030944 2024-03-19T03:42:34.008 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.008 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:34.008 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.008 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.008 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.008 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.008 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T03:42:34.009 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T03:42:34.009 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.009 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T03:42:34.009 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 884, 2024-03-19T03:42:34.009 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 859, 2024-03-19T03:42:34.009 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 5257511, 2024-03-19T03:42:34.009 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 331669, 2024-03-19T03:42:34.009 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 4, 2024-03-19T03:42:34.009 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 3, 2024-03-19T03:42:34.010 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.076924649, 2024-03-19T03:42:34.010 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.020940717, 2024-03-19T03:42:34.010 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.035116737, 2024-03-19T03:42:34.010 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.021632923, 2024-03-19T03:42:34.010 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:34.010 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 858, 2024-03-19T03:42:34.010 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.030046431, 2024-03-19T03:42:34.010 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000035019 2024-03-19T03:42:34.010 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.010 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:34.010 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.011 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.011 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.011 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.011 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 898800, 2024-03-19T03:42:34.011 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 125600 2024-03-19T03:42:34.011 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.011 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T03:42:34.011 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 1020, 2024-03-19T03:42:34.011 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 1170, 2024-03-19T03:42:34.011 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 1911549, 2024-03-19T03:42:34.011 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 658610, 2024-03-19T03:42:34.011 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 5, 2024-03-19T03:42:34.012 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 4, 2024-03-19T03:42:34.012 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.082586936, 2024-03-19T03:42:34.012 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.028127349, 2024-03-19T03:42:34.012 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.020571236, 2024-03-19T03:42:34.012 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.027315930, 2024-03-19T03:42:34.012 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:34.012 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1169, 2024-03-19T03:42:34.012 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.039610964, 2024-03-19T03:42:34.012 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000033884 2024-03-19T03:42:34.012 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.013 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:34.013 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.013 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.013 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.013 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.013 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 3232, 2024-03-19T03:42:34.013 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 335840 2024-03-19T03:42:34.013 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.013 INFO:teuthology.orchestra.run.smithi119.stdout: "cct": { 2024-03-19T03:42:34.013 INFO:teuthology.orchestra.run.smithi119.stdout: "total_workers": 1, 2024-03-19T03:42:34.013 INFO:teuthology.orchestra.run.smithi119.stdout: "unhealthy_workers": 0 2024-03-19T03:42:34.013 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.014 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-MDSRank": { 2024-03-19T03:42:34.014 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-03-19T03:42:34.014 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-03-19T03:42:34.014 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 3503, 2024-03-19T03:42:34.014 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.079722347, 2024-03-19T03:42:34.014 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000022758 2024-03-19T03:42:34.014 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:34.014 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.014 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-PurgeQueue": { 2024-03-19T03:42:34.014 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-03-19T03:42:34.015 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-03-19T03:42:34.015 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.015 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.015 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.015 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:34.015 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.015 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": { 2024-03-19T03:42:34.015 INFO:teuthology.orchestra.run.smithi119.stdout: "request": 0, 2024-03-19T03:42:34.015 INFO:teuthology.orchestra.run.smithi119.stdout: "reply": 0, 2024-03-19T03:42:34.015 INFO:teuthology.orchestra.run.smithi119.stdout: "reply_latency": { 2024-03-19T03:42:34.015 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.016 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.016 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.016 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.016 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_reply": 0, 2024-03-19T03:42:34.016 INFO:teuthology.orchestra.run.smithi119.stdout: "forward": 0, 2024-03-19T03:42:34.016 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_complete": 0, 2024-03-19T03:42:34.016 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_keys": 0, 2024-03-19T03:42:34.016 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_commit": 0, 2024-03-19T03:42:34.016 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_split": 0, 2024-03-19T03:42:34.016 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_merge": 0, 2024-03-19T03:42:34.016 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes": 1420, 2024-03-19T03:42:34.017 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_top": 0, 2024-03-19T03:42:34.017 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_bottom": 1420, 2024-03-19T03:42:34.017 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pin_tail": 0, 2024-03-19T03:42:34.017 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pinned": 1420, 2024-03-19T03:42:34.017 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_expired": 0, 2024-03-19T03:42:34.017 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_with_caps": 0, 2024-03-19T03:42:34.017 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": 0, 2024-03-19T03:42:34.017 INFO:teuthology.orchestra.run.smithi119.stdout: "subtrees": 1, 2024-03-19T03:42:34.017 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse": 0, 2024-03-19T03:42:34.017 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_hit": 0, 2024-03-19T03:42:34.017 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_forward": 0, 2024-03-19T03:42:34.017 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_discover": 0, 2024-03-19T03:42:34.018 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_dir_fetch": 0, 2024-03-19T03:42:34.018 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_remote_ino": 0, 2024-03-19T03:42:34.018 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_lock": 0, 2024-03-19T03:42:34.018 INFO:teuthology.orchestra.run.smithi119.stdout: "load_cent": 0, 2024-03-19T03:42:34.018 INFO:teuthology.orchestra.run.smithi119.stdout: "q": 0, 2024-03-19T03:42:34.018 INFO:teuthology.orchestra.run.smithi119.stdout: "exported": 0, 2024-03-19T03:42:34.018 INFO:teuthology.orchestra.run.smithi119.stdout: "exported_inodes": 0, 2024-03-19T03:42:34.018 INFO:teuthology.orchestra.run.smithi119.stdout: "imported": 0, 2024-03-19T03:42:34.019 INFO:teuthology.orchestra.run.smithi119.stdout: "imported_inodes": 0, 2024-03-19T03:42:34.019 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_dir_fetch": 0, 2024-03-19T03:42:34.019 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_backtrace_fetch": 0, 2024-03-19T03:42:34.019 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_peer_discover": 0, 2024-03-19T03:42:34.019 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rfiles": 0, 2024-03-19T03:42:34.019 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rbytes": 0, 2024-03-19T03:42:34.019 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rsnaps": 0, 2024-03-19T03:42:34.019 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T03:42:34.019 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_set_tag": 0, 2024-03-19T03:42:34.019 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T03:42:34.019 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_inotable_repaired": 0, 2024-03-19T03:42:34.019 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_inodes": 0, 2024-03-19T03:42:34.020 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T03:42:34.020 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T03:42:34.020 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_file_inodes": 0, 2024-03-19T03:42:34.020 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_inode_file_caps": 0, 2024-03-19T03:42:34.020 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_revoke": 0, 2024-03-19T03:42:34.020 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_grant": 0, 2024-03-19T03:42:34.020 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T03:42:34.020 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T03:42:34.020 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T03:42:34.020 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps": 0, 2024-03-19T03:42:34.020 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps_dirty": 0, 2024-03-19T03:42:34.020 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_cap_release": 0, 2024-03-19T03:42:34.020 INFO:teuthology.orchestra.run.smithi119.stdout: "process_request_cap_release": 0 2024-03-19T03:42:34.021 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.021 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache": { 2024-03-19T03:42:34.021 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update": 0, 2024-03-19T03:42:34.021 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update_receipt": 0, 2024-03-19T03:42:34.021 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_try_discover": 0, 2024-03-19T03:42:34.021 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_send_discover": 0, 2024-03-19T03:42:34.021 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_handle_discover": 0, 2024-03-19T03:42:34.021 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays": 0, 2024-03-19T03:42:34.021 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_delayed": 0, 2024-03-19T03:42:34.021 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_enqueuing": 0, 2024-03-19T03:42:34.021 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_created": 0, 2024-03-19T03:42:34.021 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_enqueued": 0, 2024-03-19T03:42:34.021 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_reintegrated": 0, 2024-03-19T03:42:34.021 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_migrated": 0, 2024-03-19T03:42:34.021 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_processing": 0, 2024-03-19T03:42:34.022 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_enqueued": 0, 2024-03-19T03:42:34.022 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_prioritized": 0, 2024-03-19T03:42:34.022 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_started": 0, 2024-03-19T03:42:34.022 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_completed": 0, 2024-03-19T03:42:34.022 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_path": 0, 2024-03-19T03:42:34.022 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_inode": 0, 2024-03-19T03:42:34.022 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T03:42:34.022 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_exportdir": 0, 2024-03-19T03:42:34.022 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_flush": 0, 2024-03-19T03:42:34.022 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragmentdir": 0, 2024-03-19T03:42:34.022 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragstats": 0, 2024-03-19T03:42:34.022 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_inodestats": 0 2024-03-19T03:42:34.022 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.022 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log": { 2024-03-19T03:42:34.022 INFO:teuthology.orchestra.run.smithi119.stdout: "evlrg": 0, 2024-03-19T03:42:34.023 INFO:teuthology.orchestra.run.smithi119.stdout: "evadd": 0, 2024-03-19T03:42:34.023 INFO:teuthology.orchestra.run.smithi119.stdout: "evex": 0, 2024-03-19T03:42:34.023 INFO:teuthology.orchestra.run.smithi119.stdout: "evtrm": 0, 2024-03-19T03:42:34.023 INFO:teuthology.orchestra.run.smithi119.stdout: "ev": 6429, 2024-03-19T03:42:34.023 INFO:teuthology.orchestra.run.smithi119.stdout: "evexg": 0, 2024-03-19T03:42:34.023 INFO:teuthology.orchestra.run.smithi119.stdout: "evexd": 0, 2024-03-19T03:42:34.023 INFO:teuthology.orchestra.run.smithi119.stdout: "segadd": 0, 2024-03-19T03:42:34.023 INFO:teuthology.orchestra.run.smithi119.stdout: "segex": 0, 2024-03-19T03:42:34.023 INFO:teuthology.orchestra.run.smithi119.stdout: "segtrm": 0, 2024-03-19T03:42:34.023 INFO:teuthology.orchestra.run.smithi119.stdout: "seg": 9, 2024-03-19T03:42:34.023 INFO:teuthology.orchestra.run.smithi119.stdout: "segmjr": 2, 2024-03-19T03:42:34.023 INFO:teuthology.orchestra.run.smithi119.stdout: "segexg": 0, 2024-03-19T03:42:34.023 INFO:teuthology.orchestra.run.smithi119.stdout: "segexd": 0, 2024-03-19T03:42:34.023 INFO:teuthology.orchestra.run.smithi119.stdout: "expos": 4194304, 2024-03-19T03:42:34.023 INFO:teuthology.orchestra.run.smithi119.stdout: "wrpos": 14178433, 2024-03-19T03:42:34.024 INFO:teuthology.orchestra.run.smithi119.stdout: "rdpos": 14177295, 2024-03-19T03:42:34.024 INFO:teuthology.orchestra.run.smithi119.stdout: "jlat": { 2024-03-19T03:42:34.024 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.024 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.024 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.024 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.024 INFO:teuthology.orchestra.run.smithi119.stdout: "replayed": 6429 2024-03-19T03:42:34.024 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.024 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_mem": { 2024-03-19T03:42:34.024 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 10, 2024-03-19T03:42:34.024 INFO:teuthology.orchestra.run.smithi119.stdout: "ino+": 1444, 2024-03-19T03:42:34.024 INFO:teuthology.orchestra.run.smithi119.stdout: "ino-": 1434, 2024-03-19T03:42:34.024 INFO:teuthology.orchestra.run.smithi119.stdout: "dir": 8, 2024-03-19T03:42:34.024 INFO:teuthology.orchestra.run.smithi119.stdout: "dir+": 62, 2024-03-19T03:42:34.025 INFO:teuthology.orchestra.run.smithi119.stdout: "dir-": 54, 2024-03-19T03:42:34.025 INFO:teuthology.orchestra.run.smithi119.stdout: "dn": 1420, 2024-03-19T03:42:34.025 INFO:teuthology.orchestra.run.smithi119.stdout: "dn+": 2626, 2024-03-19T03:42:34.025 INFO:teuthology.orchestra.run.smithi119.stdout: "dn-": 1206, 2024-03-19T03:42:34.025 INFO:teuthology.orchestra.run.smithi119.stdout: "cap": 0, 2024-03-19T03:42:34.025 INFO:teuthology.orchestra.run.smithi119.stdout: "cap+": 0, 2024-03-19T03:42:34.025 INFO:teuthology.orchestra.run.smithi119.stdout: "cap-": 0, 2024-03-19T03:42:34.025 INFO:teuthology.orchestra.run.smithi119.stdout: "rss": 52904, 2024-03-19T03:42:34.025 INFO:teuthology.orchestra.run.smithi119.stdout: "heap": 215420 2024-03-19T03:42:34.025 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.025 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_server": { 2024-03-19T03:42:34.025 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_client_request": 0, 2024-03-19T03:42:34.025 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_server_request": 0, 2024-03-19T03:42:34.025 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_request": 0, 2024-03-19T03:42:34.025 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_session": 0, 2024-03-19T03:42:34.026 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_peer_request": 0, 2024-03-19T03:42:34.026 INFO:teuthology.orchestra.run.smithi119.stdout: "req_create_latency": { 2024-03-19T03:42:34.026 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.026 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.026 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.026 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.026 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getattr_latency": { 2024-03-19T03:42:34.026 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.026 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.026 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.026 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.026 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getfilelock_latency": { 2024-03-19T03:42:34.026 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.026 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.026 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.027 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.027 INFO:teuthology.orchestra.run.smithi119.stdout: "req_link_latency": { 2024-03-19T03:42:34.027 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.027 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.027 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.027 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.027 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookup_latency": { 2024-03-19T03:42:34.027 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.027 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.027 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.027 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.027 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookuphash_latency": { 2024-03-19T03:42:34.027 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.027 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.028 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.028 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.028 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupino_latency": { 2024-03-19T03:42:34.028 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.028 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.028 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.028 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.028 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupname_latency": { 2024-03-19T03:42:34.028 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.028 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.028 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.028 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.028 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupparent_latency": { 2024-03-19T03:42:34.028 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.028 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.028 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.029 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.029 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupsnap_latency": { 2024-03-19T03:42:34.029 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.029 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.029 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.029 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.029 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lssnap_latency": { 2024-03-19T03:42:34.029 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.029 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.029 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.029 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.029 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mkdir_latency": { 2024-03-19T03:42:34.030 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.030 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.030 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.030 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.030 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mknod_latency": { 2024-03-19T03:42:34.030 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.030 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.030 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.030 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.031 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mksnap_latency": { 2024-03-19T03:42:34.031 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.031 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.031 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.031 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.031 INFO:teuthology.orchestra.run.smithi119.stdout: "req_open_latency": { 2024-03-19T03:42:34.031 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.031 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.031 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.031 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.031 INFO:teuthology.orchestra.run.smithi119.stdout: "req_readdir_latency": { 2024-03-19T03:42:34.031 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.031 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.031 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.031 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.032 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rename_latency": { 2024-03-19T03:42:34.032 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.032 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.032 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.032 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.032 INFO:teuthology.orchestra.run.smithi119.stdout: "req_renamesnap_latency": { 2024-03-19T03:42:34.032 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.032 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.032 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.032 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.032 INFO:teuthology.orchestra.run.smithi119.stdout: "req_snapdiff_latency": { 2024-03-19T03:42:34.032 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.032 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.032 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.032 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.032 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmdir_latency": { 2024-03-19T03:42:34.033 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.033 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.033 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.033 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.033 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmsnap_latency": { 2024-03-19T03:42:34.033 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.033 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.033 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.033 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.033 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmxattr_latency": { 2024-03-19T03:42:34.033 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.033 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.033 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.033 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.033 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setattr_latency": { 2024-03-19T03:42:34.034 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.034 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.034 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.034 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.034 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setdirlayout_latency": { 2024-03-19T03:42:34.034 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.034 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.034 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.034 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.034 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setfilelock_latency": { 2024-03-19T03:42:34.034 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.034 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.034 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.035 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.035 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setlayout_latency": { 2024-03-19T03:42:34.035 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.035 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.035 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.035 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.035 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setxattr_latency": { 2024-03-19T03:42:34.035 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.035 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.035 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.035 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.035 INFO:teuthology.orchestra.run.smithi119.stdout: "req_symlink_latency": { 2024-03-19T03:42:34.035 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.035 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.035 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.036 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.036 INFO:teuthology.orchestra.run.smithi119.stdout: "req_unlink_latency": { 2024-03-19T03:42:34.036 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.036 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.036 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.036 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.036 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_revoke_eviction": 0, 2024-03-19T03:42:34.036 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_acquisition_throttle": 0, 2024-03-19T03:42:34.036 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getvxattr_latency": { 2024-03-19T03:42:34.036 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.036 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.036 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.036 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:34.036 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.037 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_sessions": { 2024-03-19T03:42:34.037 INFO:teuthology.orchestra.run.smithi119.stdout: "session_count": 1, 2024-03-19T03:42:34.037 INFO:teuthology.orchestra.run.smithi119.stdout: "session_add": 2, 2024-03-19T03:42:34.037 INFO:teuthology.orchestra.run.smithi119.stdout: "session_remove": 1, 2024-03-19T03:42:34.037 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_open": 1, 2024-03-19T03:42:34.037 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_stale": 0, 2024-03-19T03:42:34.037 INFO:teuthology.orchestra.run.smithi119.stdout: "total_load": 0, 2024-03-19T03:42:34.037 INFO:teuthology.orchestra.run.smithi119.stdout: "average_load": 0, 2024-03-19T03:42:34.037 INFO:teuthology.orchestra.run.smithi119.stdout: "avg_session_uptime": 3731, 2024-03-19T03:42:34.037 INFO:teuthology.orchestra.run.smithi119.stdout: "mdthresh_evicted": 0 2024-03-19T03:42:34.037 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.037 INFO:teuthology.orchestra.run.smithi119.stdout: "mempool": { 2024-03-19T03:42:34.037 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_bytes": 0, 2024-03-19T03:42:34.037 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_items": 0, 2024-03-19T03:42:34.037 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T03:42:34.038 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_items": 0, 2024-03-19T03:42:34.038 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T03:42:34.038 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_items": 0, 2024-03-19T03:42:34.038 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T03:42:34.038 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T03:42:34.038 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T03:42:34.038 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T03:42:34.038 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T03:42:34.038 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_items": 0, 2024-03-19T03:42:34.038 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T03:42:34.038 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T03:42:34.038 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_bytes": 0, 2024-03-19T03:42:34.038 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_items": 0, 2024-03-19T03:42:34.038 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_bytes": 0, 2024-03-19T03:42:34.038 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_items": 0, 2024-03-19T03:42:34.039 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T03:42:34.039 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T03:42:34.039 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T03:42:34.039 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T03:42:34.039 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T03:42:34.039 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_items": 0, 2024-03-19T03:42:34.039 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_bytes": 0, 2024-03-19T03:42:34.039 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_items": 0, 2024-03-19T03:42:34.039 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T03:42:34.039 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T03:42:34.039 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_bytes": 0, 2024-03-19T03:42:34.039 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_items": 0, 2024-03-19T03:42:34.039 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_bytes": 0, 2024-03-19T03:42:34.039 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_items": 0, 2024-03-19T03:42:34.039 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T03:42:34.040 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_items": 0, 2024-03-19T03:42:34.040 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T03:42:34.040 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_items": 0, 2024-03-19T03:42:34.040 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_bytes": 198287, 2024-03-19T03:42:34.040 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_items": 57, 2024-03-19T03:42:34.040 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_bytes": 0, 2024-03-19T03:42:34.040 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_items": 0, 2024-03-19T03:42:34.040 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_bytes": 0, 2024-03-19T03:42:34.040 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_items": 0, 2024-03-19T03:42:34.040 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_bytes": 0, 2024-03-19T03:42:34.040 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_items": 0, 2024-03-19T03:42:34.040 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_bytes": 0, 2024-03-19T03:42:34.040 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_items": 0, 2024-03-19T03:42:34.040 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_bytes": 9184, 2024-03-19T03:42:34.040 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_items": 177, 2024-03-19T03:42:34.041 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T03:42:34.041 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_items": 0, 2024-03-19T03:42:34.041 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_bytes": 0, 2024-03-19T03:42:34.041 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_items": 0, 2024-03-19T03:42:34.041 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_bytes": 1032600, 2024-03-19T03:42:34.041 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_items": 5775, 2024-03-19T03:42:34.041 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_bytes": 0, 2024-03-19T03:42:34.041 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_items": 0, 2024-03-19T03:42:34.041 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_bytes": 0, 2024-03-19T03:42:34.041 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_items": 0 2024-03-19T03:42:34.041 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.041 INFO:teuthology.orchestra.run.smithi119.stdout: "objecter": { 2024-03-19T03:42:34.041 INFO:teuthology.orchestra.run.smithi119.stdout: "op_active": 0, 2024-03-19T03:42:34.041 INFO:teuthology.orchestra.run.smithi119.stdout: "op_laggy": 0, 2024-03-19T03:42:34.041 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send": 2062, 2024-03-19T03:42:34.042 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send_bytes": 21, 2024-03-19T03:42:34.042 INFO:teuthology.orchestra.run.smithi119.stdout: "op_resend": 3, 2024-03-19T03:42:34.042 INFO:teuthology.orchestra.run.smithi119.stdout: "op_reply": 2062, 2024-03-19T03:42:34.042 INFO:teuthology.orchestra.run.smithi119.stdout: "op_latency": { 2024-03-19T03:42:34.042 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2062, 2024-03-19T03:42:34.042 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 8.402790715, 2024-03-19T03:42:34.042 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.004075068 2024-03-19T03:42:34.042 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.042 INFO:teuthology.orchestra.run.smithi119.stdout: "op_inflight": 0, 2024-03-19T03:42:34.042 INFO:teuthology.orchestra.run.smithi119.stdout: "oplen_avg": { 2024-03-19T03:42:34.042 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2062, 2024-03-19T03:42:34.042 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 2063 2024-03-19T03:42:34.042 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.042 INFO:teuthology.orchestra.run.smithi119.stdout: "op": 2062, 2024-03-19T03:42:34.043 INFO:teuthology.orchestra.run.smithi119.stdout: "op_r": 2062, 2024-03-19T03:42:34.043 INFO:teuthology.orchestra.run.smithi119.stdout: "op_w": 0, 2024-03-19T03:42:34.043 INFO:teuthology.orchestra.run.smithi119.stdout: "op_rmw": 0, 2024-03-19T03:42:34.043 INFO:teuthology.orchestra.run.smithi119.stdout: "op_pg": 0, 2024-03-19T03:42:34.043 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_stat": 1264, 2024-03-19T03:42:34.043 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_create": 0, 2024-03-19T03:42:34.043 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_read": 796, 2024-03-19T03:42:34.043 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_write": 0, 2024-03-19T03:42:34.043 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writefull": 0, 2024-03-19T03:42:34.043 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writesame": 0, 2024-03-19T03:42:34.043 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_append": 0, 2024-03-19T03:42:34.043 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_zero": 0, 2024-03-19T03:42:34.043 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_truncate": 0, 2024-03-19T03:42:34.043 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_delete": 0, 2024-03-19T03:42:34.043 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_mapext": 0, 2024-03-19T03:42:34.044 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_sparse_read": 0, 2024-03-19T03:42:34.044 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_clonerange": 0, 2024-03-19T03:42:34.044 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_getxattr": 1, 2024-03-19T03:42:34.044 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_setxattr": 0, 2024-03-19T03:42:34.044 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_cmpxattr": 0, 2024-03-19T03:42:34.044 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_rmxattr": 0, 2024-03-19T03:42:34.044 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_resetxattrs": 0, 2024-03-19T03:42:34.044 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_call": 0, 2024-03-19T03:42:34.044 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_watch": 0, 2024-03-19T03:42:34.044 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_notify": 0, 2024-03-19T03:42:34.044 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T03:42:34.044 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls": 0, 2024-03-19T03:42:34.044 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls_filter": 0, 2024-03-19T03:42:34.044 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_other": 0, 2024-03-19T03:42:34.045 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_active": 0, 2024-03-19T03:42:34.045 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_send": 0, 2024-03-19T03:42:34.045 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_resend": 0, 2024-03-19T03:42:34.045 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_ping": 0, 2024-03-19T03:42:34.045 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_active": 0, 2024-03-19T03:42:34.045 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_send": 0, 2024-03-19T03:42:34.045 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_resend": 0, 2024-03-19T03:42:34.045 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_active": 0, 2024-03-19T03:42:34.045 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_send": 0, 2024-03-19T03:42:34.045 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_resend": 0, 2024-03-19T03:42:34.045 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_active": 0, 2024-03-19T03:42:34.045 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_send": 0, 2024-03-19T03:42:34.045 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_resend": 0, 2024-03-19T03:42:34.045 INFO:teuthology.orchestra.run.smithi119.stdout: "command_active": 0, 2024-03-19T03:42:34.045 INFO:teuthology.orchestra.run.smithi119.stdout: "command_send": 0, 2024-03-19T03:42:34.046 INFO:teuthology.orchestra.run.smithi119.stdout: "command_resend": 0, 2024-03-19T03:42:34.046 INFO:teuthology.orchestra.run.smithi119.stdout: "map_epoch": 93, 2024-03-19T03:42:34.046 INFO:teuthology.orchestra.run.smithi119.stdout: "map_full": 0, 2024-03-19T03:42:34.046 INFO:teuthology.orchestra.run.smithi119.stdout: "map_inc": 4, 2024-03-19T03:42:34.046 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_sessions": 6, 2024-03-19T03:42:34.046 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_open": 6, 2024-03-19T03:42:34.046 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_close": 0, 2024-03-19T03:42:34.046 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_laggy": 0, 2024-03-19T03:42:34.046 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_wr": 0, 2024-03-19T03:42:34.046 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_rd": 2, 2024-03-19T03:42:34.046 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_del": 0 2024-03-19T03:42:34.046 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.046 INFO:teuthology.orchestra.run.smithi119.stdout: "oft": { 2024-03-19T03:42:34.046 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_objs": 0, 2024-03-19T03:42:34.047 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_kv_pairs": 0, 2024-03-19T03:42:34.047 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_updates": 0, 2024-03-19T03:42:34.047 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_removes": 0 2024-03-19T03:42:34.047 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.047 INFO:teuthology.orchestra.run.smithi119.stdout: "osd-slow-ops": { 2024-03-19T03:42:34.047 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_ops_count": 0 2024-03-19T03:42:34.047 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.047 INFO:teuthology.orchestra.run.smithi119.stdout: "purge_queue": { 2024-03-19T03:42:34.047 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops": 0, 2024-03-19T03:42:34.047 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops_high_water": 0, 2024-03-19T03:42:34.047 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing": 0, 2024-03-19T03:42:34.047 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_high_water": 0, 2024-03-19T03:42:34.047 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed_ops": 0, 2024-03-19T03:42:34.047 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed": 0, 2024-03-19T03:42:34.047 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_item_in_journal": 0 2024-03-19T03:42:34.048 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.048 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T03:42:34.048 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-19T03:42:34.048 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-03-19T03:42:34.048 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-19T03:42:34.048 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 2299, 2024-03-19T03:42:34.048 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 11246240, 2024-03-19T03:42:34.048 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:34.048 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 2299, 2024-03-19T03:42:34.048 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-19T03:42:34.048 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-19T03:42:34.048 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 2299, 2024-03-19T03:42:34.048 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 11246240, 2024-03-19T03:42:34.048 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-19T03:42:34.049 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.049 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.049 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.049 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:34.049 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.049 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_bytes": { 2024-03-19T03:42:34.049 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-19T03:42:34.049 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-03-19T03:42:34.049 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-19T03:42:34.049 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-19T03:42:34.049 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-19T03:42:34.049 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:34.049 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:34.049 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 2062, 2024-03-19T03:42:34.049 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 9986295, 2024-03-19T03:42:34.050 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 2062, 2024-03-19T03:42:34.050 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 9986295, 2024-03-19T03:42:34.050 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-19T03:42:34.050 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.050 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.050 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.050 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:34.050 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.050 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_ops": { 2024-03-19T03:42:34.050 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-19T03:42:34.050 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 1024, 2024-03-19T03:42:34.050 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-19T03:42:34.050 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-19T03:42:34.051 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-19T03:42:34.051 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:34.051 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:34.051 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 2062, 2024-03-19T03:42:34.051 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 2062, 2024-03-19T03:42:34.051 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 2062, 2024-03-19T03:42:34.051 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 2062, 2024-03-19T03:42:34.051 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-19T03:42:34.051 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.051 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.051 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.051 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:34.051 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.052 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle": { 2024-03-19T03:42:34.052 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-19T03:42:34.052 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-03-19T03:42:34.052 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-19T03:42:34.052 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-19T03:42:34.052 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-19T03:42:34.052 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:34.052 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:34.052 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-19T03:42:34.052 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-19T03:42:34.052 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 0, 2024-03-19T03:42:34.052 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 0, 2024-03-19T03:42:34.052 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-19T03:42:34.052 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.053 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.053 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.053 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:34.053 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:34.053 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle-0x55de959c26e0": { 2024-03-19T03:42:34.053 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-19T03:42:34.053 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-03-19T03:42:34.053 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-19T03:42:34.053 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-19T03:42:34.053 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-19T03:42:34.053 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:34.053 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:34.053 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-19T03:42:34.053 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-19T03:42:34.054 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 0, 2024-03-19T03:42:34.054 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 0, 2024-03-19T03:42:34.054 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-19T03:42:34.054 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:34.054 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:34.054 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:34.054 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:34.054 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:34.054 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-19T03:42:34.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:34 smithi119 ceph-mon[66861]: pgmap v681: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-19T03:42:35.032 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-19T03:42:35.032 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-19T03:42:35.032 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-03-19T03:42:35.032 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-03-19T03:42:35.032 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-03-19T03:42:35.032 DEBUG:tasks.check_counter:Getting stats from h 2024-03-19T03:42:35.033 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-19T03:42:35.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:34 smithi117 ceph-mon[31838]: pgmap v681: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-19T03:42:35.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:34 smithi178 ceph-mon[67578]: pgmap v681: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-19T03:42:35.346 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T03:42:36.796 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:36 smithi119 ceph-mon[66861]: pgmap v682: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-19T03:42:37.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:36 smithi117 ceph-mon[31838]: pgmap v682: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-19T03:42:37.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:36 smithi178 ceph-mon[67578]: pgmap v682: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-19T03:42:37.195 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-19T03:42:37.196 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T03:42:37.196 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 1818, 2024-03-19T03:42:37.196 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 1846, 2024-03-19T03:42:37.196 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 1594059, 2024-03-19T03:42:37.196 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 5297559, 2024-03-19T03:42:37.196 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 15, 2024-03-19T03:42:37.196 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 7, 2024-03-19T03:42:37.196 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.153437790, 2024-03-19T03:42:37.196 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.058564687, 2024-03-19T03:42:37.196 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.069355698, 2024-03-19T03:42:37.196 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.030403873, 2024-03-19T03:42:37.196 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:37.197 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1846, 2024-03-19T03:42:37.197 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.127611395, 2024-03-19T03:42:37.197 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000069128 2024-03-19T03:42:37.197 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.197 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:37.197 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1343, 2024-03-19T03:42:37.197 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000239237, 2024-03-19T03:42:37.197 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000178 2024-03-19T03:42:37.197 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.197 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 910544, 2024-03-19T03:42:37.197 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 131120 2024-03-19T03:42:37.197 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.197 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T03:42:37.198 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 18360, 2024-03-19T03:42:37.198 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 22215, 2024-03-19T03:42:37.198 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 6240319, 2024-03-19T03:42:37.198 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 12774757, 2024-03-19T03:42:37.198 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 9, 2024-03-19T03:42:37.198 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 7, 2024-03-19T03:42:37.198 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.828299252, 2024-03-19T03:42:37.198 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.399289966, 2024-03-19T03:42:37.198 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.533136672, 2024-03-19T03:42:37.198 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.025597069, 2024-03-19T03:42:37.198 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:37.199 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 22213, 2024-03-19T03:42:37.199 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 1.840164372, 2024-03-19T03:42:37.199 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000082841 2024-03-19T03:42:37.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.199 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:37.199 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 15390, 2024-03-19T03:42:37.199 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.004864716, 2024-03-19T03:42:37.199 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000316 2024-03-19T03:42:37.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.199 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-19T03:42:37.199 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 336832 2024-03-19T03:42:37.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.199 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T03:42:37.199 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 15332, 2024-03-19T03:42:37.199 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 12609, 2024-03-19T03:42:37.200 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 6476673, 2024-03-19T03:42:37.200 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 3379316, 2024-03-19T03:42:37.200 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 8, 2024-03-19T03:42:37.200 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 6, 2024-03-19T03:42:37.200 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 1.012019872, 2024-03-19T03:42:37.200 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.467780779, 2024-03-19T03:42:37.200 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.674638704, 2024-03-19T03:42:37.200 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.020620795, 2024-03-19T03:42:37.200 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:37.200 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 12607, 2024-03-19T03:42:37.200 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 1.046274543, 2024-03-19T03:42:37.200 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000082991 2024-03-19T03:42:37.200 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.200 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:37.200 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 25884, 2024-03-19T03:42:37.201 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.004697419, 2024-03-19T03:42:37.201 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000181 2024-03-19T03:42:37.201 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.201 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T03:42:37.201 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T03:42:37.201 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.201 INFO:teuthology.orchestra.run.smithi119.stdout: "cct": { 2024-03-19T03:42:37.201 INFO:teuthology.orchestra.run.smithi119.stdout: "total_workers": 1, 2024-03-19T03:42:37.201 INFO:teuthology.orchestra.run.smithi119.stdout: "unhealthy_workers": 0 2024-03-19T03:42:37.201 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.202 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-MDSRank": { 2024-03-19T03:42:37.202 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-03-19T03:42:37.202 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-03-19T03:42:37.202 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 716, 2024-03-19T03:42:37.202 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 2.716357714, 2024-03-19T03:42:37.202 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.003793795 2024-03-19T03:42:37.202 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:37.202 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.202 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-PurgeQueue": { 2024-03-19T03:42:37.202 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-03-19T03:42:37.202 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-03-19T03:42:37.202 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1882, 2024-03-19T03:42:37.202 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.209383595, 2024-03-19T03:42:37.203 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000111255 2024-03-19T03:42:37.203 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:37.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.203 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": { 2024-03-19T03:42:37.203 INFO:teuthology.orchestra.run.smithi119.stdout: "request": 7407, 2024-03-19T03:42:37.203 INFO:teuthology.orchestra.run.smithi119.stdout: "reply": 6783, 2024-03-19T03:42:37.203 INFO:teuthology.orchestra.run.smithi119.stdout: "reply_latency": { 2024-03-19T03:42:37.203 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 6783, 2024-03-19T03:42:37.203 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 66.336507479, 2024-03-19T03:42:37.203 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.009779818 2024-03-19T03:42:37.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.203 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_reply": 0, 2024-03-19T03:42:37.203 INFO:teuthology.orchestra.run.smithi119.stdout: "forward": 624, 2024-03-19T03:42:37.203 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_complete": 0, 2024-03-19T03:42:37.204 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_keys": 0, 2024-03-19T03:42:37.204 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_commit": 38, 2024-03-19T03:42:37.204 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_split": 3, 2024-03-19T03:42:37.204 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_merge": 0, 2024-03-19T03:42:37.204 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes": 24, 2024-03-19T03:42:37.204 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_top": 6, 2024-03-19T03:42:37.205 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_bottom": 8, 2024-03-19T03:42:37.205 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pin_tail": 10, 2024-03-19T03:42:37.205 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pinned": 13, 2024-03-19T03:42:37.205 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_expired": 398, 2024-03-19T03:42:37.205 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_with_caps": 0, 2024-03-19T03:42:37.205 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": 0, 2024-03-19T03:42:37.205 INFO:teuthology.orchestra.run.smithi119.stdout: "subtrees": 3, 2024-03-19T03:42:37.205 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse": 17938, 2024-03-19T03:42:37.205 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_hit": 14085, 2024-03-19T03:42:37.205 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_forward": 624, 2024-03-19T03:42:37.205 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_discover": 207, 2024-03-19T03:42:37.205 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_dir_fetch": 0, 2024-03-19T03:42:37.205 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_remote_ino": 0, 2024-03-19T03:42:37.205 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_lock": 133, 2024-03-19T03:42:37.205 INFO:teuthology.orchestra.run.smithi119.stdout: "load_cent": 0, 2024-03-19T03:42:37.206 INFO:teuthology.orchestra.run.smithi119.stdout: "q": 0, 2024-03-19T03:42:37.206 INFO:teuthology.orchestra.run.smithi119.stdout: "exported": 24, 2024-03-19T03:42:37.206 INFO:teuthology.orchestra.run.smithi119.stdout: "exported_inodes": 26, 2024-03-19T03:42:37.206 INFO:teuthology.orchestra.run.smithi119.stdout: "imported": 3, 2024-03-19T03:42:37.206 INFO:teuthology.orchestra.run.smithi119.stdout: "imported_inodes": 552, 2024-03-19T03:42:37.206 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_dir_fetch": 0, 2024-03-19T03:42:37.206 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_backtrace_fetch": 0, 2024-03-19T03:42:37.206 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_peer_discover": 0, 2024-03-19T03:42:37.206 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rfiles": 3308, 2024-03-19T03:42:37.206 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rbytes": 68731964, 2024-03-19T03:42:37.206 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rsnaps": 0, 2024-03-19T03:42:37.206 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T03:42:37.206 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_set_tag": 0, 2024-03-19T03:42:37.207 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T03:42:37.207 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_inotable_repaired": 0, 2024-03-19T03:42:37.207 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_inodes": 0, 2024-03-19T03:42:37.207 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T03:42:37.207 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T03:42:37.207 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_file_inodes": 0, 2024-03-19T03:42:37.207 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_inode_file_caps": 53, 2024-03-19T03:42:37.207 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_revoke": 919, 2024-03-19T03:42:37.207 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_grant": 1582, 2024-03-19T03:42:37.207 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T03:42:37.207 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T03:42:37.207 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T03:42:37.208 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps": 1816, 2024-03-19T03:42:37.208 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps_dirty": 899, 2024-03-19T03:42:37.208 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_cap_release": 55, 2024-03-19T03:42:37.208 INFO:teuthology.orchestra.run.smithi119.stdout: "process_request_cap_release": 3516 2024-03-19T03:42:37.208 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.208 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache": { 2024-03-19T03:42:37.208 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update": 565, 2024-03-19T03:42:37.208 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update_receipt": 4266, 2024-03-19T03:42:37.208 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_try_discover": 2749, 2024-03-19T03:42:37.209 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_send_discover": 2970, 2024-03-19T03:42:37.209 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_handle_discover": 95, 2024-03-19T03:42:37.209 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays": 0, 2024-03-19T03:42:37.209 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_delayed": 0, 2024-03-19T03:42:37.209 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_enqueuing": 0, 2024-03-19T03:42:37.209 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_created": 899, 2024-03-19T03:42:37.209 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_enqueued": 899, 2024-03-19T03:42:37.209 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_reintegrated": 0, 2024-03-19T03:42:37.209 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_migrated": 0, 2024-03-19T03:42:37.209 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_processing": 0, 2024-03-19T03:42:37.209 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_enqueued": 0, 2024-03-19T03:42:37.209 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_prioritized": 0, 2024-03-19T03:42:37.209 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_started": 0, 2024-03-19T03:42:37.209 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_completed": 0, 2024-03-19T03:42:37.210 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_path": 0, 2024-03-19T03:42:37.210 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_inode": 0, 2024-03-19T03:42:37.210 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T03:42:37.210 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_exportdir": 24, 2024-03-19T03:42:37.210 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_flush": 0, 2024-03-19T03:42:37.210 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragmentdir": 3, 2024-03-19T03:42:37.210 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragstats": 0, 2024-03-19T03:42:37.211 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_inodestats": 0 2024-03-19T03:42:37.211 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.211 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log": { 2024-03-19T03:42:37.211 INFO:teuthology.orchestra.run.smithi119.stdout: "evlrg": 0, 2024-03-19T03:42:37.211 INFO:teuthology.orchestra.run.smithi119.stdout: "evadd": 5287, 2024-03-19T03:42:37.211 INFO:teuthology.orchestra.run.smithi119.stdout: "evex": 0, 2024-03-19T03:42:37.211 INFO:teuthology.orchestra.run.smithi119.stdout: "evtrm": 0, 2024-03-19T03:42:37.211 INFO:teuthology.orchestra.run.smithi119.stdout: "ev": 5287, 2024-03-19T03:42:37.211 INFO:teuthology.orchestra.run.smithi119.stdout: "evexg": 0, 2024-03-19T03:42:37.211 INFO:teuthology.orchestra.run.smithi119.stdout: "evexd": 0, 2024-03-19T03:42:37.212 INFO:teuthology.orchestra.run.smithi119.stdout: "segadd": 8, 2024-03-19T03:42:37.212 INFO:teuthology.orchestra.run.smithi119.stdout: "segex": 0, 2024-03-19T03:42:37.212 INFO:teuthology.orchestra.run.smithi119.stdout: "segtrm": 0, 2024-03-19T03:42:37.212 INFO:teuthology.orchestra.run.smithi119.stdout: "seg": 8, 2024-03-19T03:42:37.212 INFO:teuthology.orchestra.run.smithi119.stdout: "segmjr": 2, 2024-03-19T03:42:37.212 INFO:teuthology.orchestra.run.smithi119.stdout: "segexg": 0, 2024-03-19T03:42:37.212 INFO:teuthology.orchestra.run.smithi119.stdout: "segexd": 0, 2024-03-19T03:42:37.212 INFO:teuthology.orchestra.run.smithi119.stdout: "expos": 4194304, 2024-03-19T03:42:37.212 INFO:teuthology.orchestra.run.smithi119.stdout: "wrpos": 11643678, 2024-03-19T03:42:37.212 INFO:teuthology.orchestra.run.smithi119.stdout: "rdpos": 0, 2024-03-19T03:42:37.212 INFO:teuthology.orchestra.run.smithi119.stdout: "jlat": { 2024-03-19T03:42:37.212 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 252, 2024-03-19T03:42:37.212 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 3.684681606, 2024-03-19T03:42:37.213 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.014621752 2024-03-19T03:42:37.213 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.213 INFO:teuthology.orchestra.run.smithi119.stdout: "replayed": 0 2024-03-19T03:42:37.213 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.213 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_mem": { 2024-03-19T03:42:37.213 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 28, 2024-03-19T03:42:37.213 INFO:teuthology.orchestra.run.smithi119.stdout: "ino+": 1126, 2024-03-19T03:42:37.213 INFO:teuthology.orchestra.run.smithi119.stdout: "ino-": 1098, 2024-03-19T03:42:37.213 INFO:teuthology.orchestra.run.smithi119.stdout: "dir": 27, 2024-03-19T03:42:37.213 INFO:teuthology.orchestra.run.smithi119.stdout: "dir+": 1523, 2024-03-19T03:42:37.213 INFO:teuthology.orchestra.run.smithi119.stdout: "dir-": 1496, 2024-03-19T03:42:37.213 INFO:teuthology.orchestra.run.smithi119.stdout: "dn": 24, 2024-03-19T03:42:37.213 INFO:teuthology.orchestra.run.smithi119.stdout: "dn+": 2608, 2024-03-19T03:42:37.213 INFO:teuthology.orchestra.run.smithi119.stdout: "dn-": 2584, 2024-03-19T03:42:37.213 INFO:teuthology.orchestra.run.smithi119.stdout: "cap": 0, 2024-03-19T03:42:37.214 INFO:teuthology.orchestra.run.smithi119.stdout: "cap+": 1529, 2024-03-19T03:42:37.214 INFO:teuthology.orchestra.run.smithi119.stdout: "cap-": 1529, 2024-03-19T03:42:37.214 INFO:teuthology.orchestra.run.smithi119.stdout: "rss": 64640, 2024-03-19T03:42:37.214 INFO:teuthology.orchestra.run.smithi119.stdout: "heap": 223612 2024-03-19T03:42:37.214 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.214 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_server": { 2024-03-19T03:42:37.214 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_client_request": 9809, 2024-03-19T03:42:37.214 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_server_request": 46, 2024-03-19T03:42:37.214 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_request": 7407, 2024-03-19T03:42:37.214 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_session": 12, 2024-03-19T03:42:37.214 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_peer_request": 2072, 2024-03-19T03:42:37.214 INFO:teuthology.orchestra.run.smithi119.stdout: "req_create_latency": { 2024-03-19T03:42:37.214 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 657, 2024-03-19T03:42:37.215 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 7.744960777, 2024-03-19T03:42:37.215 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.011788372 2024-03-19T03:42:37.215 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.215 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getattr_latency": { 2024-03-19T03:42:37.215 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.215 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.215 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.215 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.215 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getfilelock_latency": { 2024-03-19T03:42:37.215 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.215 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.215 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.215 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.215 INFO:teuthology.orchestra.run.smithi119.stdout: "req_link_latency": { 2024-03-19T03:42:37.215 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.215 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.216 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.216 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.216 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookup_latency": { 2024-03-19T03:42:37.216 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 4194, 2024-03-19T03:42:37.216 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 47.422944236, 2024-03-19T03:42:37.216 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.011307330 2024-03-19T03:42:37.216 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.216 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookuphash_latency": { 2024-03-19T03:42:37.216 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.216 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.216 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.216 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.216 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupino_latency": { 2024-03-19T03:42:37.216 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.217 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.217 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.217 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.217 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupname_latency": { 2024-03-19T03:42:37.217 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.217 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.217 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.217 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.217 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupparent_latency": { 2024-03-19T03:42:37.217 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.217 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.217 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.217 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.217 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupsnap_latency": { 2024-03-19T03:42:37.218 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.218 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.218 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.218 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.218 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lssnap_latency": { 2024-03-19T03:42:37.218 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.219 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.219 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.219 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.219 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mkdir_latency": { 2024-03-19T03:42:37.219 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.219 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.219 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.219 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.219 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mknod_latency": { 2024-03-19T03:42:37.220 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.220 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.220 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.220 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.220 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mksnap_latency": { 2024-03-19T03:42:37.220 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.220 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.220 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.220 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.220 INFO:teuthology.orchestra.run.smithi119.stdout: "req_open_latency": { 2024-03-19T03:42:37.220 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 623, 2024-03-19T03:42:37.220 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.837866813, 2024-03-19T03:42:37.220 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.001344890 2024-03-19T03:42:37.220 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.220 INFO:teuthology.orchestra.run.smithi119.stdout: "req_readdir_latency": { 2024-03-19T03:42:37.221 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 32, 2024-03-19T03:42:37.221 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.092093768, 2024-03-19T03:42:37.221 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.002877930 2024-03-19T03:42:37.221 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.221 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rename_latency": { 2024-03-19T03:42:37.221 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 657, 2024-03-19T03:42:37.221 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 8.775064049, 2024-03-19T03:42:37.221 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.013356261 2024-03-19T03:42:37.221 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.221 INFO:teuthology.orchestra.run.smithi119.stdout: "req_renamesnap_latency": { 2024-03-19T03:42:37.221 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.221 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.221 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.221 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.222 INFO:teuthology.orchestra.run.smithi119.stdout: "req_snapdiff_latency": { 2024-03-19T03:42:37.222 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.222 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.222 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.222 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.222 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmdir_latency": { 2024-03-19T03:42:37.222 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.222 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.222 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.222 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.222 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmsnap_latency": { 2024-03-19T03:42:37.222 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.223 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.223 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.223 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.223 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmxattr_latency": { 2024-03-19T03:42:37.223 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.223 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.223 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.223 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.223 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setattr_latency": { 2024-03-19T03:42:37.223 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.223 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.223 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.224 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.224 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setdirlayout_latency": { 2024-03-19T03:42:37.224 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.224 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.224 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.224 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.224 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setfilelock_latency": { 2024-03-19T03:42:37.224 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.224 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.224 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.224 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.224 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setlayout_latency": { 2024-03-19T03:42:37.224 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.224 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.225 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.225 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.225 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setxattr_latency": { 2024-03-19T03:42:37.225 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.225 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.225 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.225 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.225 INFO:teuthology.orchestra.run.smithi119.stdout: "req_symlink_latency": { 2024-03-19T03:42:37.225 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.225 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.225 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.225 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.225 INFO:teuthology.orchestra.run.smithi119.stdout: "req_unlink_latency": { 2024-03-19T03:42:37.225 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 620, 2024-03-19T03:42:37.225 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 1.463577836, 2024-03-19T03:42:37.226 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.002360609 2024-03-19T03:42:37.226 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.226 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_revoke_eviction": 0, 2024-03-19T03:42:37.226 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_acquisition_throttle": 0, 2024-03-19T03:42:37.226 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getvxattr_latency": { 2024-03-19T03:42:37.226 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.226 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.226 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.226 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:37.226 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.226 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_sessions": { 2024-03-19T03:42:37.226 INFO:teuthology.orchestra.run.smithi119.stdout: "session_count": 1, 2024-03-19T03:42:37.226 INFO:teuthology.orchestra.run.smithi119.stdout: "session_add": 2, 2024-03-19T03:42:37.227 INFO:teuthology.orchestra.run.smithi119.stdout: "session_remove": 1, 2024-03-19T03:42:37.227 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_open": 1, 2024-03-19T03:42:37.227 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_stale": 0, 2024-03-19T03:42:37.227 INFO:teuthology.orchestra.run.smithi119.stdout: "total_load": 1217, 2024-03-19T03:42:37.227 INFO:teuthology.orchestra.run.smithi119.stdout: "average_load": 1217, 2024-03-19T03:42:37.227 INFO:teuthology.orchestra.run.smithi119.stdout: "avg_session_uptime": 476, 2024-03-19T03:42:37.227 INFO:teuthology.orchestra.run.smithi119.stdout: "mdthresh_evicted": 0 2024-03-19T03:42:37.227 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.227 INFO:teuthology.orchestra.run.smithi119.stdout: "mempool": { 2024-03-19T03:42:37.227 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_bytes": 0, 2024-03-19T03:42:37.227 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_items": 0, 2024-03-19T03:42:37.227 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T03:42:37.227 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_items": 0, 2024-03-19T03:42:37.227 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T03:42:37.227 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_items": 0, 2024-03-19T03:42:37.228 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T03:42:37.228 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T03:42:37.228 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T03:42:37.228 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T03:42:37.228 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T03:42:37.228 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_items": 0, 2024-03-19T03:42:37.228 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T03:42:37.228 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T03:42:37.228 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_bytes": 0, 2024-03-19T03:42:37.228 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_items": 0, 2024-03-19T03:42:37.228 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_bytes": 0, 2024-03-19T03:42:37.228 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_items": 0, 2024-03-19T03:42:37.228 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T03:42:37.229 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T03:42:37.229 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T03:42:37.229 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T03:42:37.229 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T03:42:37.229 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_items": 0, 2024-03-19T03:42:37.229 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_bytes": 0, 2024-03-19T03:42:37.229 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_items": 0, 2024-03-19T03:42:37.229 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T03:42:37.229 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T03:42:37.229 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_bytes": 0, 2024-03-19T03:42:37.229 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_items": 0, 2024-03-19T03:42:37.230 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_bytes": 0, 2024-03-19T03:42:37.230 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_items": 0, 2024-03-19T03:42:37.230 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T03:42:37.230 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_items": 0, 2024-03-19T03:42:37.230 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T03:42:37.230 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_items": 0, 2024-03-19T03:42:37.230 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_bytes": 310787, 2024-03-19T03:42:37.230 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_items": 142, 2024-03-19T03:42:37.230 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_bytes": 880, 2024-03-19T03:42:37.230 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_items": 10, 2024-03-19T03:42:37.230 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_bytes": 0, 2024-03-19T03:42:37.230 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_items": 0, 2024-03-19T03:42:37.230 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_bytes": 0, 2024-03-19T03:42:37.231 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_items": 0, 2024-03-19T03:42:37.231 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_bytes": 0, 2024-03-19T03:42:37.231 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_items": 0, 2024-03-19T03:42:37.231 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_bytes": 9184, 2024-03-19T03:42:37.231 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_items": 177, 2024-03-19T03:42:37.231 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T03:42:37.231 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_items": 0, 2024-03-19T03:42:37.231 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_bytes": 0, 2024-03-19T03:42:37.231 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_items": 0, 2024-03-19T03:42:37.231 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_bytes": 114861, 2024-03-19T03:42:37.231 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_items": 476, 2024-03-19T03:42:37.231 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_bytes": 0, 2024-03-19T03:42:37.231 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_items": 0, 2024-03-19T03:42:37.231 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_bytes": 0, 2024-03-19T03:42:37.232 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_items": 0 2024-03-19T03:42:37.232 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.232 INFO:teuthology.orchestra.run.smithi119.stdout: "objecter": { 2024-03-19T03:42:37.232 INFO:teuthology.orchestra.run.smithi119.stdout: "op_active": 0, 2024-03-19T03:42:37.232 INFO:teuthology.orchestra.run.smithi119.stdout: "op_laggy": 0, 2024-03-19T03:42:37.232 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send": 2484, 2024-03-19T03:42:37.232 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send_bytes": 8214994, 2024-03-19T03:42:37.232 INFO:teuthology.orchestra.run.smithi119.stdout: "op_resend": 33, 2024-03-19T03:42:37.232 INFO:teuthology.orchestra.run.smithi119.stdout: "op_reply": 2484, 2024-03-19T03:42:37.232 INFO:teuthology.orchestra.run.smithi119.stdout: "op_latency": { 2024-03-19T03:42:37.232 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2484, 2024-03-19T03:42:37.232 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 35.622116909, 2024-03-19T03:42:37.232 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.014340626 2024-03-19T03:42:37.233 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.233 INFO:teuthology.orchestra.run.smithi119.stdout: "op_inflight": 0, 2024-03-19T03:42:37.233 INFO:teuthology.orchestra.run.smithi119.stdout: "oplen_avg": { 2024-03-19T03:42:37.233 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2484, 2024-03-19T03:42:37.233 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 2721 2024-03-19T03:42:37.233 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.233 INFO:teuthology.orchestra.run.smithi119.stdout: "op": 2484, 2024-03-19T03:42:37.233 INFO:teuthology.orchestra.run.smithi119.stdout: "op_r": 72, 2024-03-19T03:42:37.233 INFO:teuthology.orchestra.run.smithi119.stdout: "op_w": 2412, 2024-03-19T03:42:37.233 INFO:teuthology.orchestra.run.smithi119.stdout: "op_rmw": 0, 2024-03-19T03:42:37.233 INFO:teuthology.orchestra.run.smithi119.stdout: "op_pg": 0, 2024-03-19T03:42:37.233 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_stat": 0, 2024-03-19T03:42:37.233 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_create": 10, 2024-03-19T03:42:37.233 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_read": 72, 2024-03-19T03:42:37.233 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_write": 324, 2024-03-19T03:42:37.233 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writefull": 69, 2024-03-19T03:42:37.234 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writesame": 0, 2024-03-19T03:42:37.234 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_append": 0, 2024-03-19T03:42:37.234 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_zero": 0, 2024-03-19T03:42:37.234 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_truncate": 3, 2024-03-19T03:42:37.234 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_delete": 1811, 2024-03-19T03:42:37.234 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_mapext": 0, 2024-03-19T03:42:37.234 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_sparse_read": 0, 2024-03-19T03:42:37.234 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_clonerange": 0, 2024-03-19T03:42:37.234 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_getxattr": 0, 2024-03-19T03:42:37.234 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_setxattr": 20, 2024-03-19T03:42:37.234 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_cmpxattr": 0, 2024-03-19T03:42:37.234 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_rmxattr": 0, 2024-03-19T03:42:37.234 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_resetxattrs": 0, 2024-03-19T03:42:37.234 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_call": 0, 2024-03-19T03:42:37.235 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_watch": 0, 2024-03-19T03:42:37.235 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_notify": 0, 2024-03-19T03:42:37.235 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T03:42:37.235 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls": 0, 2024-03-19T03:42:37.235 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls_filter": 0, 2024-03-19T03:42:37.235 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_other": 0, 2024-03-19T03:42:37.235 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_active": 0, 2024-03-19T03:42:37.235 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_send": 0, 2024-03-19T03:42:37.235 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_resend": 0, 2024-03-19T03:42:37.235 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_ping": 0, 2024-03-19T03:42:37.235 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_active": 0, 2024-03-19T03:42:37.235 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_send": 0, 2024-03-19T03:42:37.236 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_resend": 0, 2024-03-19T03:42:37.236 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_active": 0, 2024-03-19T03:42:37.236 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_send": 0, 2024-03-19T03:42:37.236 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_resend": 0, 2024-03-19T03:42:37.236 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_active": 0, 2024-03-19T03:42:37.236 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_send": 0, 2024-03-19T03:42:37.236 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_resend": 0, 2024-03-19T03:42:37.236 INFO:teuthology.orchestra.run.smithi119.stdout: "command_active": 0, 2024-03-19T03:42:37.236 INFO:teuthology.orchestra.run.smithi119.stdout: "command_send": 0, 2024-03-19T03:42:37.236 INFO:teuthology.orchestra.run.smithi119.stdout: "command_resend": 0, 2024-03-19T03:42:37.236 INFO:teuthology.orchestra.run.smithi119.stdout: "map_epoch": 93, 2024-03-19T03:42:37.236 INFO:teuthology.orchestra.run.smithi119.stdout: "map_full": 0, 2024-03-19T03:42:37.236 INFO:teuthology.orchestra.run.smithi119.stdout: "map_inc": 4, 2024-03-19T03:42:37.237 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_sessions": 12, 2024-03-19T03:42:37.237 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_open": 12, 2024-03-19T03:42:37.237 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_close": 0, 2024-03-19T03:42:37.237 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_laggy": 0, 2024-03-19T03:42:37.237 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_wr": 334, 2024-03-19T03:42:37.237 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_rd": 0, 2024-03-19T03:42:37.237 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_del": 78 2024-03-19T03:42:37.237 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.237 INFO:teuthology.orchestra.run.smithi119.stdout: "oft": { 2024-03-19T03:42:37.237 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_objs": 1, 2024-03-19T03:42:37.237 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_kv_pairs": 0, 2024-03-19T03:42:37.237 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_updates": 0, 2024-03-19T03:42:37.237 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_removes": 0 2024-03-19T03:42:37.237 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.237 INFO:teuthology.orchestra.run.smithi119.stdout: "osd-slow-ops": { 2024-03-19T03:42:37.238 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_ops_count": 0 2024-03-19T03:42:37.238 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.238 INFO:teuthology.orchestra.run.smithi119.stdout: "purge_queue": { 2024-03-19T03:42:37.238 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops": 0, 2024-03-19T03:42:37.238 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops_high_water": 10, 2024-03-19T03:42:37.238 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing": 0, 2024-03-19T03:42:37.238 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_high_water": 5, 2024-03-19T03:42:37.238 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed_ops": 1798, 2024-03-19T03:42:37.238 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed": 899, 2024-03-19T03:42:37.238 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_item_in_journal": 0 2024-03-19T03:42:37.238 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.238 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T03:42:37.238 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-19T03:42:37.238 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-03-19T03:42:37.239 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-19T03:42:37.239 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 35506, 2024-03-19T03:42:37.239 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 11378141, 2024-03-19T03:42:37.239 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:37.239 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 35506, 2024-03-19T03:42:37.239 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-19T03:42:37.239 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-19T03:42:37.239 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 35506, 2024-03-19T03:42:37.239 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 11378141, 2024-03-19T03:42:37.239 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-19T03:42:37.239 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.239 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.239 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.239 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:37.240 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.240 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_bytes": { 2024-03-19T03:42:37.240 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-19T03:42:37.240 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-03-19T03:42:37.240 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-19T03:42:37.240 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-19T03:42:37.240 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-19T03:42:37.240 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:37.240 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:37.240 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 2484, 2024-03-19T03:42:37.240 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 8312985, 2024-03-19T03:42:37.240 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 2484, 2024-03-19T03:42:37.240 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 8312985, 2024-03-19T03:42:37.240 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-19T03:42:37.240 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.241 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.241 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.241 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:37.241 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.241 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_ops": { 2024-03-19T03:42:37.241 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-19T03:42:37.241 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 1024, 2024-03-19T03:42:37.241 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-19T03:42:37.241 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-19T03:42:37.241 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-19T03:42:37.241 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:37.242 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:37.242 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 2484, 2024-03-19T03:42:37.242 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 2484, 2024-03-19T03:42:37.242 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 2484, 2024-03-19T03:42:37.242 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 2484, 2024-03-19T03:42:37.242 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-19T03:42:37.242 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.242 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.242 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.242 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:37.242 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.242 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle": { 2024-03-19T03:42:37.242 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-19T03:42:37.243 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-03-19T03:42:37.243 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-19T03:42:37.243 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 899, 2024-03-19T03:42:37.243 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 97991, 2024-03-19T03:42:37.243 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:37.243 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 899, 2024-03-19T03:42:37.243 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-19T03:42:37.243 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-19T03:42:37.243 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 72, 2024-03-19T03:42:37.243 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 97991, 2024-03-19T03:42:37.243 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-19T03:42:37.243 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.243 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.243 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.243 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:37.244 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:37.244 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle-0x5599dc48a500": { 2024-03-19T03:42:37.244 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-19T03:42:37.244 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-03-19T03:42:37.244 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-19T03:42:37.244 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 5287, 2024-03-19T03:42:37.244 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 7449374, 2024-03-19T03:42:37.244 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:37.244 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 5287, 2024-03-19T03:42:37.244 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-19T03:42:37.244 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-19T03:42:37.244 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 252, 2024-03-19T03:42:37.244 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 7449374, 2024-03-19T03:42:37.244 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-19T03:42:37.245 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:37.245 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:37.245 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:37.245 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:37.245 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:37.245 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-19T03:42:37.968 INFO:tasks.check_counter:Daemon mds.h mds.exported=24 2024-03-19T03:42:37.968 INFO:tasks.check_counter:Daemon mds.h mds.imported=3 2024-03-19T03:42:37.968 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=565 2024-03-19T03:42:37.968 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=4266 2024-03-19T03:42:37.968 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=3 2024-03-19T03:42:37.969 DEBUG:tasks.check_counter:Getting stats from k 2024-03-19T03:42:37.969 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-19T03:42:38.281 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T03:42:38.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:38 smithi119 ceph-mon[66861]: pgmap v683: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 852 B/s wr, 11 op/s 2024-03-19T03:42:39.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:38 smithi117 ceph-mon[31838]: pgmap v683: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 852 B/s wr, 11 op/s 2024-03-19T03:42:39.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:38 smithi178 ceph-mon[67578]: pgmap v683: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 852 B/s wr, 11 op/s 2024-03-19T03:42:40.215 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-03-19T03:42:40.215 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T03:42:40.215 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 17678, 2024-03-19T03:42:40.215 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 15559, 2024-03-19T03:42:40.216 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 7165493, 2024-03-19T03:42:40.216 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 8673995, 2024-03-19T03:42:40.216 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 10, 2024-03-19T03:42:40.216 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 5, 2024-03-19T03:42:40.216 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 1.239169777, 2024-03-19T03:42:40.216 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.577070813, 2024-03-19T03:42:40.216 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.772084674, 2024-03-19T03:42:40.216 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.041926742, 2024-03-19T03:42:40.216 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:40.216 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 15557, 2024-03-19T03:42:40.216 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.570114485, 2024-03-19T03:42:40.216 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000036646 2024-03-19T03:42:40.216 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.216 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:40.217 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 29885, 2024-03-19T03:42:40.217 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.005367528, 2024-03-19T03:42:40.217 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000179 2024-03-19T03:42:40.217 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.217 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T03:42:40.217 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T03:42:40.217 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.217 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T03:42:40.217 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 18398, 2024-03-19T03:42:40.217 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 23263, 2024-03-19T03:42:40.217 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 7022872, 2024-03-19T03:42:40.217 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 11228310, 2024-03-19T03:42:40.217 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 13, 2024-03-19T03:42:40.217 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 8, 2024-03-19T03:42:40.218 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.942652421, 2024-03-19T03:42:40.218 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.430726443, 2024-03-19T03:42:40.218 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.696095887, 2024-03-19T03:42:40.218 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.031003898, 2024-03-19T03:42:40.218 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:40.218 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 23263, 2024-03-19T03:42:40.218 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 1.393043922, 2024-03-19T03:42:40.218 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000059882 2024-03-19T03:42:40.218 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.218 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:40.218 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 17842, 2024-03-19T03:42:40.218 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.005892240, 2024-03-19T03:42:40.218 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000330 2024-03-19T03:42:40.218 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.219 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 902208, 2024-03-19T03:42:40.219 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 133760 2024-03-19T03:42:40.219 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.219 INFO:teuthology.orchestra.run.smithi119.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T03:42:40.219 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_messages": 1460, 2024-03-19T03:42:40.219 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages": 1639, 2024-03-19T03:42:40.219 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_bytes": 534788, 2024-03-19T03:42:40.219 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_bytes": 6340602, 2024-03-19T03:42:40.219 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_created_connections": 10, 2024-03-19T03:42:40.219 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_active_connections": 7, 2024-03-19T03:42:40.219 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_total_time": 0.117042945, 2024-03-19T03:42:40.219 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_send_time": 0.047085095, 2024-03-19T03:42:40.219 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_recv_time": 0.044797459, 2024-03-19T03:42:40.219 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_running_fast_dispatch_time": 0.025142516, 2024-03-19T03:42:40.220 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:40.220 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1636, 2024-03-19T03:42:40.220 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.119871163, 2024-03-19T03:42:40.220 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000073270 2024-03-19T03:42:40.220 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.220 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:40.220 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 1259, 2024-03-19T03:42:40.220 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000219229, 2024-03-19T03:42:40.220 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000174 2024-03-19T03:42:40.220 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.220 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_recv_encrypted_bytes": 1248, 2024-03-19T03:42:40.220 INFO:teuthology.orchestra.run.smithi119.stdout: "msgr_send_encrypted_bytes": 416592 2024-03-19T03:42:40.220 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.220 INFO:teuthology.orchestra.run.smithi119.stdout: "cct": { 2024-03-19T03:42:40.221 INFO:teuthology.orchestra.run.smithi119.stdout: "total_workers": 1, 2024-03-19T03:42:40.221 INFO:teuthology.orchestra.run.smithi119.stdout: "unhealthy_workers": 0 2024-03-19T03:42:40.221 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.221 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-MDSRank": { 2024-03-19T03:42:40.221 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-03-19T03:42:40.221 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-03-19T03:42:40.221 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 765, 2024-03-19T03:42:40.221 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 3.787036029, 2024-03-19T03:42:40.221 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.004950373 2024-03-19T03:42:40.221 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:40.221 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.221 INFO:teuthology.orchestra.run.smithi119.stdout: "finisher-PurgeQueue": { 2024-03-19T03:42:40.221 INFO:teuthology.orchestra.run.smithi119.stdout: "queue_len": 0, 2024-03-19T03:42:40.221 INFO:teuthology.orchestra.run.smithi119.stdout: "complete_latency": { 2024-03-19T03:42:40.222 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2442, 2024-03-19T03:42:40.222 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.286651241, 2024-03-19T03:42:40.222 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000117383 2024-03-19T03:42:40.222 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:40.222 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.222 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": { 2024-03-19T03:42:40.222 INFO:teuthology.orchestra.run.smithi119.stdout: "request": 8387, 2024-03-19T03:42:40.222 INFO:teuthology.orchestra.run.smithi119.stdout: "reply": 8307, 2024-03-19T03:42:40.222 INFO:teuthology.orchestra.run.smithi119.stdout: "reply_latency": { 2024-03-19T03:42:40.222 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 8307, 2024-03-19T03:42:40.222 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 254.472335946, 2024-03-19T03:42:40.222 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.030633482 2024-03-19T03:42:40.222 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.222 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_reply": 0, 2024-03-19T03:42:40.223 INFO:teuthology.orchestra.run.smithi119.stdout: "forward": 80, 2024-03-19T03:42:40.223 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_complete": 0, 2024-03-19T03:42:40.223 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_fetch_keys": 0, 2024-03-19T03:42:40.223 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_commit": 47, 2024-03-19T03:42:40.223 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_split": 4, 2024-03-19T03:42:40.223 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_merge": 0, 2024-03-19T03:42:40.223 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes": 20, 2024-03-19T03:42:40.223 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_top": 3, 2024-03-19T03:42:40.223 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_bottom": 7, 2024-03-19T03:42:40.223 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pin_tail": 10, 2024-03-19T03:42:40.223 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_pinned": 13, 2024-03-19T03:42:40.223 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_expired": 859, 2024-03-19T03:42:40.223 INFO:teuthology.orchestra.run.smithi119.stdout: "inodes_with_caps": 2, 2024-03-19T03:42:40.223 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": 2, 2024-03-19T03:42:40.224 INFO:teuthology.orchestra.run.smithi119.stdout: "subtrees": 3, 2024-03-19T03:42:40.224 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse": 21494, 2024-03-19T03:42:40.224 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_hit": 16099, 2024-03-19T03:42:40.224 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_forward": 80, 2024-03-19T03:42:40.224 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_discover": 237, 2024-03-19T03:42:40.224 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_dir_fetch": 0, 2024-03-19T03:42:40.224 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_remote_ino": 0, 2024-03-19T03:42:40.224 INFO:teuthology.orchestra.run.smithi119.stdout: "traverse_lock": 152, 2024-03-19T03:42:40.224 INFO:teuthology.orchestra.run.smithi119.stdout: "load_cent": 0, 2024-03-19T03:42:40.224 INFO:teuthology.orchestra.run.smithi119.stdout: "q": 0, 2024-03-19T03:42:40.224 INFO:teuthology.orchestra.run.smithi119.stdout: "exported": 33, 2024-03-19T03:42:40.224 INFO:teuthology.orchestra.run.smithi119.stdout: "exported_inodes": 320, 2024-03-19T03:42:40.224 INFO:teuthology.orchestra.run.smithi119.stdout: "imported": 5, 2024-03-19T03:42:40.224 INFO:teuthology.orchestra.run.smithi119.stdout: "imported_inodes": 738, 2024-03-19T03:42:40.225 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_dir_fetch": 0, 2024-03-19T03:42:40.225 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_backtrace_fetch": 0, 2024-03-19T03:42:40.225 INFO:teuthology.orchestra.run.smithi119.stdout: "openino_peer_discover": 0, 2024-03-19T03:42:40.225 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rfiles": 3308, 2024-03-19T03:42:40.225 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rbytes": 68731964, 2024-03-19T03:42:40.225 INFO:teuthology.orchestra.run.smithi119.stdout: "root_rsnaps": 0, 2024-03-19T03:42:40.225 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T03:42:40.225 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_set_tag": 0, 2024-03-19T03:42:40.225 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T03:42:40.225 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_inotable_repaired": 0, 2024-03-19T03:42:40.225 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_inodes": 0, 2024-03-19T03:42:40.225 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T03:42:40.225 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T03:42:40.225 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_file_inodes": 0, 2024-03-19T03:42:40.226 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_inode_file_caps": 68, 2024-03-19T03:42:40.226 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_revoke": 1080, 2024-03-19T03:42:40.226 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_grant": 2482, 2024-03-19T03:42:40.226 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T03:42:40.226 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T03:42:40.226 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T03:42:40.226 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps": 2551, 2024-03-19T03:42:40.226 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_caps_dirty": 1238, 2024-03-19T03:42:40.226 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_cap_release": 214, 2024-03-19T03:42:40.226 INFO:teuthology.orchestra.run.smithi119.stdout: "process_request_cap_release": 4679 2024-03-19T03:42:40.226 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.226 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache": { 2024-03-19T03:42:40.226 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update": 484, 2024-03-19T03:42:40.226 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_update_receipt": 4874, 2024-03-19T03:42:40.227 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_try_discover": 3210, 2024-03-19T03:42:40.227 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_send_discover": 3381, 2024-03-19T03:42:40.227 INFO:teuthology.orchestra.run.smithi119.stdout: "dir_handle_discover": 218, 2024-03-19T03:42:40.227 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays": 0, 2024-03-19T03:42:40.227 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_delayed": 0, 2024-03-19T03:42:40.227 INFO:teuthology.orchestra.run.smithi119.stdout: "num_strays_enqueuing": 0, 2024-03-19T03:42:40.227 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_created": 1197, 2024-03-19T03:42:40.227 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_enqueued": 1197, 2024-03-19T03:42:40.227 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_reintegrated": 0, 2024-03-19T03:42:40.227 INFO:teuthology.orchestra.run.smithi119.stdout: "strays_migrated": 0, 2024-03-19T03:42:40.227 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_processing": 0, 2024-03-19T03:42:40.227 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_enqueued": 0, 2024-03-19T03:42:40.227 INFO:teuthology.orchestra.run.smithi119.stdout: "num_recovering_prioritized": 0, 2024-03-19T03:42:40.227 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_started": 0, 2024-03-19T03:42:40.228 INFO:teuthology.orchestra.run.smithi119.stdout: "recovery_completed": 0, 2024-03-19T03:42:40.228 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_path": 0, 2024-03-19T03:42:40.228 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_quiesce_inode": 0, 2024-03-19T03:42:40.228 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T03:42:40.228 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_exportdir": 33, 2024-03-19T03:42:40.228 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_flush": 0, 2024-03-19T03:42:40.228 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragmentdir": 4, 2024-03-19T03:42:40.228 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_fragstats": 0, 2024-03-19T03:42:40.228 INFO:teuthology.orchestra.run.smithi119.stdout: "ireq_inodestats": 0 2024-03-19T03:42:40.228 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.228 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log": { 2024-03-19T03:42:40.228 INFO:teuthology.orchestra.run.smithi119.stdout: "evlrg": 0, 2024-03-19T03:42:40.228 INFO:teuthology.orchestra.run.smithi119.stdout: "evadd": 6494, 2024-03-19T03:42:40.228 INFO:teuthology.orchestra.run.smithi119.stdout: "evex": 0, 2024-03-19T03:42:40.228 INFO:teuthology.orchestra.run.smithi119.stdout: "evtrm": 0, 2024-03-19T03:42:40.229 INFO:teuthology.orchestra.run.smithi119.stdout: "ev": 6494, 2024-03-19T03:42:40.229 INFO:teuthology.orchestra.run.smithi119.stdout: "evexg": 0, 2024-03-19T03:42:40.229 INFO:teuthology.orchestra.run.smithi119.stdout: "evexd": 0, 2024-03-19T03:42:40.229 INFO:teuthology.orchestra.run.smithi119.stdout: "segadd": 8, 2024-03-19T03:42:40.229 INFO:teuthology.orchestra.run.smithi119.stdout: "segex": 0, 2024-03-19T03:42:40.229 INFO:teuthology.orchestra.run.smithi119.stdout: "segtrm": 0, 2024-03-19T03:42:40.229 INFO:teuthology.orchestra.run.smithi119.stdout: "seg": 8, 2024-03-19T03:42:40.229 INFO:teuthology.orchestra.run.smithi119.stdout: "segmjr": 2, 2024-03-19T03:42:40.229 INFO:teuthology.orchestra.run.smithi119.stdout: "segexg": 0, 2024-03-19T03:42:40.229 INFO:teuthology.orchestra.run.smithi119.stdout: "segexd": 0, 2024-03-19T03:42:40.229 INFO:teuthology.orchestra.run.smithi119.stdout: "expos": 4194304, 2024-03-19T03:42:40.229 INFO:teuthology.orchestra.run.smithi119.stdout: "wrpos": 13833607, 2024-03-19T03:42:40.229 INFO:teuthology.orchestra.run.smithi119.stdout: "rdpos": 0, 2024-03-19T03:42:40.229 INFO:teuthology.orchestra.run.smithi119.stdout: "jlat": { 2024-03-19T03:42:40.230 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 268, 2024-03-19T03:42:40.230 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 13.835345761, 2024-03-19T03:42:40.230 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.051624424 2024-03-19T03:42:40.230 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.230 INFO:teuthology.orchestra.run.smithi119.stdout: "replayed": 0 2024-03-19T03:42:40.230 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.230 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_mem": { 2024-03-19T03:42:40.230 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 24, 2024-03-19T03:42:40.230 INFO:teuthology.orchestra.run.smithi119.stdout: "ino+": 1475, 2024-03-19T03:42:40.230 INFO:teuthology.orchestra.run.smithi119.stdout: "ino-": 1451, 2024-03-19T03:42:40.230 INFO:teuthology.orchestra.run.smithi119.stdout: "dir": 23, 2024-03-19T03:42:40.230 INFO:teuthology.orchestra.run.smithi119.stdout: "dir+": 1624, 2024-03-19T03:42:40.230 INFO:teuthology.orchestra.run.smithi119.stdout: "dir-": 1601, 2024-03-19T03:42:40.231 INFO:teuthology.orchestra.run.smithi119.stdout: "dn": 20, 2024-03-19T03:42:40.231 INFO:teuthology.orchestra.run.smithi119.stdout: "dn+": 3329, 2024-03-19T03:42:40.231 INFO:teuthology.orchestra.run.smithi119.stdout: "dn-": 3309, 2024-03-19T03:42:40.231 INFO:teuthology.orchestra.run.smithi119.stdout: "cap": 2, 2024-03-19T03:42:40.231 INFO:teuthology.orchestra.run.smithi119.stdout: "cap+": 1785, 2024-03-19T03:42:40.231 INFO:teuthology.orchestra.run.smithi119.stdout: "cap-": 1783, 2024-03-19T03:42:40.231 INFO:teuthology.orchestra.run.smithi119.stdout: "rss": 66276, 2024-03-19T03:42:40.231 INFO:teuthology.orchestra.run.smithi119.stdout: "heap": 223612 2024-03-19T03:42:40.231 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.231 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_server": { 2024-03-19T03:42:40.231 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_client_request": 11596, 2024-03-19T03:42:40.231 INFO:teuthology.orchestra.run.smithi119.stdout: "dispatch_server_request": 199, 2024-03-19T03:42:40.231 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_request": 8387, 2024-03-19T03:42:40.231 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_client_session": 12, 2024-03-19T03:42:40.231 INFO:teuthology.orchestra.run.smithi119.stdout: "handle_peer_request": 3032, 2024-03-19T03:42:40.231 INFO:teuthology.orchestra.run.smithi119.stdout: "req_create_latency": { 2024-03-19T03:42:40.232 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 876, 2024-03-19T03:42:40.232 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 10.981503112, 2024-03-19T03:42:40.232 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.012535962 2024-03-19T03:42:40.232 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.232 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getattr_latency": { 2024-03-19T03:42:40.232 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.232 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.232 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.232 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.232 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getfilelock_latency": { 2024-03-19T03:42:40.232 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.232 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.232 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.232 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.232 INFO:teuthology.orchestra.run.smithi119.stdout: "req_link_latency": { 2024-03-19T03:42:40.233 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.233 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.233 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.233 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.233 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookup_latency": { 2024-03-19T03:42:40.233 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 4930, 2024-03-19T03:42:40.233 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 224.763862144, 2024-03-19T03:42:40.233 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.045591047 2024-03-19T03:42:40.233 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.233 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookuphash_latency": { 2024-03-19T03:42:40.233 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.233 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.233 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.233 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.233 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupino_latency": { 2024-03-19T03:42:40.234 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 2, 2024-03-19T03:42:40.234 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000210512, 2024-03-19T03:42:40.234 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000105256 2024-03-19T03:42:40.234 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.234 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupname_latency": { 2024-03-19T03:42:40.234 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.234 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.234 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.234 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.234 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupparent_latency": { 2024-03-19T03:42:40.234 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.234 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.234 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.234 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.235 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lookupsnap_latency": { 2024-03-19T03:42:40.235 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.235 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.235 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.235 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.235 INFO:teuthology.orchestra.run.smithi119.stdout: "req_lssnap_latency": { 2024-03-19T03:42:40.235 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.235 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.235 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.235 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.235 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mkdir_latency": { 2024-03-19T03:42:40.235 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.235 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.235 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.235 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.236 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mknod_latency": { 2024-03-19T03:42:40.236 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.236 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.236 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.236 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.236 INFO:teuthology.orchestra.run.smithi119.stdout: "req_mksnap_latency": { 2024-03-19T03:42:40.236 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.236 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.236 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.236 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.236 INFO:teuthology.orchestra.run.smithi119.stdout: "req_open_latency": { 2024-03-19T03:42:40.236 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 762, 2024-03-19T03:42:40.236 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 1.228123357, 2024-03-19T03:42:40.236 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.001611710 2024-03-19T03:42:40.237 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.237 INFO:teuthology.orchestra.run.smithi119.stdout: "req_readdir_latency": { 2024-03-19T03:42:40.237 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 33, 2024-03-19T03:42:40.237 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.042679268, 2024-03-19T03:42:40.237 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.001293311 2024-03-19T03:42:40.237 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.237 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rename_latency": { 2024-03-19T03:42:40.237 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 879, 2024-03-19T03:42:40.237 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 15.203036851, 2024-03-19T03:42:40.237 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.017295832 2024-03-19T03:42:40.237 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.237 INFO:teuthology.orchestra.run.smithi119.stdout: "req_renamesnap_latency": { 2024-03-19T03:42:40.237 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.237 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.237 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.237 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.238 INFO:teuthology.orchestra.run.smithi119.stdout: "req_snapdiff_latency": { 2024-03-19T03:42:40.238 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.238 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.238 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.238 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.238 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmdir_latency": { 2024-03-19T03:42:40.238 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.238 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.238 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.238 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.238 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmsnap_latency": { 2024-03-19T03:42:40.238 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.238 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.238 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.239 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.239 INFO:teuthology.orchestra.run.smithi119.stdout: "req_rmxattr_latency": { 2024-03-19T03:42:40.239 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.239 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.239 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.239 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.239 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setattr_latency": { 2024-03-19T03:42:40.239 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.239 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.239 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.239 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.239 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setdirlayout_latency": { 2024-03-19T03:42:40.239 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.239 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.239 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.240 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.240 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setfilelock_latency": { 2024-03-19T03:42:40.240 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.240 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.240 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.240 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.240 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setlayout_latency": { 2024-03-19T03:42:40.240 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.240 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.240 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.240 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.240 INFO:teuthology.orchestra.run.smithi119.stdout: "req_setxattr_latency": { 2024-03-19T03:42:40.240 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.240 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.241 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.241 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.241 INFO:teuthology.orchestra.run.smithi119.stdout: "req_symlink_latency": { 2024-03-19T03:42:40.241 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.241 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.241 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.241 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.241 INFO:teuthology.orchestra.run.smithi119.stdout: "req_unlink_latency": { 2024-03-19T03:42:40.241 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 825, 2024-03-19T03:42:40.241 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 2.252920702, 2024-03-19T03:42:40.241 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.002730812 2024-03-19T03:42:40.241 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.241 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_revoke_eviction": 0, 2024-03-19T03:42:40.241 INFO:teuthology.orchestra.run.smithi119.stdout: "cap_acquisition_throttle": 0, 2024-03-19T03:42:40.241 INFO:teuthology.orchestra.run.smithi119.stdout: "req_getvxattr_latency": { 2024-03-19T03:42:40.242 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.242 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.242 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.242 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:40.242 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.242 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_sessions": { 2024-03-19T03:42:40.242 INFO:teuthology.orchestra.run.smithi119.stdout: "session_count": 1, 2024-03-19T03:42:40.242 INFO:teuthology.orchestra.run.smithi119.stdout: "session_add": 2, 2024-03-19T03:42:40.242 INFO:teuthology.orchestra.run.smithi119.stdout: "session_remove": 1, 2024-03-19T03:42:40.242 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_open": 1, 2024-03-19T03:42:40.242 INFO:teuthology.orchestra.run.smithi119.stdout: "sessions_stale": 0, 2024-03-19T03:42:40.242 INFO:teuthology.orchestra.run.smithi119.stdout: "total_load": 1055, 2024-03-19T03:42:40.242 INFO:teuthology.orchestra.run.smithi119.stdout: "average_load": 1055, 2024-03-19T03:42:40.242 INFO:teuthology.orchestra.run.smithi119.stdout: "avg_session_uptime": 550, 2024-03-19T03:42:40.243 INFO:teuthology.orchestra.run.smithi119.stdout: "mdthresh_evicted": 0 2024-03-19T03:42:40.243 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.243 INFO:teuthology.orchestra.run.smithi119.stdout: "mempool": { 2024-03-19T03:42:40.243 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_bytes": 0, 2024-03-19T03:42:40.243 INFO:teuthology.orchestra.run.smithi119.stdout: "bloom_filter_items": 0, 2024-03-19T03:42:40.243 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T03:42:40.243 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_items": 0, 2024-03-19T03:42:40.243 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T03:42:40.243 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_data_items": 0, 2024-03-19T03:42:40.243 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T03:42:40.243 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T03:42:40.243 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T03:42:40.243 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T03:42:40.243 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T03:42:40.243 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_other_items": 0, 2024-03-19T03:42:40.244 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T03:42:40.244 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T03:42:40.244 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_bytes": 0, 2024-03-19T03:42:40.244 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_items": 0, 2024-03-19T03:42:40.244 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_bytes": 0, 2024-03-19T03:42:40.244 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blob_items": 0, 2024-03-19T03:42:40.244 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T03:42:40.244 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T03:42:40.244 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T03:42:40.244 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T03:42:40.244 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T03:42:40.244 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_items": 0, 2024-03-19T03:42:40.244 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_bytes": 0, 2024-03-19T03:42:40.244 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_txc_items": 0, 2024-03-19T03:42:40.245 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T03:42:40.245 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T03:42:40.245 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_bytes": 0, 2024-03-19T03:42:40.245 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_writing_items": 0, 2024-03-19T03:42:40.245 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_bytes": 0, 2024-03-19T03:42:40.245 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_items": 0, 2024-03-19T03:42:40.245 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T03:42:40.245 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_reader_items": 0, 2024-03-19T03:42:40.245 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T03:42:40.245 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_file_writer_items": 0, 2024-03-19T03:42:40.245 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_bytes": 292919, 2024-03-19T03:42:40.245 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_anon_items": 129, 2024-03-19T03:42:40.245 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_bytes": 0, 2024-03-19T03:42:40.245 INFO:teuthology.orchestra.run.smithi119.stdout: "buffer_meta_items": 0, 2024-03-19T03:42:40.246 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_bytes": 0, 2024-03-19T03:42:40.246 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_items": 0, 2024-03-19T03:42:40.246 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_bytes": 0, 2024-03-19T03:42:40.246 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mapbl_items": 0, 2024-03-19T03:42:40.246 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_bytes": 0, 2024-03-19T03:42:40.246 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pglog_items": 0, 2024-03-19T03:42:40.246 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_bytes": 9184, 2024-03-19T03:42:40.246 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_items": 177, 2024-03-19T03:42:40.246 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T03:42:40.246 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_mapping_items": 0, 2024-03-19T03:42:40.246 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_bytes": 0, 2024-03-19T03:42:40.246 INFO:teuthology.orchestra.run.smithi119.stdout: "pgmap_items": 0, 2024-03-19T03:42:40.246 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_bytes": 98965, 2024-03-19T03:42:40.247 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_co_items": 437, 2024-03-19T03:42:40.247 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_bytes": 0, 2024-03-19T03:42:40.247 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_1_items": 0, 2024-03-19T03:42:40.247 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_bytes": 0, 2024-03-19T03:42:40.247 INFO:teuthology.orchestra.run.smithi119.stdout: "unittest_2_items": 0 2024-03-19T03:42:40.247 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.247 INFO:teuthology.orchestra.run.smithi119.stdout: "objecter": { 2024-03-19T03:42:40.247 INFO:teuthology.orchestra.run.smithi119.stdout: "op_active": 0, 2024-03-19T03:42:40.247 INFO:teuthology.orchestra.run.smithi119.stdout: "op_laggy": 0, 2024-03-19T03:42:40.247 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send": 3091, 2024-03-19T03:42:40.247 INFO:teuthology.orchestra.run.smithi119.stdout: "op_send_bytes": 10607732, 2024-03-19T03:42:40.247 INFO:teuthology.orchestra.run.smithi119.stdout: "op_resend": 33, 2024-03-19T03:42:40.247 INFO:teuthology.orchestra.run.smithi119.stdout: "op_reply": 3091, 2024-03-19T03:42:40.248 INFO:teuthology.orchestra.run.smithi119.stdout: "op_latency": { 2024-03-19T03:42:40.248 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 3091, 2024-03-19T03:42:40.248 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 64.722395918, 2024-03-19T03:42:40.248 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.020938982 2024-03-19T03:42:40.248 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.248 INFO:teuthology.orchestra.run.smithi119.stdout: "op_inflight": 0, 2024-03-19T03:42:40.248 INFO:teuthology.orchestra.run.smithi119.stdout: "oplen_avg": { 2024-03-19T03:42:40.248 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 3091, 2024-03-19T03:42:40.248 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 3325 2024-03-19T03:42:40.248 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.248 INFO:teuthology.orchestra.run.smithi119.stdout: "op": 3091, 2024-03-19T03:42:40.248 INFO:teuthology.orchestra.run.smithi119.stdout: "op_r": 71, 2024-03-19T03:42:40.248 INFO:teuthology.orchestra.run.smithi119.stdout: "op_w": 3020, 2024-03-19T03:42:40.248 INFO:teuthology.orchestra.run.smithi119.stdout: "op_rmw": 0, 2024-03-19T03:42:40.249 INFO:teuthology.orchestra.run.smithi119.stdout: "op_pg": 0, 2024-03-19T03:42:40.249 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_stat": 0, 2024-03-19T03:42:40.249 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_create": 10, 2024-03-19T03:42:40.249 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_read": 71, 2024-03-19T03:42:40.249 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_write": 339, 2024-03-19T03:42:40.249 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writefull": 68, 2024-03-19T03:42:40.249 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_writesame": 0, 2024-03-19T03:42:40.249 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_append": 0, 2024-03-19T03:42:40.249 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_zero": 0, 2024-03-19T03:42:40.249 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_truncate": 4, 2024-03-19T03:42:40.249 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_delete": 2408, 2024-03-19T03:42:40.249 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_mapext": 0, 2024-03-19T03:42:40.249 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_sparse_read": 0, 2024-03-19T03:42:40.249 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_clonerange": 0, 2024-03-19T03:42:40.249 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_getxattr": 0, 2024-03-19T03:42:40.250 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_setxattr": 20, 2024-03-19T03:42:40.250 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_cmpxattr": 0, 2024-03-19T03:42:40.250 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_rmxattr": 0, 2024-03-19T03:42:40.250 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_resetxattrs": 0, 2024-03-19T03:42:40.250 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_call": 0, 2024-03-19T03:42:40.250 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_watch": 0, 2024-03-19T03:42:40.250 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_notify": 0, 2024-03-19T03:42:40.250 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T03:42:40.250 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls": 0, 2024-03-19T03:42:40.250 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_pgls_filter": 0, 2024-03-19T03:42:40.250 INFO:teuthology.orchestra.run.smithi119.stdout: "osdop_other": 0, 2024-03-19T03:42:40.250 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_active": 0, 2024-03-19T03:42:40.250 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_send": 0, 2024-03-19T03:42:40.250 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_resend": 0, 2024-03-19T03:42:40.251 INFO:teuthology.orchestra.run.smithi119.stdout: "linger_ping": 0, 2024-03-19T03:42:40.251 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_active": 0, 2024-03-19T03:42:40.251 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_send": 0, 2024-03-19T03:42:40.251 INFO:teuthology.orchestra.run.smithi119.stdout: "poolop_resend": 0, 2024-03-19T03:42:40.251 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_active": 0, 2024-03-19T03:42:40.251 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_send": 0, 2024-03-19T03:42:40.251 INFO:teuthology.orchestra.run.smithi119.stdout: "poolstat_resend": 0, 2024-03-19T03:42:40.251 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_active": 0, 2024-03-19T03:42:40.251 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_send": 0, 2024-03-19T03:42:40.251 INFO:teuthology.orchestra.run.smithi119.stdout: "statfs_resend": 0, 2024-03-19T03:42:40.251 INFO:teuthology.orchestra.run.smithi119.stdout: "command_active": 0, 2024-03-19T03:42:40.251 INFO:teuthology.orchestra.run.smithi119.stdout: "command_send": 0, 2024-03-19T03:42:40.251 INFO:teuthology.orchestra.run.smithi119.stdout: "command_resend": 0, 2024-03-19T03:42:40.251 INFO:teuthology.orchestra.run.smithi119.stdout: "map_epoch": 93, 2024-03-19T03:42:40.252 INFO:teuthology.orchestra.run.smithi119.stdout: "map_full": 0, 2024-03-19T03:42:40.252 INFO:teuthology.orchestra.run.smithi119.stdout: "map_inc": 4, 2024-03-19T03:42:40.252 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_sessions": 12, 2024-03-19T03:42:40.252 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_open": 12, 2024-03-19T03:42:40.252 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_session_close": 0, 2024-03-19T03:42:40.252 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_laggy": 0, 2024-03-19T03:42:40.252 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_wr": 331, 2024-03-19T03:42:40.252 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_rd": 0, 2024-03-19T03:42:40.252 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_del": 74 2024-03-19T03:42:40.252 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.252 INFO:teuthology.orchestra.run.smithi119.stdout: "oft": { 2024-03-19T03:42:40.252 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_objs": 1, 2024-03-19T03:42:40.252 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_kv_pairs": 0, 2024-03-19T03:42:40.252 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_updates": 0, 2024-03-19T03:42:40.253 INFO:teuthology.orchestra.run.smithi119.stdout: "omap_total_removes": 0 2024-03-19T03:42:40.253 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.253 INFO:teuthology.orchestra.run.smithi119.stdout: "osd-slow-ops": { 2024-03-19T03:42:40.253 INFO:teuthology.orchestra.run.smithi119.stdout: "slow_ops_count": 0 2024-03-19T03:42:40.253 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.253 INFO:teuthology.orchestra.run.smithi119.stdout: "purge_queue": { 2024-03-19T03:42:40.253 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops": 0, 2024-03-19T03:42:40.253 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_ops_high_water": 10, 2024-03-19T03:42:40.253 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing": 0, 2024-03-19T03:42:40.253 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executing_high_water": 5, 2024-03-19T03:42:40.253 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed_ops": 2394, 2024-03-19T03:42:40.253 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_executed": 1197, 2024-03-19T03:42:40.253 INFO:teuthology.orchestra.run.smithi119.stdout: "pq_item_in_journal": 0 2024-03-19T03:42:40.253 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.253 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T03:42:40.254 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-19T03:42:40.254 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-03-19T03:42:40.254 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-19T03:42:40.254 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 37532, 2024-03-19T03:42:40.254 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 11597618, 2024-03-19T03:42:40.254 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:40.254 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 37532, 2024-03-19T03:42:40.254 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-19T03:42:40.254 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-19T03:42:40.254 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 37532, 2024-03-19T03:42:40.254 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 11597618, 2024-03-19T03:42:40.254 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-19T03:42:40.254 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.254 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.254 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.255 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:40.255 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.255 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_bytes": { 2024-03-19T03:42:40.255 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-19T03:42:40.255 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 104857600, 2024-03-19T03:42:40.255 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-19T03:42:40.255 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-19T03:42:40.255 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-19T03:42:40.255 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:40.255 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:40.255 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 3091, 2024-03-19T03:42:40.255 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 10738205, 2024-03-19T03:42:40.255 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 3091, 2024-03-19T03:42:40.255 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 10738205, 2024-03-19T03:42:40.256 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-19T03:42:40.256 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.256 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.256 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.256 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:40.256 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.256 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-objecter_ops": { 2024-03-19T03:42:40.256 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-19T03:42:40.256 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 1024, 2024-03-19T03:42:40.256 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-19T03:42:40.256 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 0, 2024-03-19T03:42:40.256 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 0, 2024-03-19T03:42:40.256 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:40.256 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:40.256 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 3091, 2024-03-19T03:42:40.257 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 3091, 2024-03-19T03:42:40.257 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 3091, 2024-03-19T03:42:40.257 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 3091, 2024-03-19T03:42:40.257 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-19T03:42:40.257 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.257 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.257 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.257 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:40.257 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.257 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle": { 2024-03-19T03:42:40.257 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-19T03:42:40.257 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-03-19T03:42:40.257 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-19T03:42:40.257 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 1197, 2024-03-19T03:42:40.257 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 130473, 2024-03-19T03:42:40.257 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:40.258 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 1197, 2024-03-19T03:42:40.258 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-19T03:42:40.258 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-19T03:42:40.258 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 71, 2024-03-19T03:42:40.258 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 130473, 2024-03-19T03:42:40.258 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-19T03:42:40.258 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.258 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.258 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.258 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:40.258 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-03-19T03:42:40.258 INFO:teuthology.orchestra.run.smithi119.stdout: "throttle-write_buf_throttle-0x556eebccc500": { 2024-03-19T03:42:40.258 INFO:teuthology.orchestra.run.smithi119.stdout: "val": 0, 2024-03-19T03:42:40.259 INFO:teuthology.orchestra.run.smithi119.stdout: "max": 3758096384, 2024-03-19T03:42:40.259 INFO:teuthology.orchestra.run.smithi119.stdout: "get_started": 0, 2024-03-19T03:42:40.259 INFO:teuthology.orchestra.run.smithi119.stdout: "get": 6494, 2024-03-19T03:42:40.259 INFO:teuthology.orchestra.run.smithi119.stdout: "get_sum": 9639303, 2024-03-19T03:42:40.259 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:40.259 INFO:teuthology.orchestra.run.smithi119.stdout: "get_or_fail_success": 6494, 2024-03-19T03:42:40.259 INFO:teuthology.orchestra.run.smithi119.stdout: "take": 0, 2024-03-19T03:42:40.259 INFO:teuthology.orchestra.run.smithi119.stdout: "take_sum": 0, 2024-03-19T03:42:40.259 INFO:teuthology.orchestra.run.smithi119.stdout: "put": 268, 2024-03-19T03:42:40.259 INFO:teuthology.orchestra.run.smithi119.stdout: "put_sum": 9639303, 2024-03-19T03:42:40.259 INFO:teuthology.orchestra.run.smithi119.stdout: "wait": { 2024-03-19T03:42:40.259 INFO:teuthology.orchestra.run.smithi119.stdout: "avgcount": 0, 2024-03-19T03:42:40.259 INFO:teuthology.orchestra.run.smithi119.stdout: "sum": 0.000000000, 2024-03-19T03:42:40.259 INFO:teuthology.orchestra.run.smithi119.stdout: "avgtime": 0.000000000 2024-03-19T03:42:40.259 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:40.260 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-03-19T03:42:40.260 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-03-19T03:42:40.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:40 smithi119 ceph-mon[66861]: pgmap v684: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-19T03:42:41.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:40 smithi117 ceph-mon[31838]: pgmap v684: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-19T03:42:41.052 INFO:tasks.check_counter:Daemon mds.k mds.exported=33 2024-03-19T03:42:41.052 INFO:tasks.check_counter:Daemon mds.k mds.imported=5 2024-03-19T03:42:41.052 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update=484 2024-03-19T03:42:41.052 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update_receipt=4874 2024-03-19T03:42:41.052 INFO:tasks.check_counter:Daemon mds.k mds.dir_split=4 2024-03-19T03:42:41.052 DEBUG:tasks.check_counter:Getting stats from c 2024-03-19T03:42:41.053 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-19T03:42:41.082 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:40 smithi178 ceph-mon[67578]: pgmap v684: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-19T03:42:41.302 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T03:42:41.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:41 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:42:42.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:41 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:42:42.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:41 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T03:42:42.971 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:42 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:42:42.972 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:42 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:42:42.972 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:42 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:42:42.972 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:42 smithi178 ceph-mon[67578]: pgmap v685: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T03:42:42.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:42 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:42:42.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:42 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:42:42.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:42 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:42:42.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:42 smithi119 ceph-mon[66861]: pgmap v685: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T03:42:43.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:42 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T03:42:43.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:42 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T03:42:43.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:42 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' 2024-03-19T03:42:43.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:42 smithi117 ceph-mon[31838]: pgmap v685: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T03:42:43.099 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-03-19T03:42:43.099 INFO:teuthology.orchestra.run.smithi178.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T03:42:43.099 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_messages": 928, 2024-03-19T03:42:43.099 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages": 903, 2024-03-19T03:42:43.099 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_bytes": 1131312, 2024-03-19T03:42:43.100 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_bytes": 345656, 2024-03-19T03:42:43.100 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_created_connections": 5, 2024-03-19T03:42:43.100 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_active_connections": 1, 2024-03-19T03:42:43.100 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_total_time": 0.106859742, 2024-03-19T03:42:43.100 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_send_time": 0.034536921, 2024-03-19T03:42:43.100 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_recv_time": 0.038598865, 2024-03-19T03:42:43.100 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_fast_dispatch_time": 0.028404898, 2024-03-19T03:42:43.100 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:43.100 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 902, 2024-03-19T03:42:43.100 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.042213939, 2024-03-19T03:42:43.100 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000046800 2024-03-19T03:42:43.100 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.100 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:43.101 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.101 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.101 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.101 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.101 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_encrypted_bytes": 898800, 2024-03-19T03:42:43.101 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_encrypted_bytes": 125600 2024-03-19T03:42:43.101 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.101 INFO:teuthology.orchestra.run.smithi178.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T03:42:43.101 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_messages": 7, 2024-03-19T03:42:43.101 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages": 158, 2024-03-19T03:42:43.101 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_bytes": 3788, 2024-03-19T03:42:43.101 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_bytes": 339946, 2024-03-19T03:42:43.101 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_created_connections": 4, 2024-03-19T03:42:43.101 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_active_connections": 3, 2024-03-19T03:42:43.101 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_total_time": 0.009638949, 2024-03-19T03:42:43.102 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_send_time": 0.006297624, 2024-03-19T03:42:43.102 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_recv_time": 0.001929228, 2024-03-19T03:42:43.102 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_fast_dispatch_time": 0.000053348, 2024-03-19T03:42:43.102 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:43.102 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 157, 2024-03-19T03:42:43.102 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.011982515, 2024-03-19T03:42:43.102 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000076321 2024-03-19T03:42:43.102 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.102 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:43.102 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.102 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.102 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.102 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.103 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_encrypted_bytes": 3232, 2024-03-19T03:42:43.103 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_encrypted_bytes": 339248 2024-03-19T03:42:43.103 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.103 INFO:teuthology.orchestra.run.smithi178.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T03:42:43.103 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_messages": 1395, 2024-03-19T03:42:43.103 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages": 1397, 2024-03-19T03:42:43.103 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_bytes": 5687955, 2024-03-19T03:42:43.103 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_bytes": 443617, 2024-03-19T03:42:43.103 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_created_connections": 4, 2024-03-19T03:42:43.103 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_active_connections": 3, 2024-03-19T03:42:43.103 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_total_time": 0.088740760, 2024-03-19T03:42:43.103 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_send_time": 0.028171201, 2024-03-19T03:42:43.103 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_recv_time": 0.024664467, 2024-03-19T03:42:43.103 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_fast_dispatch_time": 0.029996104, 2024-03-19T03:42:43.103 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:43.104 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 1395, 2024-03-19T03:42:43.104 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.043288211, 2024-03-19T03:42:43.104 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000031030 2024-03-19T03:42:43.104 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.104 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:43.104 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.104 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.104 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.104 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.104 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T03:42:43.104 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T03:42:43.104 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.104 INFO:teuthology.orchestra.run.smithi178.stdout: "cct": { 2024-03-19T03:42:43.104 INFO:teuthology.orchestra.run.smithi178.stdout: "total_workers": 1, 2024-03-19T03:42:43.105 INFO:teuthology.orchestra.run.smithi178.stdout: "unhealthy_workers": 0 2024-03-19T03:42:43.105 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.105 INFO:teuthology.orchestra.run.smithi178.stdout: "finisher-MDSRank": { 2024-03-19T03:42:43.105 INFO:teuthology.orchestra.run.smithi178.stdout: "queue_len": 0, 2024-03-19T03:42:43.105 INFO:teuthology.orchestra.run.smithi178.stdout: "complete_latency": { 2024-03-19T03:42:43.105 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 3560, 2024-03-19T03:42:43.105 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.084433425, 2024-03-19T03:42:43.105 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000023717 2024-03-19T03:42:43.105 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:43.105 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.105 INFO:teuthology.orchestra.run.smithi178.stdout: "finisher-PurgeQueue": { 2024-03-19T03:42:43.105 INFO:teuthology.orchestra.run.smithi178.stdout: "queue_len": 0, 2024-03-19T03:42:43.106 INFO:teuthology.orchestra.run.smithi178.stdout: "complete_latency": { 2024-03-19T03:42:43.106 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.106 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.106 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.106 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:43.106 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.106 INFO:teuthology.orchestra.run.smithi178.stdout: "mds": { 2024-03-19T03:42:43.106 INFO:teuthology.orchestra.run.smithi178.stdout: "request": 0, 2024-03-19T03:42:43.106 INFO:teuthology.orchestra.run.smithi178.stdout: "reply": 0, 2024-03-19T03:42:43.106 INFO:teuthology.orchestra.run.smithi178.stdout: "reply_latency": { 2024-03-19T03:42:43.106 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.106 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.106 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.107 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.107 INFO:teuthology.orchestra.run.smithi178.stdout: "slow_reply": 0, 2024-03-19T03:42:43.107 INFO:teuthology.orchestra.run.smithi178.stdout: "forward": 0, 2024-03-19T03:42:43.107 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_fetch_complete": 0, 2024-03-19T03:42:43.107 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_fetch_keys": 0, 2024-03-19T03:42:43.107 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_commit": 0, 2024-03-19T03:42:43.107 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_split": 0, 2024-03-19T03:42:43.107 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_merge": 0, 2024-03-19T03:42:43.107 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes": 683, 2024-03-19T03:42:43.107 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_top": 0, 2024-03-19T03:42:43.107 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_bottom": 683, 2024-03-19T03:42:43.107 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_pin_tail": 0, 2024-03-19T03:42:43.107 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_pinned": 683, 2024-03-19T03:42:43.107 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_expired": 0, 2024-03-19T03:42:43.107 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_with_caps": 0, 2024-03-19T03:42:43.107 INFO:teuthology.orchestra.run.smithi178.stdout: "caps": 0, 2024-03-19T03:42:43.108 INFO:teuthology.orchestra.run.smithi178.stdout: "subtrees": 1, 2024-03-19T03:42:43.108 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse": 0, 2024-03-19T03:42:43.108 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_hit": 0, 2024-03-19T03:42:43.108 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_forward": 0, 2024-03-19T03:42:43.108 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_discover": 0, 2024-03-19T03:42:43.108 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_dir_fetch": 0, 2024-03-19T03:42:43.108 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_remote_ino": 0, 2024-03-19T03:42:43.108 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_lock": 0, 2024-03-19T03:42:43.108 INFO:teuthology.orchestra.run.smithi178.stdout: "load_cent": 0, 2024-03-19T03:42:43.108 INFO:teuthology.orchestra.run.smithi178.stdout: "q": 0, 2024-03-19T03:42:43.108 INFO:teuthology.orchestra.run.smithi178.stdout: "exported": 0, 2024-03-19T03:42:43.108 INFO:teuthology.orchestra.run.smithi178.stdout: "exported_inodes": 0, 2024-03-19T03:42:43.108 INFO:teuthology.orchestra.run.smithi178.stdout: "imported": 0, 2024-03-19T03:42:43.109 INFO:teuthology.orchestra.run.smithi178.stdout: "imported_inodes": 0, 2024-03-19T03:42:43.109 INFO:teuthology.orchestra.run.smithi178.stdout: "openino_dir_fetch": 0, 2024-03-19T03:42:43.109 INFO:teuthology.orchestra.run.smithi178.stdout: "openino_backtrace_fetch": 0, 2024-03-19T03:42:43.109 INFO:teuthology.orchestra.run.smithi178.stdout: "openino_peer_discover": 0, 2024-03-19T03:42:43.109 INFO:teuthology.orchestra.run.smithi178.stdout: "root_rfiles": 0, 2024-03-19T03:42:43.109 INFO:teuthology.orchestra.run.smithi178.stdout: "root_rbytes": 0, 2024-03-19T03:42:43.109 INFO:teuthology.orchestra.run.smithi178.stdout: "root_rsnaps": 0, 2024-03-19T03:42:43.109 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T03:42:43.109 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_set_tag": 0, 2024-03-19T03:42:43.109 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T03:42:43.109 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_inotable_repaired": 0, 2024-03-19T03:42:43.109 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_dir_inodes": 0, 2024-03-19T03:42:43.109 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T03:42:43.110 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T03:42:43.110 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_file_inodes": 0, 2024-03-19T03:42:43.110 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_inode_file_caps": 0, 2024-03-19T03:42:43.110 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_revoke": 0, 2024-03-19T03:42:43.110 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_grant": 0, 2024-03-19T03:42:43.110 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T03:42:43.110 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T03:42:43.110 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T03:42:43.110 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_caps": 0, 2024-03-19T03:42:43.110 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_caps_dirty": 0, 2024-03-19T03:42:43.110 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_cap_release": 0, 2024-03-19T03:42:43.110 INFO:teuthology.orchestra.run.smithi178.stdout: "process_request_cap_release": 0 2024-03-19T03:42:43.110 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.111 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_cache": { 2024-03-19T03:42:43.111 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_update": 0, 2024-03-19T03:42:43.111 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_update_receipt": 0, 2024-03-19T03:42:43.111 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_try_discover": 0, 2024-03-19T03:42:43.111 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_send_discover": 0, 2024-03-19T03:42:43.111 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_handle_discover": 0, 2024-03-19T03:42:43.111 INFO:teuthology.orchestra.run.smithi178.stdout: "num_strays": 0, 2024-03-19T03:42:43.111 INFO:teuthology.orchestra.run.smithi178.stdout: "num_strays_delayed": 0, 2024-03-19T03:42:43.111 INFO:teuthology.orchestra.run.smithi178.stdout: "num_strays_enqueuing": 0, 2024-03-19T03:42:43.111 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_created": 0, 2024-03-19T03:42:43.111 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_enqueued": 0, 2024-03-19T03:42:43.111 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_reintegrated": 0, 2024-03-19T03:42:43.111 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_migrated": 0, 2024-03-19T03:42:43.111 INFO:teuthology.orchestra.run.smithi178.stdout: "num_recovering_processing": 0, 2024-03-19T03:42:43.112 INFO:teuthology.orchestra.run.smithi178.stdout: "num_recovering_enqueued": 0, 2024-03-19T03:42:43.112 INFO:teuthology.orchestra.run.smithi178.stdout: "num_recovering_prioritized": 0, 2024-03-19T03:42:43.112 INFO:teuthology.orchestra.run.smithi178.stdout: "recovery_started": 0, 2024-03-19T03:42:43.112 INFO:teuthology.orchestra.run.smithi178.stdout: "recovery_completed": 0, 2024-03-19T03:42:43.112 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_quiesce_path": 0, 2024-03-19T03:42:43.112 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_quiesce_inode": 0, 2024-03-19T03:42:43.112 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T03:42:43.112 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_exportdir": 0, 2024-03-19T03:42:43.112 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_flush": 0, 2024-03-19T03:42:43.112 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_fragmentdir": 0, 2024-03-19T03:42:43.112 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_fragstats": 0, 2024-03-19T03:42:43.112 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_inodestats": 0 2024-03-19T03:42:43.112 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.112 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_log": { 2024-03-19T03:42:43.113 INFO:teuthology.orchestra.run.smithi178.stdout: "evlrg": 0, 2024-03-19T03:42:43.113 INFO:teuthology.orchestra.run.smithi178.stdout: "evadd": 0, 2024-03-19T03:42:43.113 INFO:teuthology.orchestra.run.smithi178.stdout: "evex": 0, 2024-03-19T03:42:43.113 INFO:teuthology.orchestra.run.smithi178.stdout: "evtrm": 0, 2024-03-19T03:42:43.113 INFO:teuthology.orchestra.run.smithi178.stdout: "ev": 3520, 2024-03-19T03:42:43.113 INFO:teuthology.orchestra.run.smithi178.stdout: "evexg": 0, 2024-03-19T03:42:43.113 INFO:teuthology.orchestra.run.smithi178.stdout: "evexd": 0, 2024-03-19T03:42:43.113 INFO:teuthology.orchestra.run.smithi178.stdout: "segadd": 0, 2024-03-19T03:42:43.113 INFO:teuthology.orchestra.run.smithi178.stdout: "segex": 0, 2024-03-19T03:42:43.113 INFO:teuthology.orchestra.run.smithi178.stdout: "segtrm": 0, 2024-03-19T03:42:43.113 INFO:teuthology.orchestra.run.smithi178.stdout: "seg": 5, 2024-03-19T03:42:43.113 INFO:teuthology.orchestra.run.smithi178.stdout: "segmjr": 2, 2024-03-19T03:42:43.114 INFO:teuthology.orchestra.run.smithi178.stdout: "segexg": 0, 2024-03-19T03:42:43.114 INFO:teuthology.orchestra.run.smithi178.stdout: "segexd": 0, 2024-03-19T03:42:43.114 INFO:teuthology.orchestra.run.smithi178.stdout: "expos": 4194304, 2024-03-19T03:42:43.114 INFO:teuthology.orchestra.run.smithi178.stdout: "wrpos": 9524560, 2024-03-19T03:42:43.114 INFO:teuthology.orchestra.run.smithi178.stdout: "rdpos": 9523422, 2024-03-19T03:42:43.114 INFO:teuthology.orchestra.run.smithi178.stdout: "jlat": { 2024-03-19T03:42:43.114 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.114 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.114 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.114 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.114 INFO:teuthology.orchestra.run.smithi178.stdout: "replayed": 3520 2024-03-19T03:42:43.114 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.114 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_mem": { 2024-03-19T03:42:43.115 INFO:teuthology.orchestra.run.smithi178.stdout: "ino": 7, 2024-03-19T03:42:43.115 INFO:teuthology.orchestra.run.smithi178.stdout: "ino+": 701, 2024-03-19T03:42:43.115 INFO:teuthology.orchestra.run.smithi178.stdout: "ino-": 694, 2024-03-19T03:42:43.115 INFO:teuthology.orchestra.run.smithi178.stdout: "dir": 5, 2024-03-19T03:42:43.115 INFO:teuthology.orchestra.run.smithi178.stdout: "dir+": 37, 2024-03-19T03:42:43.115 INFO:teuthology.orchestra.run.smithi178.stdout: "dir-": 32, 2024-03-19T03:42:43.115 INFO:teuthology.orchestra.run.smithi178.stdout: "dn": 683, 2024-03-19T03:42:43.115 INFO:teuthology.orchestra.run.smithi178.stdout: "dn+": 1474, 2024-03-19T03:42:43.115 INFO:teuthology.orchestra.run.smithi178.stdout: "dn-": 791, 2024-03-19T03:42:43.115 INFO:teuthology.orchestra.run.smithi178.stdout: "cap": 0, 2024-03-19T03:42:43.115 INFO:teuthology.orchestra.run.smithi178.stdout: "cap+": 0, 2024-03-19T03:42:43.115 INFO:teuthology.orchestra.run.smithi178.stdout: "cap-": 0, 2024-03-19T03:42:43.115 INFO:teuthology.orchestra.run.smithi178.stdout: "rss": 48088, 2024-03-19T03:42:43.115 INFO:teuthology.orchestra.run.smithi178.stdout: "heap": 215420 2024-03-19T03:42:43.116 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.116 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_server": { 2024-03-19T03:42:43.116 INFO:teuthology.orchestra.run.smithi178.stdout: "dispatch_client_request": 0, 2024-03-19T03:42:43.116 INFO:teuthology.orchestra.run.smithi178.stdout: "dispatch_server_request": 0, 2024-03-19T03:42:43.116 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_request": 0, 2024-03-19T03:42:43.116 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_session": 0, 2024-03-19T03:42:43.116 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_peer_request": 0, 2024-03-19T03:42:43.116 INFO:teuthology.orchestra.run.smithi178.stdout: "req_create_latency": { 2024-03-19T03:42:43.116 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.116 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.116 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.116 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.116 INFO:teuthology.orchestra.run.smithi178.stdout: "req_getattr_latency": { 2024-03-19T03:42:43.117 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.117 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.117 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.117 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.117 INFO:teuthology.orchestra.run.smithi178.stdout: "req_getfilelock_latency": { 2024-03-19T03:42:43.117 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.117 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.117 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.117 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.117 INFO:teuthology.orchestra.run.smithi178.stdout: "req_link_latency": { 2024-03-19T03:42:43.117 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.117 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.117 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.117 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.118 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookup_latency": { 2024-03-19T03:42:43.118 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.118 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.118 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.118 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.118 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookuphash_latency": { 2024-03-19T03:42:43.118 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.118 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.118 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.118 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.118 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupino_latency": { 2024-03-19T03:42:43.118 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.119 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.119 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.119 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.119 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupname_latency": { 2024-03-19T03:42:43.119 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.119 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.119 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.119 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.119 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupparent_latency": { 2024-03-19T03:42:43.119 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.119 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.119 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.120 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.120 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupsnap_latency": { 2024-03-19T03:42:43.120 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.120 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.120 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.120 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.120 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lssnap_latency": { 2024-03-19T03:42:43.120 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.120 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.120 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.120 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.120 INFO:teuthology.orchestra.run.smithi178.stdout: "req_mkdir_latency": { 2024-03-19T03:42:43.121 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.121 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.121 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.121 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.121 INFO:teuthology.orchestra.run.smithi178.stdout: "req_mknod_latency": { 2024-03-19T03:42:43.121 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.121 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.122 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.122 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.122 INFO:teuthology.orchestra.run.smithi178.stdout: "req_mksnap_latency": { 2024-03-19T03:42:43.122 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.122 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.122 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.122 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.122 INFO:teuthology.orchestra.run.smithi178.stdout: "req_open_latency": { 2024-03-19T03:42:43.122 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.122 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.122 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.122 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.122 INFO:teuthology.orchestra.run.smithi178.stdout: "req_readdir_latency": { 2024-03-19T03:42:43.123 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.123 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.123 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.123 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.123 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rename_latency": { 2024-03-19T03:42:43.123 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.123 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.123 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.123 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.123 INFO:teuthology.orchestra.run.smithi178.stdout: "req_renamesnap_latency": { 2024-03-19T03:42:43.123 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.123 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.123 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.123 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.124 INFO:teuthology.orchestra.run.smithi178.stdout: "req_snapdiff_latency": { 2024-03-19T03:42:43.124 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.124 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.124 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.124 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.124 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rmdir_latency": { 2024-03-19T03:42:43.124 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.124 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.124 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.124 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.124 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rmsnap_latency": { 2024-03-19T03:42:43.124 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.124 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.124 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.125 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.125 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rmxattr_latency": { 2024-03-19T03:42:43.125 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.125 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.125 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.125 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.125 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setattr_latency": { 2024-03-19T03:42:43.125 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.125 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.125 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.125 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.125 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setdirlayout_latency": { 2024-03-19T03:42:43.125 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.125 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.126 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.126 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.126 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setfilelock_latency": { 2024-03-19T03:42:43.126 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.126 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.126 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.126 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.126 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setlayout_latency": { 2024-03-19T03:42:43.126 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.126 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.126 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.126 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.127 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setxattr_latency": { 2024-03-19T03:42:43.127 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.127 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.127 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.127 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.127 INFO:teuthology.orchestra.run.smithi178.stdout: "req_symlink_latency": { 2024-03-19T03:42:43.127 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.127 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.127 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.127 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.127 INFO:teuthology.orchestra.run.smithi178.stdout: "req_unlink_latency": { 2024-03-19T03:42:43.127 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.127 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.128 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.128 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.128 INFO:teuthology.orchestra.run.smithi178.stdout: "cap_revoke_eviction": 0, 2024-03-19T03:42:43.128 INFO:teuthology.orchestra.run.smithi178.stdout: "cap_acquisition_throttle": 0, 2024-03-19T03:42:43.128 INFO:teuthology.orchestra.run.smithi178.stdout: "req_getvxattr_latency": { 2024-03-19T03:42:43.128 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.128 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.128 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.128 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:43.128 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.128 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_sessions": { 2024-03-19T03:42:43.128 INFO:teuthology.orchestra.run.smithi178.stdout: "session_count": 1, 2024-03-19T03:42:43.128 INFO:teuthology.orchestra.run.smithi178.stdout: "session_add": 3, 2024-03-19T03:42:43.128 INFO:teuthology.orchestra.run.smithi178.stdout: "session_remove": 2, 2024-03-19T03:42:43.128 INFO:teuthology.orchestra.run.smithi178.stdout: "sessions_open": 2, 2024-03-19T03:42:43.129 INFO:teuthology.orchestra.run.smithi178.stdout: "sessions_stale": 0, 2024-03-19T03:42:43.129 INFO:teuthology.orchestra.run.smithi178.stdout: "total_load": 0, 2024-03-19T03:42:43.129 INFO:teuthology.orchestra.run.smithi178.stdout: "average_load": 0, 2024-03-19T03:42:43.129 INFO:teuthology.orchestra.run.smithi178.stdout: "avg_session_uptime": 6868, 2024-03-19T03:42:43.129 INFO:teuthology.orchestra.run.smithi178.stdout: "mdthresh_evicted": 0 2024-03-19T03:42:43.129 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.129 INFO:teuthology.orchestra.run.smithi178.stdout: "mempool": { 2024-03-19T03:42:43.129 INFO:teuthology.orchestra.run.smithi178.stdout: "bloom_filter_bytes": 0, 2024-03-19T03:42:43.129 INFO:teuthology.orchestra.run.smithi178.stdout: "bloom_filter_items": 0, 2024-03-19T03:42:43.129 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T03:42:43.129 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_alloc_items": 0, 2024-03-19T03:42:43.129 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T03:42:43.129 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_data_items": 0, 2024-03-19T03:42:43.129 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T03:42:43.129 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T03:42:43.129 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T03:42:43.130 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T03:42:43.130 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T03:42:43.130 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_other_items": 0, 2024-03-19T03:42:43.130 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T03:42:43.130 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T03:42:43.130 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_extent_bytes": 0, 2024-03-19T03:42:43.130 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_extent_items": 0, 2024-03-19T03:42:43.130 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_blob_bytes": 0, 2024-03-19T03:42:43.130 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_blob_items": 0, 2024-03-19T03:42:43.130 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T03:42:43.130 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T03:42:43.130 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T03:42:43.130 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T03:42:43.131 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T03:42:43.131 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_items": 0, 2024-03-19T03:42:43.131 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_txc_bytes": 0, 2024-03-19T03:42:43.131 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_txc_items": 0, 2024-03-19T03:42:43.131 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T03:42:43.131 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T03:42:43.131 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_bytes": 0, 2024-03-19T03:42:43.131 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_items": 0, 2024-03-19T03:42:43.131 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_bytes": 0, 2024-03-19T03:42:43.131 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_items": 0, 2024-03-19T03:42:43.131 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T03:42:43.131 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_reader_items": 0, 2024-03-19T03:42:43.131 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T03:42:43.131 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_writer_items": 0, 2024-03-19T03:42:43.131 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_anon_bytes": 181321, 2024-03-19T03:42:43.131 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_anon_items": 47, 2024-03-19T03:42:43.132 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_meta_bytes": 0, 2024-03-19T03:42:43.132 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_meta_items": 0, 2024-03-19T03:42:43.132 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_bytes": 0, 2024-03-19T03:42:43.132 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_items": 0, 2024-03-19T03:42:43.132 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mapbl_bytes": 0, 2024-03-19T03:42:43.132 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mapbl_items": 0, 2024-03-19T03:42:43.132 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pglog_bytes": 0, 2024-03-19T03:42:43.132 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pglog_items": 0, 2024-03-19T03:42:43.132 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_bytes": 9184, 2024-03-19T03:42:43.132 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_items": 177, 2024-03-19T03:42:43.132 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T03:42:43.132 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_mapping_items": 0, 2024-03-19T03:42:43.132 INFO:teuthology.orchestra.run.smithi178.stdout: "pgmap_bytes": 0, 2024-03-19T03:42:43.132 INFO:teuthology.orchestra.run.smithi178.stdout: "pgmap_items": 0, 2024-03-19T03:42:43.132 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_co_bytes": 503288, 2024-03-19T03:42:43.133 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_co_items": 2794, 2024-03-19T03:42:43.133 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_1_bytes": 0, 2024-03-19T03:42:43.133 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_1_items": 0, 2024-03-19T03:42:43.133 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_2_bytes": 0, 2024-03-19T03:42:43.133 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_2_items": 0 2024-03-19T03:42:43.133 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.133 INFO:teuthology.orchestra.run.smithi178.stdout: "objecter": { 2024-03-19T03:42:43.133 INFO:teuthology.orchestra.run.smithi178.stdout: "op_active": 0, 2024-03-19T03:42:43.133 INFO:teuthology.orchestra.run.smithi178.stdout: "op_laggy": 0, 2024-03-19T03:42:43.133 INFO:teuthology.orchestra.run.smithi178.stdout: "op_send": 2089, 2024-03-19T03:42:43.133 INFO:teuthology.orchestra.run.smithi178.stdout: "op_send_bytes": 21, 2024-03-19T03:42:43.133 INFO:teuthology.orchestra.run.smithi178.stdout: "op_resend": 3, 2024-03-19T03:42:43.133 INFO:teuthology.orchestra.run.smithi178.stdout: "op_reply": 2089, 2024-03-19T03:42:43.133 INFO:teuthology.orchestra.run.smithi178.stdout: "op_latency": { 2024-03-19T03:42:43.133 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 2089, 2024-03-19T03:42:43.134 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 2.973941451, 2024-03-19T03:42:43.134 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.001423619 2024-03-19T03:42:43.134 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.134 INFO:teuthology.orchestra.run.smithi178.stdout: "op_inflight": 0, 2024-03-19T03:42:43.134 INFO:teuthology.orchestra.run.smithi178.stdout: "oplen_avg": { 2024-03-19T03:42:43.134 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 2089, 2024-03-19T03:42:43.134 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 2090 2024-03-19T03:42:43.134 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.134 INFO:teuthology.orchestra.run.smithi178.stdout: "op": 2089, 2024-03-19T03:42:43.134 INFO:teuthology.orchestra.run.smithi178.stdout: "op_r": 2089, 2024-03-19T03:42:43.134 INFO:teuthology.orchestra.run.smithi178.stdout: "op_w": 0, 2024-03-19T03:42:43.134 INFO:teuthology.orchestra.run.smithi178.stdout: "op_rmw": 0, 2024-03-19T03:42:43.134 INFO:teuthology.orchestra.run.smithi178.stdout: "op_pg": 0, 2024-03-19T03:42:43.135 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_stat": 1294, 2024-03-19T03:42:43.135 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_create": 0, 2024-03-19T03:42:43.135 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_read": 793, 2024-03-19T03:42:43.135 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_write": 0, 2024-03-19T03:42:43.135 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_writefull": 0, 2024-03-19T03:42:43.135 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_writesame": 0, 2024-03-19T03:42:43.135 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_append": 0, 2024-03-19T03:42:43.135 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_zero": 0, 2024-03-19T03:42:43.135 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_truncate": 0, 2024-03-19T03:42:43.135 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_delete": 0, 2024-03-19T03:42:43.135 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_mapext": 0, 2024-03-19T03:42:43.135 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_sparse_read": 0, 2024-03-19T03:42:43.135 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_clonerange": 0, 2024-03-19T03:42:43.135 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_getxattr": 1, 2024-03-19T03:42:43.135 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_setxattr": 0, 2024-03-19T03:42:43.135 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_cmpxattr": 0, 2024-03-19T03:42:43.136 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_rmxattr": 0, 2024-03-19T03:42:43.136 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_resetxattrs": 0, 2024-03-19T03:42:43.136 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_call": 0, 2024-03-19T03:42:43.136 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_watch": 0, 2024-03-19T03:42:43.136 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_notify": 0, 2024-03-19T03:42:43.136 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T03:42:43.137 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_pgls": 0, 2024-03-19T03:42:43.137 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_pgls_filter": 0, 2024-03-19T03:42:43.137 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_other": 0, 2024-03-19T03:42:43.138 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_active": 0, 2024-03-19T03:42:43.138 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_send": 0, 2024-03-19T03:42:43.138 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_resend": 0, 2024-03-19T03:42:43.138 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_ping": 0, 2024-03-19T03:42:43.138 INFO:teuthology.orchestra.run.smithi178.stdout: "poolop_active": 0, 2024-03-19T03:42:43.138 INFO:teuthology.orchestra.run.smithi178.stdout: "poolop_send": 0, 2024-03-19T03:42:43.138 INFO:teuthology.orchestra.run.smithi178.stdout: "poolop_resend": 0, 2024-03-19T03:42:43.138 INFO:teuthology.orchestra.run.smithi178.stdout: "poolstat_active": 0, 2024-03-19T03:42:43.138 INFO:teuthology.orchestra.run.smithi178.stdout: "poolstat_send": 0, 2024-03-19T03:42:43.138 INFO:teuthology.orchestra.run.smithi178.stdout: "poolstat_resend": 0, 2024-03-19T03:42:43.138 INFO:teuthology.orchestra.run.smithi178.stdout: "statfs_active": 0, 2024-03-19T03:42:43.138 INFO:teuthology.orchestra.run.smithi178.stdout: "statfs_send": 0, 2024-03-19T03:42:43.138 INFO:teuthology.orchestra.run.smithi178.stdout: "statfs_resend": 0, 2024-03-19T03:42:43.138 INFO:teuthology.orchestra.run.smithi178.stdout: "command_active": 0, 2024-03-19T03:42:43.138 INFO:teuthology.orchestra.run.smithi178.stdout: "command_send": 0, 2024-03-19T03:42:43.139 INFO:teuthology.orchestra.run.smithi178.stdout: "command_resend": 0, 2024-03-19T03:42:43.139 INFO:teuthology.orchestra.run.smithi178.stdout: "map_epoch": 93, 2024-03-19T03:42:43.139 INFO:teuthology.orchestra.run.smithi178.stdout: "map_full": 0, 2024-03-19T03:42:43.139 INFO:teuthology.orchestra.run.smithi178.stdout: "map_inc": 4, 2024-03-19T03:42:43.139 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_sessions": 4, 2024-03-19T03:42:43.139 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_session_open": 4, 2024-03-19T03:42:43.139 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_session_close": 0, 2024-03-19T03:42:43.139 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_laggy": 0, 2024-03-19T03:42:43.139 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_wr": 0, 2024-03-19T03:42:43.139 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_rd": 2, 2024-03-19T03:42:43.139 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_del": 0 2024-03-19T03:42:43.139 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.139 INFO:teuthology.orchestra.run.smithi178.stdout: "oft": { 2024-03-19T03:42:43.139 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_objs": 0, 2024-03-19T03:42:43.139 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_kv_pairs": 0, 2024-03-19T03:42:43.139 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_updates": 0, 2024-03-19T03:42:43.140 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_removes": 0 2024-03-19T03:42:43.140 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.140 INFO:teuthology.orchestra.run.smithi178.stdout: "osd-slow-ops": { 2024-03-19T03:42:43.140 INFO:teuthology.orchestra.run.smithi178.stdout: "slow_ops_count": 0 2024-03-19T03:42:43.140 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.140 INFO:teuthology.orchestra.run.smithi178.stdout: "purge_queue": { 2024-03-19T03:42:43.140 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing_ops": 0, 2024-03-19T03:42:43.140 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing_ops_high_water": 0, 2024-03-19T03:42:43.140 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing": 0, 2024-03-19T03:42:43.140 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing_high_water": 0, 2024-03-19T03:42:43.140 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executed_ops": 0, 2024-03-19T03:42:43.140 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executed": 0, 2024-03-19T03:42:43.140 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_item_in_journal": 0 2024-03-19T03:42:43.140 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.140 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T03:42:43.141 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-19T03:42:43.141 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 104857600, 2024-03-19T03:42:43.141 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-19T03:42:43.141 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 2326, 2024-03-19T03:42:43.141 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 6598676, 2024-03-19T03:42:43.141 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:43.141 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 2326, 2024-03-19T03:42:43.141 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 0, 2024-03-19T03:42:43.141 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 0, 2024-03-19T03:42:43.141 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 2326, 2024-03-19T03:42:43.141 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 6598676, 2024-03-19T03:42:43.141 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-19T03:42:43.141 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.141 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.141 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.142 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:43.142 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.142 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-objecter_bytes": { 2024-03-19T03:42:43.142 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-19T03:42:43.142 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 104857600, 2024-03-19T03:42:43.142 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-19T03:42:43.142 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-19T03:42:43.142 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-19T03:42:43.142 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:43.142 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:43.142 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 2089, 2024-03-19T03:42:43.142 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 5332753, 2024-03-19T03:42:43.142 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 2089, 2024-03-19T03:42:43.142 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 5332753, 2024-03-19T03:42:43.142 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-19T03:42:43.142 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.143 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.143 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.143 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:43.143 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.143 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-objecter_ops": { 2024-03-19T03:42:43.143 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-19T03:42:43.143 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 1024, 2024-03-19T03:42:43.143 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-19T03:42:43.143 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-19T03:42:43.143 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-19T03:42:43.143 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:43.143 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:43.143 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 2089, 2024-03-19T03:42:43.143 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 2089, 2024-03-19T03:42:43.143 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 2089, 2024-03-19T03:42:43.143 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 2089, 2024-03-19T03:42:43.144 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-19T03:42:43.144 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.144 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.144 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.144 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:43.144 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.144 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-write_buf_throttle": { 2024-03-19T03:42:43.144 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-19T03:42:43.144 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 3758096384, 2024-03-19T03:42:43.144 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-19T03:42:43.144 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-19T03:42:43.144 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-19T03:42:43.144 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:43.144 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:43.144 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 0, 2024-03-19T03:42:43.144 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 0, 2024-03-19T03:42:43.145 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 0, 2024-03-19T03:42:43.145 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 0, 2024-03-19T03:42:43.145 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-19T03:42:43.145 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.145 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.145 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.145 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:43.145 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:43.145 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-write_buf_throttle-0x55f6746466e0": { 2024-03-19T03:42:43.145 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-19T03:42:43.145 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 3758096384, 2024-03-19T03:42:43.145 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-19T03:42:43.145 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-19T03:42:43.145 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-19T03:42:43.146 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:43.146 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:43.146 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 0, 2024-03-19T03:42:43.146 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 0, 2024-03-19T03:42:43.146 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 0, 2024-03-19T03:42:43.146 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 0, 2024-03-19T03:42:43.146 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-19T03:42:43.146 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:43.146 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:43.146 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:43.146 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:43.146 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:43.146 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-03-19T03:42:43.934 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-19T03:42:43.935 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-19T03:42:43.935 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-03-19T03:42:43.935 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-03-19T03:42:43.935 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-03-19T03:42:43.935 DEBUG:tasks.check_counter:Getting stats from f 2024-03-19T03:42:43.935 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-19T03:42:44.211 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T03:42:44.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:44 smithi119 ceph-mon[66861]: pgmap v686: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T03:42:45.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:44 smithi117 ceph-mon[31838]: pgmap v686: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T03:42:45.046 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:44 smithi178 ceph-mon[67578]: pgmap v686: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T03:42:45.955 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-03-19T03:42:45.955 INFO:teuthology.orchestra.run.smithi178.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T03:42:45.955 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_messages": 1238, 2024-03-19T03:42:45.955 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages": 1238, 2024-03-19T03:42:45.955 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_bytes": 4570524, 2024-03-19T03:42:45.955 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_bytes": 392861, 2024-03-19T03:42:45.955 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_created_connections": 6, 2024-03-19T03:42:45.956 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_active_connections": 2, 2024-03-19T03:42:45.956 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_total_time": 0.115391505, 2024-03-19T03:42:45.956 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_send_time": 0.039329037, 2024-03-19T03:42:45.956 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_recv_time": 0.030973441, 2024-03-19T03:42:45.956 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_fast_dispatch_time": 0.034171787, 2024-03-19T03:42:45.956 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:45.956 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 1237, 2024-03-19T03:42:45.956 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.049153725, 2024-03-19T03:42:45.956 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000039736 2024-03-19T03:42:45.956 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.956 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:45.956 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.956 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.956 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.956 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.957 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-19T03:42:45.957 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_encrypted_bytes": 448 2024-03-19T03:42:45.957 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.957 INFO:teuthology.orchestra.run.smithi178.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T03:42:45.957 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_messages": 817, 2024-03-19T03:42:45.957 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages": 792, 2024-03-19T03:42:45.957 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_bytes": 4301116, 2024-03-19T03:42:45.957 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_bytes": 311070, 2024-03-19T03:42:45.957 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_created_connections": 4, 2024-03-19T03:42:45.957 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_active_connections": 3, 2024-03-19T03:42:45.957 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_total_time": 0.084061145, 2024-03-19T03:42:45.957 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_send_time": 0.028895138, 2024-03-19T03:42:45.957 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_recv_time": 0.034459045, 2024-03-19T03:42:45.958 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_fast_dispatch_time": 0.019967197, 2024-03-19T03:42:45.958 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:45.958 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 791, 2024-03-19T03:42:45.958 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.029624906, 2024-03-19T03:42:45.958 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000037452 2024-03-19T03:42:45.958 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.958 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:45.958 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.958 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.958 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.958 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.958 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_encrypted_bytes": 899344, 2024-03-19T03:42:45.958 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_encrypted_bytes": 126880 2024-03-19T03:42:45.959 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.959 INFO:teuthology.orchestra.run.smithi178.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T03:42:45.959 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_messages": 224, 2024-03-19T03:42:45.959 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages": 379, 2024-03-19T03:42:45.959 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_bytes": 54954, 2024-03-19T03:42:45.959 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_bytes": 489508, 2024-03-19T03:42:45.959 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_created_connections": 6, 2024-03-19T03:42:45.959 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_active_connections": 4, 2024-03-19T03:42:45.959 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_total_time": 0.027250613, 2024-03-19T03:42:45.959 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_send_time": 0.011500284, 2024-03-19T03:42:45.959 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_recv_time": 0.005622878, 2024-03-19T03:42:45.959 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_fast_dispatch_time": 0.007037822, 2024-03-19T03:42:45.959 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:45.959 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 376, 2024-03-19T03:42:45.959 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.034987485, 2024-03-19T03:42:45.960 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000093051 2024-03-19T03:42:45.960 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.960 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:45.960 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.960 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.960 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.960 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.960 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-19T03:42:45.960 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_encrypted_bytes": 418912 2024-03-19T03:42:45.960 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.960 INFO:teuthology.orchestra.run.smithi178.stdout: "cct": { 2024-03-19T03:42:45.960 INFO:teuthology.orchestra.run.smithi178.stdout: "total_workers": 1, 2024-03-19T03:42:45.960 INFO:teuthology.orchestra.run.smithi178.stdout: "unhealthy_workers": 0 2024-03-19T03:42:45.960 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.961 INFO:teuthology.orchestra.run.smithi178.stdout: "finisher-MDSRank": { 2024-03-19T03:42:45.961 INFO:teuthology.orchestra.run.smithi178.stdout: "queue_len": 0, 2024-03-19T03:42:45.961 INFO:teuthology.orchestra.run.smithi178.stdout: "complete_latency": { 2024-03-19T03:42:45.961 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 3482, 2024-03-19T03:42:45.961 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.085804729, 2024-03-19T03:42:45.961 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000024642 2024-03-19T03:42:45.961 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:45.961 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.961 INFO:teuthology.orchestra.run.smithi178.stdout: "finisher-PurgeQueue": { 2024-03-19T03:42:45.961 INFO:teuthology.orchestra.run.smithi178.stdout: "queue_len": 0, 2024-03-19T03:42:45.961 INFO:teuthology.orchestra.run.smithi178.stdout: "complete_latency": { 2024-03-19T03:42:45.961 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.961 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.961 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.962 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:45.962 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.962 INFO:teuthology.orchestra.run.smithi178.stdout: "mds": { 2024-03-19T03:42:45.962 INFO:teuthology.orchestra.run.smithi178.stdout: "request": 0, 2024-03-19T03:42:45.962 INFO:teuthology.orchestra.run.smithi178.stdout: "reply": 0, 2024-03-19T03:42:45.962 INFO:teuthology.orchestra.run.smithi178.stdout: "reply_latency": { 2024-03-19T03:42:45.962 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.962 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.962 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.963 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.963 INFO:teuthology.orchestra.run.smithi178.stdout: "slow_reply": 0, 2024-03-19T03:42:45.963 INFO:teuthology.orchestra.run.smithi178.stdout: "forward": 0, 2024-03-19T03:42:45.963 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_fetch_complete": 0, 2024-03-19T03:42:45.963 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_fetch_keys": 0, 2024-03-19T03:42:45.963 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_commit": 0, 2024-03-19T03:42:45.963 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_split": 0, 2024-03-19T03:42:45.963 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_merge": 0, 2024-03-19T03:42:45.963 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes": 904, 2024-03-19T03:42:45.963 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_top": 0, 2024-03-19T03:42:45.963 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_bottom": 904, 2024-03-19T03:42:45.963 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_pin_tail": 0, 2024-03-19T03:42:45.963 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_pinned": 904, 2024-03-19T03:42:45.963 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_expired": 0, 2024-03-19T03:42:45.964 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_with_caps": 0, 2024-03-19T03:42:45.964 INFO:teuthology.orchestra.run.smithi178.stdout: "caps": 0, 2024-03-19T03:42:45.964 INFO:teuthology.orchestra.run.smithi178.stdout: "subtrees": 1, 2024-03-19T03:42:45.964 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse": 0, 2024-03-19T03:42:45.964 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_hit": 0, 2024-03-19T03:42:45.964 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_forward": 0, 2024-03-19T03:42:45.964 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_discover": 0, 2024-03-19T03:42:45.964 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_dir_fetch": 0, 2024-03-19T03:42:45.964 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_remote_ino": 0, 2024-03-19T03:42:45.964 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_lock": 0, 2024-03-19T03:42:45.964 INFO:teuthology.orchestra.run.smithi178.stdout: "load_cent": 0, 2024-03-19T03:42:45.965 INFO:teuthology.orchestra.run.smithi178.stdout: "q": 0, 2024-03-19T03:42:45.965 INFO:teuthology.orchestra.run.smithi178.stdout: "exported": 0, 2024-03-19T03:42:45.965 INFO:teuthology.orchestra.run.smithi178.stdout: "exported_inodes": 0, 2024-03-19T03:42:45.965 INFO:teuthology.orchestra.run.smithi178.stdout: "imported": 0, 2024-03-19T03:42:45.965 INFO:teuthology.orchestra.run.smithi178.stdout: "imported_inodes": 0, 2024-03-19T03:42:45.965 INFO:teuthology.orchestra.run.smithi178.stdout: "openino_dir_fetch": 0, 2024-03-19T03:42:45.965 INFO:teuthology.orchestra.run.smithi178.stdout: "openino_backtrace_fetch": 0, 2024-03-19T03:42:45.965 INFO:teuthology.orchestra.run.smithi178.stdout: "openino_peer_discover": 0, 2024-03-19T03:42:45.965 INFO:teuthology.orchestra.run.smithi178.stdout: "root_rfiles": 0, 2024-03-19T03:42:45.965 INFO:teuthology.orchestra.run.smithi178.stdout: "root_rbytes": 0, 2024-03-19T03:42:45.965 INFO:teuthology.orchestra.run.smithi178.stdout: "root_rsnaps": 0, 2024-03-19T03:42:45.965 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T03:42:45.965 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_set_tag": 0, 2024-03-19T03:42:45.965 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T03:42:45.965 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_inotable_repaired": 0, 2024-03-19T03:42:45.966 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_dir_inodes": 0, 2024-03-19T03:42:45.966 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T03:42:45.966 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T03:42:45.966 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_file_inodes": 0, 2024-03-19T03:42:45.966 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_inode_file_caps": 0, 2024-03-19T03:42:45.966 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_revoke": 0, 2024-03-19T03:42:45.966 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_grant": 0, 2024-03-19T03:42:45.966 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T03:42:45.966 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T03:42:45.966 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T03:42:45.966 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_caps": 0, 2024-03-19T03:42:45.966 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_caps_dirty": 0, 2024-03-19T03:42:45.966 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_cap_release": 0, 2024-03-19T03:42:45.966 INFO:teuthology.orchestra.run.smithi178.stdout: "process_request_cap_release": 0 2024-03-19T03:42:45.967 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.967 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_cache": { 2024-03-19T03:42:45.967 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_update": 0, 2024-03-19T03:42:45.967 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_update_receipt": 0, 2024-03-19T03:42:45.967 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_try_discover": 0, 2024-03-19T03:42:45.967 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_send_discover": 0, 2024-03-19T03:42:45.967 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_handle_discover": 0, 2024-03-19T03:42:45.967 INFO:teuthology.orchestra.run.smithi178.stdout: "num_strays": 0, 2024-03-19T03:42:45.967 INFO:teuthology.orchestra.run.smithi178.stdout: "num_strays_delayed": 0, 2024-03-19T03:42:45.967 INFO:teuthology.orchestra.run.smithi178.stdout: "num_strays_enqueuing": 0, 2024-03-19T03:42:45.967 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_created": 0, 2024-03-19T03:42:45.967 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_enqueued": 0, 2024-03-19T03:42:45.967 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_reintegrated": 0, 2024-03-19T03:42:45.967 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_migrated": 0, 2024-03-19T03:42:45.968 INFO:teuthology.orchestra.run.smithi178.stdout: "num_recovering_processing": 0, 2024-03-19T03:42:45.968 INFO:teuthology.orchestra.run.smithi178.stdout: "num_recovering_enqueued": 0, 2024-03-19T03:42:45.968 INFO:teuthology.orchestra.run.smithi178.stdout: "num_recovering_prioritized": 0, 2024-03-19T03:42:45.968 INFO:teuthology.orchestra.run.smithi178.stdout: "recovery_started": 0, 2024-03-19T03:42:45.968 INFO:teuthology.orchestra.run.smithi178.stdout: "recovery_completed": 0, 2024-03-19T03:42:45.968 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_quiesce_path": 0, 2024-03-19T03:42:45.968 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_quiesce_inode": 0, 2024-03-19T03:42:45.968 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T03:42:45.968 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_exportdir": 0, 2024-03-19T03:42:45.968 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_flush": 0, 2024-03-19T03:42:45.968 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_fragmentdir": 0, 2024-03-19T03:42:45.968 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_fragstats": 0, 2024-03-19T03:42:45.968 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_inodestats": 0 2024-03-19T03:42:45.968 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.969 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_log": { 2024-03-19T03:42:45.969 INFO:teuthology.orchestra.run.smithi178.stdout: "evlrg": 0, 2024-03-19T03:42:45.969 INFO:teuthology.orchestra.run.smithi178.stdout: "evadd": 0, 2024-03-19T03:42:45.969 INFO:teuthology.orchestra.run.smithi178.stdout: "evex": 0, 2024-03-19T03:42:45.969 INFO:teuthology.orchestra.run.smithi178.stdout: "evtrm": 0, 2024-03-19T03:42:45.969 INFO:teuthology.orchestra.run.smithi178.stdout: "ev": 5287, 2024-03-19T03:42:45.969 INFO:teuthology.orchestra.run.smithi178.stdout: "evexg": 0, 2024-03-19T03:42:45.969 INFO:teuthology.orchestra.run.smithi178.stdout: "evexd": 0, 2024-03-19T03:42:45.969 INFO:teuthology.orchestra.run.smithi178.stdout: "segadd": 0, 2024-03-19T03:42:45.969 INFO:teuthology.orchestra.run.smithi178.stdout: "segex": 0, 2024-03-19T03:42:45.969 INFO:teuthology.orchestra.run.smithi178.stdout: "segtrm": 0, 2024-03-19T03:42:45.969 INFO:teuthology.orchestra.run.smithi178.stdout: "seg": 8, 2024-03-19T03:42:45.969 INFO:teuthology.orchestra.run.smithi178.stdout: "segmjr": 2, 2024-03-19T03:42:45.969 INFO:teuthology.orchestra.run.smithi178.stdout: "segexg": 0, 2024-03-19T03:42:45.969 INFO:teuthology.orchestra.run.smithi178.stdout: "segexd": 0, 2024-03-19T03:42:45.969 INFO:teuthology.orchestra.run.smithi178.stdout: "expos": 4194304, 2024-03-19T03:42:45.970 INFO:teuthology.orchestra.run.smithi178.stdout: "wrpos": 11643678, 2024-03-19T03:42:45.970 INFO:teuthology.orchestra.run.smithi178.stdout: "rdpos": 11642540, 2024-03-19T03:42:45.970 INFO:teuthology.orchestra.run.smithi178.stdout: "jlat": { 2024-03-19T03:42:45.970 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.970 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.970 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.970 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.970 INFO:teuthology.orchestra.run.smithi178.stdout: "replayed": 5287 2024-03-19T03:42:45.970 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.970 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_mem": { 2024-03-19T03:42:45.970 INFO:teuthology.orchestra.run.smithi178.stdout: "ino": 8, 2024-03-19T03:42:45.970 INFO:teuthology.orchestra.run.smithi178.stdout: "ino+": 925, 2024-03-19T03:42:45.970 INFO:teuthology.orchestra.run.smithi178.stdout: "ino-": 917, 2024-03-19T03:42:45.970 INFO:teuthology.orchestra.run.smithi178.stdout: "dir": 6, 2024-03-19T03:42:45.971 INFO:teuthology.orchestra.run.smithi178.stdout: "dir+": 49, 2024-03-19T03:42:45.971 INFO:teuthology.orchestra.run.smithi178.stdout: "dir-": 43, 2024-03-19T03:42:45.971 INFO:teuthology.orchestra.run.smithi178.stdout: "dn": 904, 2024-03-19T03:42:45.971 INFO:teuthology.orchestra.run.smithi178.stdout: "dn+": 2219, 2024-03-19T03:42:45.971 INFO:teuthology.orchestra.run.smithi178.stdout: "dn-": 1315, 2024-03-19T03:42:45.971 INFO:teuthology.orchestra.run.smithi178.stdout: "cap": 0, 2024-03-19T03:42:45.971 INFO:teuthology.orchestra.run.smithi178.stdout: "cap+": 0, 2024-03-19T03:42:45.971 INFO:teuthology.orchestra.run.smithi178.stdout: "cap-": 0, 2024-03-19T03:42:45.971 INFO:teuthology.orchestra.run.smithi178.stdout: "rss": 50744, 2024-03-19T03:42:45.971 INFO:teuthology.orchestra.run.smithi178.stdout: "heap": 215420 2024-03-19T03:42:45.971 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.971 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_server": { 2024-03-19T03:42:45.971 INFO:teuthology.orchestra.run.smithi178.stdout: "dispatch_client_request": 0, 2024-03-19T03:42:45.971 INFO:teuthology.orchestra.run.smithi178.stdout: "dispatch_server_request": 0, 2024-03-19T03:42:45.971 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_request": 0, 2024-03-19T03:42:45.972 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_session": 0, 2024-03-19T03:42:45.972 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_peer_request": 0, 2024-03-19T03:42:45.972 INFO:teuthology.orchestra.run.smithi178.stdout: "req_create_latency": { 2024-03-19T03:42:45.972 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.972 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.972 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.972 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.972 INFO:teuthology.orchestra.run.smithi178.stdout: "req_getattr_latency": { 2024-03-19T03:42:45.972 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.972 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.972 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.972 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.972 INFO:teuthology.orchestra.run.smithi178.stdout: "req_getfilelock_latency": { 2024-03-19T03:42:45.972 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.972 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.973 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.973 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.973 INFO:teuthology.orchestra.run.smithi178.stdout: "req_link_latency": { 2024-03-19T03:42:45.973 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.973 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.973 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.973 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.973 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookup_latency": { 2024-03-19T03:42:45.973 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.973 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.973 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.973 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.973 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookuphash_latency": { 2024-03-19T03:42:45.973 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.973 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.974 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.974 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.974 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupino_latency": { 2024-03-19T03:42:45.974 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.974 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.974 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.974 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.974 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupname_latency": { 2024-03-19T03:42:45.974 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.974 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.974 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.974 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.974 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupparent_latency": { 2024-03-19T03:42:45.974 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.974 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.975 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.975 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.975 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupsnap_latency": { 2024-03-19T03:42:45.975 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.975 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.975 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.975 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.975 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lssnap_latency": { 2024-03-19T03:42:45.975 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.975 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.975 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.975 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.975 INFO:teuthology.orchestra.run.smithi178.stdout: "req_mkdir_latency": { 2024-03-19T03:42:45.976 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.976 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.976 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.976 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.976 INFO:teuthology.orchestra.run.smithi178.stdout: "req_mknod_latency": { 2024-03-19T03:42:45.976 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.976 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.976 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.976 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.977 INFO:teuthology.orchestra.run.smithi178.stdout: "req_mksnap_latency": { 2024-03-19T03:42:45.977 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.977 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.977 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.977 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.977 INFO:teuthology.orchestra.run.smithi178.stdout: "req_open_latency": { 2024-03-19T03:42:45.977 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.977 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.977 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.977 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.977 INFO:teuthology.orchestra.run.smithi178.stdout: "req_readdir_latency": { 2024-03-19T03:42:45.977 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.977 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.977 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.977 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.978 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rename_latency": { 2024-03-19T03:42:45.978 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.978 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.978 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.978 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.978 INFO:teuthology.orchestra.run.smithi178.stdout: "req_renamesnap_latency": { 2024-03-19T03:42:45.978 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.978 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.978 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.978 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.978 INFO:teuthology.orchestra.run.smithi178.stdout: "req_snapdiff_latency": { 2024-03-19T03:42:45.978 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.978 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.978 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.979 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.979 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rmdir_latency": { 2024-03-19T03:42:45.979 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.979 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.979 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.979 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.979 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rmsnap_latency": { 2024-03-19T03:42:45.979 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.979 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.979 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.979 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.979 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rmxattr_latency": { 2024-03-19T03:42:45.979 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.979 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.979 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.979 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.980 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setattr_latency": { 2024-03-19T03:42:45.980 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.980 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.980 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.980 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.980 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setdirlayout_latency": { 2024-03-19T03:42:45.980 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.980 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.980 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.980 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.980 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setfilelock_latency": { 2024-03-19T03:42:45.980 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.980 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.980 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.980 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.981 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setlayout_latency": { 2024-03-19T03:42:45.981 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.981 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.981 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.981 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.981 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setxattr_latency": { 2024-03-19T03:42:45.981 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.981 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.981 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.981 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.981 INFO:teuthology.orchestra.run.smithi178.stdout: "req_symlink_latency": { 2024-03-19T03:42:45.981 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.981 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.981 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.982 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.982 INFO:teuthology.orchestra.run.smithi178.stdout: "req_unlink_latency": { 2024-03-19T03:42:45.982 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.982 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.982 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.982 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.982 INFO:teuthology.orchestra.run.smithi178.stdout: "cap_revoke_eviction": 0, 2024-03-19T03:42:45.982 INFO:teuthology.orchestra.run.smithi178.stdout: "cap_acquisition_throttle": 0, 2024-03-19T03:42:45.982 INFO:teuthology.orchestra.run.smithi178.stdout: "req_getvxattr_latency": { 2024-03-19T03:42:45.982 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.982 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.982 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.982 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:45.982 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.983 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_sessions": { 2024-03-19T03:42:45.983 INFO:teuthology.orchestra.run.smithi178.stdout: "session_count": 1, 2024-03-19T03:42:45.983 INFO:teuthology.orchestra.run.smithi178.stdout: "session_add": 2, 2024-03-19T03:42:45.983 INFO:teuthology.orchestra.run.smithi178.stdout: "session_remove": 1, 2024-03-19T03:42:45.983 INFO:teuthology.orchestra.run.smithi178.stdout: "sessions_open": 2, 2024-03-19T03:42:45.983 INFO:teuthology.orchestra.run.smithi178.stdout: "sessions_stale": 0, 2024-03-19T03:42:45.983 INFO:teuthology.orchestra.run.smithi178.stdout: "total_load": 0, 2024-03-19T03:42:45.983 INFO:teuthology.orchestra.run.smithi178.stdout: "average_load": 0, 2024-03-19T03:42:45.983 INFO:teuthology.orchestra.run.smithi178.stdout: "avg_session_uptime": 6870, 2024-03-19T03:42:45.983 INFO:teuthology.orchestra.run.smithi178.stdout: "mdthresh_evicted": 0 2024-03-19T03:42:45.983 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.983 INFO:teuthology.orchestra.run.smithi178.stdout: "mempool": { 2024-03-19T03:42:45.983 INFO:teuthology.orchestra.run.smithi178.stdout: "bloom_filter_bytes": 0, 2024-03-19T03:42:45.983 INFO:teuthology.orchestra.run.smithi178.stdout: "bloom_filter_items": 0, 2024-03-19T03:42:45.983 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T03:42:45.984 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_alloc_items": 0, 2024-03-19T03:42:45.984 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T03:42:45.984 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_data_items": 0, 2024-03-19T03:42:45.984 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T03:42:45.984 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T03:42:45.984 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T03:42:45.984 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T03:42:45.984 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T03:42:45.984 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_other_items": 0, 2024-03-19T03:42:45.984 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T03:42:45.984 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T03:42:45.984 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_extent_bytes": 0, 2024-03-19T03:42:45.984 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_extent_items": 0, 2024-03-19T03:42:45.984 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_blob_bytes": 0, 2024-03-19T03:42:45.985 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_blob_items": 0, 2024-03-19T03:42:45.985 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T03:42:45.985 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T03:42:45.985 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T03:42:45.985 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T03:42:45.985 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T03:42:45.985 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_items": 0, 2024-03-19T03:42:45.985 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_txc_bytes": 0, 2024-03-19T03:42:45.985 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_txc_items": 0, 2024-03-19T03:42:45.985 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T03:42:45.985 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T03:42:45.985 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_bytes": 0, 2024-03-19T03:42:45.985 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_items": 0, 2024-03-19T03:42:45.985 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_bytes": 0, 2024-03-19T03:42:45.985 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_items": 0, 2024-03-19T03:42:45.986 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T03:42:45.986 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_reader_items": 0, 2024-03-19T03:42:45.986 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T03:42:45.986 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_writer_items": 0, 2024-03-19T03:42:45.986 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_anon_bytes": 198242, 2024-03-19T03:42:45.986 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_anon_items": 57, 2024-03-19T03:42:45.986 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_meta_bytes": 0, 2024-03-19T03:42:45.986 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_meta_items": 0, 2024-03-19T03:42:45.986 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_bytes": 0, 2024-03-19T03:42:45.986 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_items": 0, 2024-03-19T03:42:45.986 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mapbl_bytes": 0, 2024-03-19T03:42:45.986 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mapbl_items": 0, 2024-03-19T03:42:45.986 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pglog_bytes": 0, 2024-03-19T03:42:45.986 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pglog_items": 0, 2024-03-19T03:42:45.987 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_bytes": 9184, 2024-03-19T03:42:45.987 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_items": 177, 2024-03-19T03:42:45.987 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T03:42:45.987 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_mapping_items": 0, 2024-03-19T03:42:45.987 INFO:teuthology.orchestra.run.smithi178.stdout: "pgmap_bytes": 0, 2024-03-19T03:42:45.987 INFO:teuthology.orchestra.run.smithi178.stdout: "pgmap_items": 0, 2024-03-19T03:42:45.987 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_co_bytes": 662360, 2024-03-19T03:42:45.987 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_co_items": 3689, 2024-03-19T03:42:45.987 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_1_bytes": 0, 2024-03-19T03:42:45.987 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_1_items": 0, 2024-03-19T03:42:45.987 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_2_bytes": 0, 2024-03-19T03:42:45.987 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_2_items": 0 2024-03-19T03:42:45.987 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.987 INFO:teuthology.orchestra.run.smithi178.stdout: "objecter": { 2024-03-19T03:42:45.987 INFO:teuthology.orchestra.run.smithi178.stdout: "op_active": 0, 2024-03-19T03:42:45.988 INFO:teuthology.orchestra.run.smithi178.stdout: "op_laggy": 0, 2024-03-19T03:42:45.988 INFO:teuthology.orchestra.run.smithi178.stdout: "op_send": 2035, 2024-03-19T03:42:45.988 INFO:teuthology.orchestra.run.smithi178.stdout: "op_send_bytes": 21, 2024-03-19T03:42:45.988 INFO:teuthology.orchestra.run.smithi178.stdout: "op_resend": 3, 2024-03-19T03:42:45.988 INFO:teuthology.orchestra.run.smithi178.stdout: "op_reply": 2035, 2024-03-19T03:42:45.988 INFO:teuthology.orchestra.run.smithi178.stdout: "op_latency": { 2024-03-19T03:42:45.988 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 2035, 2024-03-19T03:42:45.988 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 11.753768655, 2024-03-19T03:42:45.988 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.005775807 2024-03-19T03:42:45.988 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.988 INFO:teuthology.orchestra.run.smithi178.stdout: "op_inflight": 0, 2024-03-19T03:42:45.988 INFO:teuthology.orchestra.run.smithi178.stdout: "oplen_avg": { 2024-03-19T03:42:45.988 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 2035, 2024-03-19T03:42:45.988 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 2036 2024-03-19T03:42:45.988 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.989 INFO:teuthology.orchestra.run.smithi178.stdout: "op": 2035, 2024-03-19T03:42:45.989 INFO:teuthology.orchestra.run.smithi178.stdout: "op_r": 2035, 2024-03-19T03:42:45.989 INFO:teuthology.orchestra.run.smithi178.stdout: "op_w": 0, 2024-03-19T03:42:45.989 INFO:teuthology.orchestra.run.smithi178.stdout: "op_rmw": 0, 2024-03-19T03:42:45.989 INFO:teuthology.orchestra.run.smithi178.stdout: "op_pg": 0, 2024-03-19T03:42:45.989 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_stat": 1227, 2024-03-19T03:42:45.989 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_create": 0, 2024-03-19T03:42:45.989 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_read": 806, 2024-03-19T03:42:45.989 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_write": 0, 2024-03-19T03:42:45.989 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_writefull": 0, 2024-03-19T03:42:45.989 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_writesame": 0, 2024-03-19T03:42:45.989 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_append": 0, 2024-03-19T03:42:45.989 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_zero": 0, 2024-03-19T03:42:45.989 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_truncate": 0, 2024-03-19T03:42:45.989 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_delete": 0, 2024-03-19T03:42:45.990 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_mapext": 0, 2024-03-19T03:42:45.990 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_sparse_read": 0, 2024-03-19T03:42:45.990 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_clonerange": 0, 2024-03-19T03:42:45.990 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_getxattr": 1, 2024-03-19T03:42:45.990 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_setxattr": 0, 2024-03-19T03:42:45.990 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_cmpxattr": 0, 2024-03-19T03:42:45.990 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_rmxattr": 0, 2024-03-19T03:42:45.990 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_resetxattrs": 0, 2024-03-19T03:42:45.990 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_call": 0, 2024-03-19T03:42:45.990 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_watch": 0, 2024-03-19T03:42:45.990 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_notify": 0, 2024-03-19T03:42:45.990 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T03:42:45.990 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_pgls": 0, 2024-03-19T03:42:45.990 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_pgls_filter": 0, 2024-03-19T03:42:45.991 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_other": 0, 2024-03-19T03:42:45.991 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_active": 0, 2024-03-19T03:42:45.991 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_send": 0, 2024-03-19T03:42:45.991 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_resend": 0, 2024-03-19T03:42:45.991 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_ping": 0, 2024-03-19T03:42:45.991 INFO:teuthology.orchestra.run.smithi178.stdout: "poolop_active": 0, 2024-03-19T03:42:45.991 INFO:teuthology.orchestra.run.smithi178.stdout: "poolop_send": 0, 2024-03-19T03:42:45.991 INFO:teuthology.orchestra.run.smithi178.stdout: "poolop_resend": 0, 2024-03-19T03:42:45.991 INFO:teuthology.orchestra.run.smithi178.stdout: "poolstat_active": 0, 2024-03-19T03:42:45.991 INFO:teuthology.orchestra.run.smithi178.stdout: "poolstat_send": 0, 2024-03-19T03:42:45.991 INFO:teuthology.orchestra.run.smithi178.stdout: "poolstat_resend": 0, 2024-03-19T03:42:45.991 INFO:teuthology.orchestra.run.smithi178.stdout: "statfs_active": 0, 2024-03-19T03:42:45.991 INFO:teuthology.orchestra.run.smithi178.stdout: "statfs_send": 0, 2024-03-19T03:42:45.991 INFO:teuthology.orchestra.run.smithi178.stdout: "statfs_resend": 0, 2024-03-19T03:42:45.991 INFO:teuthology.orchestra.run.smithi178.stdout: "command_active": 0, 2024-03-19T03:42:45.992 INFO:teuthology.orchestra.run.smithi178.stdout: "command_send": 0, 2024-03-19T03:42:45.992 INFO:teuthology.orchestra.run.smithi178.stdout: "command_resend": 0, 2024-03-19T03:42:45.992 INFO:teuthology.orchestra.run.smithi178.stdout: "map_epoch": 93, 2024-03-19T03:42:45.992 INFO:teuthology.orchestra.run.smithi178.stdout: "map_full": 0, 2024-03-19T03:42:45.992 INFO:teuthology.orchestra.run.smithi178.stdout: "map_inc": 4, 2024-03-19T03:42:45.992 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_sessions": 6, 2024-03-19T03:42:45.992 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_session_open": 6, 2024-03-19T03:42:45.992 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_session_close": 0, 2024-03-19T03:42:45.992 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_laggy": 0, 2024-03-19T03:42:45.992 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_wr": 0, 2024-03-19T03:42:45.992 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_rd": 2, 2024-03-19T03:42:45.992 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_del": 0 2024-03-19T03:42:45.992 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.992 INFO:teuthology.orchestra.run.smithi178.stdout: "oft": { 2024-03-19T03:42:45.993 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_objs": 0, 2024-03-19T03:42:45.993 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_kv_pairs": 0, 2024-03-19T03:42:45.993 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_updates": 0, 2024-03-19T03:42:45.993 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_removes": 0 2024-03-19T03:42:45.993 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.993 INFO:teuthology.orchestra.run.smithi178.stdout: "osd-slow-ops": { 2024-03-19T03:42:45.993 INFO:teuthology.orchestra.run.smithi178.stdout: "slow_ops_count": 0 2024-03-19T03:42:45.993 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.993 INFO:teuthology.orchestra.run.smithi178.stdout: "purge_queue": { 2024-03-19T03:42:45.993 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing_ops": 0, 2024-03-19T03:42:45.993 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing_ops_high_water": 0, 2024-03-19T03:42:45.993 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing": 0, 2024-03-19T03:42:45.993 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing_high_water": 0, 2024-03-19T03:42:45.993 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executed_ops": 0, 2024-03-19T03:42:45.993 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executed": 0, 2024-03-19T03:42:45.993 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_item_in_journal": 0 2024-03-19T03:42:45.994 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.994 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T03:42:45.994 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-19T03:42:45.994 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 104857600, 2024-03-19T03:42:45.994 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-19T03:42:45.994 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 2275, 2024-03-19T03:42:45.994 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 8706658, 2024-03-19T03:42:45.994 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:45.994 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 2275, 2024-03-19T03:42:45.994 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 0, 2024-03-19T03:42:45.994 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 0, 2024-03-19T03:42:45.994 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 2275, 2024-03-19T03:42:45.994 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 8706658, 2024-03-19T03:42:45.994 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-19T03:42:45.995 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.995 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.995 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.995 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:45.995 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.995 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-objecter_bytes": { 2024-03-19T03:42:45.995 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-19T03:42:45.995 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 104857600, 2024-03-19T03:42:45.995 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-19T03:42:45.995 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-19T03:42:45.995 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-19T03:42:45.995 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:45.997 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:45.997 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 2035, 2024-03-19T03:42:45.997 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 7449854, 2024-03-19T03:42:45.997 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 2035, 2024-03-19T03:42:45.997 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 7449854, 2024-03-19T03:42:45.997 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-19T03:42:45.998 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.998 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.998 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.998 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:45.998 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.998 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-objecter_ops": { 2024-03-19T03:42:45.998 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-19T03:42:45.998 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 1024, 2024-03-19T03:42:45.998 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-19T03:42:45.998 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-19T03:42:45.998 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-19T03:42:45.998 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:45.998 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:45.998 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 2035, 2024-03-19T03:42:45.999 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 2035, 2024-03-19T03:42:45.999 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 2035, 2024-03-19T03:42:45.999 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 2035, 2024-03-19T03:42:45.999 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-19T03:42:45.999 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:45.999 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:45.999 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:45.999 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:45.999 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:45.999 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-write_buf_throttle": { 2024-03-19T03:42:45.999 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-19T03:42:45.999 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 3758096384, 2024-03-19T03:42:45.999 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-19T03:42:45.999 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-19T03:42:45.999 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-19T03:42:46.000 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:46.000 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:46.000 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 0, 2024-03-19T03:42:46.000 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 0, 2024-03-19T03:42:46.000 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 0, 2024-03-19T03:42:46.000 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 0, 2024-03-19T03:42:46.000 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-19T03:42:46.000 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:46.000 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:46.000 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:46.000 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:46.000 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:46.000 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-write_buf_throttle-0x5625ac1e46e0": { 2024-03-19T03:42:46.000 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-19T03:42:46.000 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 3758096384, 2024-03-19T03:42:46.001 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-19T03:42:46.001 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-19T03:42:46.001 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-19T03:42:46.001 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:46.001 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:46.001 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 0, 2024-03-19T03:42:46.001 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 0, 2024-03-19T03:42:46.001 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 0, 2024-03-19T03:42:46.001 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 0, 2024-03-19T03:42:46.001 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-19T03:42:46.001 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:46.001 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:46.001 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:46.001 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:46.001 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:46.001 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-03-19T03:42:46.725 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-19T03:42:46.726 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-19T03:42:46.726 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-03-19T03:42:46.726 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-03-19T03:42:46.726 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-03-19T03:42:46.726 DEBUG:tasks.check_counter:Getting stats from i 2024-03-19T03:42:46.726 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-19T03:42:46.845 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:46 smithi178 ceph-mon[67578]: pgmap v687: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T03:42:46.964 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T03:42:46.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:46 smithi119 ceph-mon[66861]: pgmap v687: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T03:42:47.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:46 smithi117 ceph-mon[31838]: pgmap v687: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T03:42:48.608 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-03-19T03:42:48.608 INFO:teuthology.orchestra.run.smithi178.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T03:42:48.608 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_messages": 1627, 2024-03-19T03:42:48.608 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages": 1626, 2024-03-19T03:42:48.609 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_bytes": 206040317, 2024-03-19T03:42:48.609 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_bytes": 516254, 2024-03-19T03:42:48.609 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_created_connections": 8, 2024-03-19T03:42:48.609 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_active_connections": 4, 2024-03-19T03:42:48.609 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_total_time": 0.236492754, 2024-03-19T03:42:48.609 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_send_time": 0.043529215, 2024-03-19T03:42:48.609 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_recv_time": 0.133957334, 2024-03-19T03:42:48.609 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_fast_dispatch_time": 0.045354090, 2024-03-19T03:42:48.609 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:48.609 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 1625, 2024-03-19T03:42:48.609 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.057415379, 2024-03-19T03:42:48.609 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000035332 2024-03-19T03:42:48.609 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.610 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:48.610 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.610 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.610 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.610 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.610 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-19T03:42:48.611 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_encrypted_bytes": 448 2024-03-19T03:42:48.611 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.611 INFO:teuthology.orchestra.run.smithi178.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T03:42:48.611 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_messages": 668, 2024-03-19T03:42:48.611 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages": 821, 2024-03-19T03:42:48.611 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_bytes": 172167723, 2024-03-19T03:42:48.611 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_bytes": 552588, 2024-03-19T03:42:48.611 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_created_connections": 7, 2024-03-19T03:42:48.612 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_active_connections": 6, 2024-03-19T03:42:48.612 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_total_time": 0.142952439, 2024-03-19T03:42:48.612 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_send_time": 0.020499113, 2024-03-19T03:42:48.612 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_recv_time": 0.100014618, 2024-03-19T03:42:48.612 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_fast_dispatch_time": 0.016108608, 2024-03-19T03:42:48.612 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:48.612 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 819, 2024-03-19T03:42:48.612 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.031096932, 2024-03-19T03:42:48.613 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000037969 2024-03-19T03:42:48.613 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.613 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:48.613 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.613 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.613 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.613 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.613 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-19T03:42:48.613 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_encrypted_bytes": 340416 2024-03-19T03:42:48.614 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.614 INFO:teuthology.orchestra.run.smithi178.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T03:42:48.614 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_messages": 1003, 2024-03-19T03:42:48.614 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages": 978, 2024-03-19T03:42:48.614 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_bytes": 145710275, 2024-03-19T03:42:48.614 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_bytes": 369349, 2024-03-19T03:42:48.615 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_created_connections": 5, 2024-03-19T03:42:48.615 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_active_connections": 4, 2024-03-19T03:42:48.615 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_total_time": 0.159936242, 2024-03-19T03:42:48.615 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_send_time": 0.032576614, 2024-03-19T03:42:48.615 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_recv_time": 0.098930983, 2024-03-19T03:42:48.615 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_fast_dispatch_time": 0.024749763, 2024-03-19T03:42:48.615 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:48.616 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 977, 2024-03-19T03:42:48.616 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.035654173, 2024-03-19T03:42:48.616 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000036493 2024-03-19T03:42:48.616 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.616 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:48.616 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.617 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.617 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.617 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.617 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_encrypted_bytes": 898960, 2024-03-19T03:42:48.617 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_encrypted_bytes": 125760 2024-03-19T03:42:48.617 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.617 INFO:teuthology.orchestra.run.smithi178.stdout: "cct": { 2024-03-19T03:42:48.617 INFO:teuthology.orchestra.run.smithi178.stdout: "total_workers": 1, 2024-03-19T03:42:48.618 INFO:teuthology.orchestra.run.smithi178.stdout: "unhealthy_workers": 0 2024-03-19T03:42:48.618 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.618 INFO:teuthology.orchestra.run.smithi178.stdout: "finisher-MDSRank": { 2024-03-19T03:42:48.618 INFO:teuthology.orchestra.run.smithi178.stdout: "queue_len": 0, 2024-03-19T03:42:48.618 INFO:teuthology.orchestra.run.smithi178.stdout: "complete_latency": { 2024-03-19T03:42:48.618 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 4754, 2024-03-19T03:42:48.618 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.228537816, 2024-03-19T03:42:48.619 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000048072 2024-03-19T03:42:48.619 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:48.619 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.619 INFO:teuthology.orchestra.run.smithi178.stdout: "finisher-PurgeQueue": { 2024-03-19T03:42:48.619 INFO:teuthology.orchestra.run.smithi178.stdout: "queue_len": 0, 2024-03-19T03:42:48.619 INFO:teuthology.orchestra.run.smithi178.stdout: "complete_latency": { 2024-03-19T03:42:48.619 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.620 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.620 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.620 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:48.620 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.620 INFO:teuthology.orchestra.run.smithi178.stdout: "mds": { 2024-03-19T03:42:48.620 INFO:teuthology.orchestra.run.smithi178.stdout: "request": 0, 2024-03-19T03:42:48.621 INFO:teuthology.orchestra.run.smithi178.stdout: "reply": 0, 2024-03-19T03:42:48.621 INFO:teuthology.orchestra.run.smithi178.stdout: "reply_latency": { 2024-03-19T03:42:48.621 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.621 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.621 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.621 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.621 INFO:teuthology.orchestra.run.smithi178.stdout: "slow_reply": 0, 2024-03-19T03:42:48.621 INFO:teuthology.orchestra.run.smithi178.stdout: "forward": 0, 2024-03-19T03:42:48.621 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_fetch_complete": 0, 2024-03-19T03:42:48.622 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_fetch_keys": 0, 2024-03-19T03:42:48.622 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_commit": 0, 2024-03-19T03:42:48.622 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_split": 0, 2024-03-19T03:42:48.622 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_merge": 0, 2024-03-19T03:42:48.622 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes": 43038, 2024-03-19T03:42:48.622 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_top": 9303, 2024-03-19T03:42:48.622 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_bottom": 33735, 2024-03-19T03:42:48.622 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_pin_tail": 0, 2024-03-19T03:42:48.622 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_pinned": 29747, 2024-03-19T03:42:48.622 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_expired": 0, 2024-03-19T03:42:48.622 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_with_caps": 0, 2024-03-19T03:42:48.622 INFO:teuthology.orchestra.run.smithi178.stdout: "caps": 0, 2024-03-19T03:42:48.623 INFO:teuthology.orchestra.run.smithi178.stdout: "subtrees": 2, 2024-03-19T03:42:48.623 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse": 0, 2024-03-19T03:42:48.623 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_hit": 0, 2024-03-19T03:42:48.623 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_forward": 0, 2024-03-19T03:42:48.623 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_discover": 0, 2024-03-19T03:42:48.623 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_dir_fetch": 0, 2024-03-19T03:42:48.623 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_remote_ino": 0, 2024-03-19T03:42:48.623 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_lock": 0, 2024-03-19T03:42:48.623 INFO:teuthology.orchestra.run.smithi178.stdout: "load_cent": 0, 2024-03-19T03:42:48.623 INFO:teuthology.orchestra.run.smithi178.stdout: "q": 0, 2024-03-19T03:42:48.623 INFO:teuthology.orchestra.run.smithi178.stdout: "exported": 0, 2024-03-19T03:42:48.623 INFO:teuthology.orchestra.run.smithi178.stdout: "exported_inodes": 0, 2024-03-19T03:42:48.623 INFO:teuthology.orchestra.run.smithi178.stdout: "imported": 0, 2024-03-19T03:42:48.623 INFO:teuthology.orchestra.run.smithi178.stdout: "imported_inodes": 0, 2024-03-19T03:42:48.623 INFO:teuthology.orchestra.run.smithi178.stdout: "openino_dir_fetch": 0, 2024-03-19T03:42:48.624 INFO:teuthology.orchestra.run.smithi178.stdout: "openino_backtrace_fetch": 0, 2024-03-19T03:42:48.624 INFO:teuthology.orchestra.run.smithi178.stdout: "openino_peer_discover": 0, 2024-03-19T03:42:48.624 INFO:teuthology.orchestra.run.smithi178.stdout: "root_rfiles": 4, 2024-03-19T03:42:48.624 INFO:teuthology.orchestra.run.smithi178.stdout: "root_rbytes": 234, 2024-03-19T03:42:48.624 INFO:teuthology.orchestra.run.smithi178.stdout: "root_rsnaps": 0, 2024-03-19T03:42:48.624 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T03:42:48.624 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_set_tag": 0, 2024-03-19T03:42:48.624 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T03:42:48.624 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_inotable_repaired": 0, 2024-03-19T03:42:48.624 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_dir_inodes": 0, 2024-03-19T03:42:48.624 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T03:42:48.624 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T03:42:48.624 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_file_inodes": 0, 2024-03-19T03:42:48.624 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_inode_file_caps": 0, 2024-03-19T03:42:48.624 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_revoke": 0, 2024-03-19T03:42:48.625 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_grant": 0, 2024-03-19T03:42:48.625 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T03:42:48.625 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T03:42:48.625 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T03:42:48.625 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_caps": 0, 2024-03-19T03:42:48.625 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_caps_dirty": 0, 2024-03-19T03:42:48.625 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_cap_release": 0, 2024-03-19T03:42:48.625 INFO:teuthology.orchestra.run.smithi178.stdout: "process_request_cap_release": 0 2024-03-19T03:42:48.625 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.625 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_cache": { 2024-03-19T03:42:48.625 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_update": 0, 2024-03-19T03:42:48.625 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_update_receipt": 0, 2024-03-19T03:42:48.625 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_try_discover": 0, 2024-03-19T03:42:48.625 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_send_discover": 0, 2024-03-19T03:42:48.626 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_handle_discover": 0, 2024-03-19T03:42:48.626 INFO:teuthology.orchestra.run.smithi178.stdout: "num_strays": 0, 2024-03-19T03:42:48.626 INFO:teuthology.orchestra.run.smithi178.stdout: "num_strays_delayed": 0, 2024-03-19T03:42:48.626 INFO:teuthology.orchestra.run.smithi178.stdout: "num_strays_enqueuing": 0, 2024-03-19T03:42:48.626 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_created": 0, 2024-03-19T03:42:48.626 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_enqueued": 0, 2024-03-19T03:42:48.626 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_reintegrated": 0, 2024-03-19T03:42:48.626 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_migrated": 0, 2024-03-19T03:42:48.626 INFO:teuthology.orchestra.run.smithi178.stdout: "num_recovering_processing": 0, 2024-03-19T03:42:48.626 INFO:teuthology.orchestra.run.smithi178.stdout: "num_recovering_enqueued": 0, 2024-03-19T03:42:48.626 INFO:teuthology.orchestra.run.smithi178.stdout: "num_recovering_prioritized": 0, 2024-03-19T03:42:48.626 INFO:teuthology.orchestra.run.smithi178.stdout: "recovery_started": 0, 2024-03-19T03:42:48.626 INFO:teuthology.orchestra.run.smithi178.stdout: "recovery_completed": 0, 2024-03-19T03:42:48.626 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_quiesce_path": 0, 2024-03-19T03:42:48.627 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_quiesce_inode": 0, 2024-03-19T03:42:48.627 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T03:42:48.627 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_exportdir": 0, 2024-03-19T03:42:48.627 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_flush": 0, 2024-03-19T03:42:48.627 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_fragmentdir": 0, 2024-03-19T03:42:48.627 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_fragstats": 0, 2024-03-19T03:42:48.627 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_inodestats": 0 2024-03-19T03:42:48.627 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.627 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_log": { 2024-03-19T03:42:48.627 INFO:teuthology.orchestra.run.smithi178.stdout: "evlrg": 0, 2024-03-19T03:42:48.627 INFO:teuthology.orchestra.run.smithi178.stdout: "evadd": 0, 2024-03-19T03:42:48.627 INFO:teuthology.orchestra.run.smithi178.stdout: "evex": 0, 2024-03-19T03:42:48.627 INFO:teuthology.orchestra.run.smithi178.stdout: "evtrm": 0, 2024-03-19T03:42:48.627 INFO:teuthology.orchestra.run.smithi178.stdout: "ev": 232023, 2024-03-19T03:42:48.627 INFO:teuthology.orchestra.run.smithi178.stdout: "evexg": 0, 2024-03-19T03:42:48.628 INFO:teuthology.orchestra.run.smithi178.stdout: "evexd": 0, 2024-03-19T03:42:48.628 INFO:teuthology.orchestra.run.smithi178.stdout: "segadd": 0, 2024-03-19T03:42:48.628 INFO:teuthology.orchestra.run.smithi178.stdout: "segex": 0, 2024-03-19T03:42:48.628 INFO:teuthology.orchestra.run.smithi178.stdout: "segtrm": 0, 2024-03-19T03:42:48.628 INFO:teuthology.orchestra.run.smithi178.stdout: "seg": 141, 2024-03-19T03:42:48.628 INFO:teuthology.orchestra.run.smithi178.stdout: "segmjr": 20, 2024-03-19T03:42:48.628 INFO:teuthology.orchestra.run.smithi178.stdout: "segexg": 0, 2024-03-19T03:42:48.628 INFO:teuthology.orchestra.run.smithi178.stdout: "segexd": 0, 2024-03-19T03:42:48.628 INFO:teuthology.orchestra.run.smithi178.stdout: "expos": 305387685, 2024-03-19T03:42:48.628 INFO:teuthology.orchestra.run.smithi178.stdout: "wrpos": 526287347, 2024-03-19T03:42:48.628 INFO:teuthology.orchestra.run.smithi178.stdout: "rdpos": 526284850, 2024-03-19T03:42:48.628 INFO:teuthology.orchestra.run.smithi178.stdout: "jlat": { 2024-03-19T03:42:48.628 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.628 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.629 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.629 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.629 INFO:teuthology.orchestra.run.smithi178.stdout: "replayed": 232023 2024-03-19T03:42:48.629 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.629 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_mem": { 2024-03-19T03:42:48.629 INFO:teuthology.orchestra.run.smithi178.stdout: "ino": 28, 2024-03-19T03:42:48.629 INFO:teuthology.orchestra.run.smithi178.stdout: "ino+": 43631, 2024-03-19T03:42:48.629 INFO:teuthology.orchestra.run.smithi178.stdout: "ino-": 43603, 2024-03-19T03:42:48.629 INFO:teuthology.orchestra.run.smithi178.stdout: "dir": 33, 2024-03-19T03:42:48.629 INFO:teuthology.orchestra.run.smithi178.stdout: "dir+": 1387, 2024-03-19T03:42:48.629 INFO:teuthology.orchestra.run.smithi178.stdout: "dir-": 1354, 2024-03-19T03:42:48.629 INFO:teuthology.orchestra.run.smithi178.stdout: "dn": 43038, 2024-03-19T03:42:48.629 INFO:teuthology.orchestra.run.smithi178.stdout: "dn+": 95439, 2024-03-19T03:42:48.629 INFO:teuthology.orchestra.run.smithi178.stdout: "dn-": 52401, 2024-03-19T03:42:48.629 INFO:teuthology.orchestra.run.smithi178.stdout: "cap": 0, 2024-03-19T03:42:48.630 INFO:teuthology.orchestra.run.smithi178.stdout: "cap+": 0, 2024-03-19T03:42:48.630 INFO:teuthology.orchestra.run.smithi178.stdout: "cap-": 0, 2024-03-19T03:42:48.630 INFO:teuthology.orchestra.run.smithi178.stdout: "rss": 121876, 2024-03-19T03:42:48.630 INFO:teuthology.orchestra.run.smithi178.stdout: "heap": 215420 2024-03-19T03:42:48.630 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.630 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_server": { 2024-03-19T03:42:48.630 INFO:teuthology.orchestra.run.smithi178.stdout: "dispatch_client_request": 0, 2024-03-19T03:42:48.630 INFO:teuthology.orchestra.run.smithi178.stdout: "dispatch_server_request": 0, 2024-03-19T03:42:48.630 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_request": 0, 2024-03-19T03:42:48.630 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_session": 0, 2024-03-19T03:42:48.630 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_peer_request": 0, 2024-03-19T03:42:48.630 INFO:teuthology.orchestra.run.smithi178.stdout: "req_create_latency": { 2024-03-19T03:42:48.630 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.630 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.631 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.631 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.631 INFO:teuthology.orchestra.run.smithi178.stdout: "req_getattr_latency": { 2024-03-19T03:42:48.631 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.631 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.631 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.631 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.631 INFO:teuthology.orchestra.run.smithi178.stdout: "req_getfilelock_latency": { 2024-03-19T03:42:48.631 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.631 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.631 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.631 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.631 INFO:teuthology.orchestra.run.smithi178.stdout: "req_link_latency": { 2024-03-19T03:42:48.631 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.632 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.632 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.632 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.632 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookup_latency": { 2024-03-19T03:42:48.632 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.632 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.632 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.632 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.632 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookuphash_latency": { 2024-03-19T03:42:48.632 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.632 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.632 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.632 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.632 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupino_latency": { 2024-03-19T03:42:48.632 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.632 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.633 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.633 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.633 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupname_latency": { 2024-03-19T03:42:48.633 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.633 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.633 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.633 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.633 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupparent_latency": { 2024-03-19T03:42:48.633 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.633 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.633 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.633 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.633 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupsnap_latency": { 2024-03-19T03:42:48.633 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.633 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.633 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.634 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.634 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lssnap_latency": { 2024-03-19T03:42:48.634 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.634 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.634 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.634 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.634 INFO:teuthology.orchestra.run.smithi178.stdout: "req_mkdir_latency": { 2024-03-19T03:42:48.635 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.635 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.635 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.635 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.635 INFO:teuthology.orchestra.run.smithi178.stdout: "req_mknod_latency": { 2024-03-19T03:42:48.635 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.635 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.635 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.635 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.635 INFO:teuthology.orchestra.run.smithi178.stdout: "req_mksnap_latency": { 2024-03-19T03:42:48.635 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.635 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.635 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.636 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.636 INFO:teuthology.orchestra.run.smithi178.stdout: "req_open_latency": { 2024-03-19T03:42:48.636 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.636 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.636 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.636 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.636 INFO:teuthology.orchestra.run.smithi178.stdout: "req_readdir_latency": { 2024-03-19T03:42:48.636 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.636 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.636 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.636 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.636 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rename_latency": { 2024-03-19T03:42:48.636 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.636 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.636 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.636 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.637 INFO:teuthology.orchestra.run.smithi178.stdout: "req_renamesnap_latency": { 2024-03-19T03:42:48.637 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.637 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.637 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.637 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.637 INFO:teuthology.orchestra.run.smithi178.stdout: "req_snapdiff_latency": { 2024-03-19T03:42:48.637 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.637 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.637 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.637 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.637 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rmdir_latency": { 2024-03-19T03:42:48.637 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.637 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.637 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.637 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.637 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rmsnap_latency": { 2024-03-19T03:42:48.638 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.638 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.638 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.638 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.638 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rmxattr_latency": { 2024-03-19T03:42:48.638 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.638 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.638 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.638 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.638 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setattr_latency": { 2024-03-19T03:42:48.638 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.638 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.638 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.638 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.638 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setdirlayout_latency": { 2024-03-19T03:42:48.639 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.639 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.639 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.639 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.639 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setfilelock_latency": { 2024-03-19T03:42:48.639 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.639 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.639 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.639 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.639 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setlayout_latency": { 2024-03-19T03:42:48.639 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.639 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.639 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.639 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.639 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setxattr_latency": { 2024-03-19T03:42:48.640 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.640 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.640 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.640 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.640 INFO:teuthology.orchestra.run.smithi178.stdout: "req_symlink_latency": { 2024-03-19T03:42:48.640 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.640 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.640 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.640 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.640 INFO:teuthology.orchestra.run.smithi178.stdout: "req_unlink_latency": { 2024-03-19T03:42:48.640 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.640 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.640 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.640 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.640 INFO:teuthology.orchestra.run.smithi178.stdout: "cap_revoke_eviction": 0, 2024-03-19T03:42:48.640 INFO:teuthology.orchestra.run.smithi178.stdout: "cap_acquisition_throttle": 0, 2024-03-19T03:42:48.641 INFO:teuthology.orchestra.run.smithi178.stdout: "req_getvxattr_latency": { 2024-03-19T03:42:48.641 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.641 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.641 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.641 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:48.641 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.641 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_sessions": { 2024-03-19T03:42:48.641 INFO:teuthology.orchestra.run.smithi178.stdout: "session_count": 1, 2024-03-19T03:42:48.641 INFO:teuthology.orchestra.run.smithi178.stdout: "session_add": 4, 2024-03-19T03:42:48.641 INFO:teuthology.orchestra.run.smithi178.stdout: "session_remove": 3, 2024-03-19T03:42:48.641 INFO:teuthology.orchestra.run.smithi178.stdout: "sessions_open": 2, 2024-03-19T03:42:48.641 INFO:teuthology.orchestra.run.smithi178.stdout: "sessions_stale": 0, 2024-03-19T03:42:48.641 INFO:teuthology.orchestra.run.smithi178.stdout: "total_load": 0, 2024-03-19T03:42:48.641 INFO:teuthology.orchestra.run.smithi178.stdout: "average_load": 0, 2024-03-19T03:42:48.641 INFO:teuthology.orchestra.run.smithi178.stdout: "avg_session_uptime": 6873, 2024-03-19T03:42:48.642 INFO:teuthology.orchestra.run.smithi178.stdout: "mdthresh_evicted": 0 2024-03-19T03:42:48.642 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.642 INFO:teuthology.orchestra.run.smithi178.stdout: "mempool": { 2024-03-19T03:42:48.642 INFO:teuthology.orchestra.run.smithi178.stdout: "bloom_filter_bytes": 0, 2024-03-19T03:42:48.642 INFO:teuthology.orchestra.run.smithi178.stdout: "bloom_filter_items": 0, 2024-03-19T03:42:48.642 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T03:42:48.642 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_alloc_items": 0, 2024-03-19T03:42:48.642 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T03:42:48.642 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_data_items": 0, 2024-03-19T03:42:48.642 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T03:42:48.642 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T03:42:48.642 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T03:42:48.642 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T03:42:48.643 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T03:42:48.643 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_other_items": 0, 2024-03-19T03:42:48.643 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T03:42:48.643 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T03:42:48.643 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_extent_bytes": 0, 2024-03-19T03:42:48.643 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_extent_items": 0, 2024-03-19T03:42:48.643 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_blob_bytes": 0, 2024-03-19T03:42:48.643 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_blob_items": 0, 2024-03-19T03:42:48.643 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T03:42:48.643 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T03:42:48.643 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T03:42:48.643 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T03:42:48.643 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T03:42:48.643 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_items": 0, 2024-03-19T03:42:48.643 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_txc_bytes": 0, 2024-03-19T03:42:48.643 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_txc_items": 0, 2024-03-19T03:42:48.644 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T03:42:48.644 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T03:42:48.644 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_bytes": 0, 2024-03-19T03:42:48.644 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_items": 0, 2024-03-19T03:42:48.644 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_bytes": 0, 2024-03-19T03:42:48.644 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_items": 0, 2024-03-19T03:42:48.644 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T03:42:48.644 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_reader_items": 0, 2024-03-19T03:42:48.644 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T03:42:48.644 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_writer_items": 0, 2024-03-19T03:42:48.644 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_anon_bytes": 239634, 2024-03-19T03:42:48.644 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_anon_items": 84, 2024-03-19T03:42:48.644 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_meta_bytes": 0, 2024-03-19T03:42:48.644 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_meta_items": 0, 2024-03-19T03:42:48.645 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_bytes": 0, 2024-03-19T03:42:48.645 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_items": 0, 2024-03-19T03:42:48.645 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mapbl_bytes": 0, 2024-03-19T03:42:48.645 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mapbl_items": 0, 2024-03-19T03:42:48.645 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pglog_bytes": 0, 2024-03-19T03:42:48.645 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pglog_items": 0, 2024-03-19T03:42:48.645 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_bytes": 9184, 2024-03-19T03:42:48.645 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_items": 177, 2024-03-19T03:42:48.645 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T03:42:48.645 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_mapping_items": 0, 2024-03-19T03:42:48.645 INFO:teuthology.orchestra.run.smithi178.stdout: "pgmap_bytes": 0, 2024-03-19T03:42:48.645 INFO:teuthology.orchestra.run.smithi178.stdout: "pgmap_items": 0, 2024-03-19T03:42:48.645 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_co_bytes": 30405289, 2024-03-19T03:42:48.645 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_co_items": 174409, 2024-03-19T03:42:48.645 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_1_bytes": 0, 2024-03-19T03:42:48.646 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_1_items": 0, 2024-03-19T03:42:48.646 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_2_bytes": 0, 2024-03-19T03:42:48.646 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_2_items": 0 2024-03-19T03:42:48.646 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.646 INFO:teuthology.orchestra.run.smithi178.stdout: "objecter": { 2024-03-19T03:42:48.646 INFO:teuthology.orchestra.run.smithi178.stdout: "op_active": 0, 2024-03-19T03:42:48.646 INFO:teuthology.orchestra.run.smithi178.stdout: "op_laggy": 0, 2024-03-19T03:42:48.646 INFO:teuthology.orchestra.run.smithi178.stdout: "op_send": 3055, 2024-03-19T03:42:48.646 INFO:teuthology.orchestra.run.smithi178.stdout: "op_send_bytes": 26, 2024-03-19T03:42:48.646 INFO:teuthology.orchestra.run.smithi178.stdout: "op_resend": 4, 2024-03-19T03:42:48.646 INFO:teuthology.orchestra.run.smithi178.stdout: "op_reply": 3055, 2024-03-19T03:42:48.646 INFO:teuthology.orchestra.run.smithi178.stdout: "op_latency": { 2024-03-19T03:42:48.646 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 3055, 2024-03-19T03:42:48.647 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 13.278738491, 2024-03-19T03:42:48.647 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.004346559 2024-03-19T03:42:48.647 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.647 INFO:teuthology.orchestra.run.smithi178.stdout: "op_inflight": 0, 2024-03-19T03:42:48.647 INFO:teuthology.orchestra.run.smithi178.stdout: "oplen_avg": { 2024-03-19T03:42:48.647 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 3055, 2024-03-19T03:42:48.647 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 3056 2024-03-19T03:42:48.647 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.647 INFO:teuthology.orchestra.run.smithi178.stdout: "op": 3055, 2024-03-19T03:42:48.647 INFO:teuthology.orchestra.run.smithi178.stdout: "op_r": 3055, 2024-03-19T03:42:48.647 INFO:teuthology.orchestra.run.smithi178.stdout: "op_w": 0, 2024-03-19T03:42:48.647 INFO:teuthology.orchestra.run.smithi178.stdout: "op_rmw": 0, 2024-03-19T03:42:48.647 INFO:teuthology.orchestra.run.smithi178.stdout: "op_pg": 0, 2024-03-19T03:42:48.647 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_stat": 1828, 2024-03-19T03:42:48.648 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_create": 0, 2024-03-19T03:42:48.648 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_read": 1224, 2024-03-19T03:42:48.648 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_write": 0, 2024-03-19T03:42:48.648 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_writefull": 0, 2024-03-19T03:42:48.648 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_writesame": 0, 2024-03-19T03:42:48.648 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_append": 0, 2024-03-19T03:42:48.648 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_zero": 0, 2024-03-19T03:42:48.648 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_truncate": 0, 2024-03-19T03:42:48.648 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_delete": 0, 2024-03-19T03:42:48.648 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_mapext": 0, 2024-03-19T03:42:48.648 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_sparse_read": 0, 2024-03-19T03:42:48.649 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_clonerange": 0, 2024-03-19T03:42:48.649 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_getxattr": 2, 2024-03-19T03:42:48.649 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_setxattr": 0, 2024-03-19T03:42:48.649 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_cmpxattr": 0, 2024-03-19T03:42:48.649 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_rmxattr": 0, 2024-03-19T03:42:48.649 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_resetxattrs": 0, 2024-03-19T03:42:48.649 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_call": 0, 2024-03-19T03:42:48.649 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_watch": 0, 2024-03-19T03:42:48.649 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_notify": 0, 2024-03-19T03:42:48.649 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T03:42:48.649 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_pgls": 0, 2024-03-19T03:42:48.649 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_pgls_filter": 0, 2024-03-19T03:42:48.650 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_other": 0, 2024-03-19T03:42:48.650 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_active": 0, 2024-03-19T03:42:48.650 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_send": 0, 2024-03-19T03:42:48.650 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_resend": 0, 2024-03-19T03:42:48.650 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_ping": 0, 2024-03-19T03:42:48.650 INFO:teuthology.orchestra.run.smithi178.stdout: "poolop_active": 0, 2024-03-19T03:42:48.650 INFO:teuthology.orchestra.run.smithi178.stdout: "poolop_send": 0, 2024-03-19T03:42:48.650 INFO:teuthology.orchestra.run.smithi178.stdout: "poolop_resend": 0, 2024-03-19T03:42:48.650 INFO:teuthology.orchestra.run.smithi178.stdout: "poolstat_active": 0, 2024-03-19T03:42:48.651 INFO:teuthology.orchestra.run.smithi178.stdout: "poolstat_send": 0, 2024-03-19T03:42:48.651 INFO:teuthology.orchestra.run.smithi178.stdout: "poolstat_resend": 0, 2024-03-19T03:42:48.651 INFO:teuthology.orchestra.run.smithi178.stdout: "statfs_active": 0, 2024-03-19T03:42:48.651 INFO:teuthology.orchestra.run.smithi178.stdout: "statfs_send": 0, 2024-03-19T03:42:48.651 INFO:teuthology.orchestra.run.smithi178.stdout: "statfs_resend": 0, 2024-03-19T03:42:48.651 INFO:teuthology.orchestra.run.smithi178.stdout: "command_active": 0, 2024-03-19T03:42:48.651 INFO:teuthology.orchestra.run.smithi178.stdout: "command_send": 0, 2024-03-19T03:42:48.651 INFO:teuthology.orchestra.run.smithi178.stdout: "command_resend": 0, 2024-03-19T03:42:48.651 INFO:teuthology.orchestra.run.smithi178.stdout: "map_epoch": 93, 2024-03-19T03:42:48.651 INFO:teuthology.orchestra.run.smithi178.stdout: "map_full": 0, 2024-03-19T03:42:48.651 INFO:teuthology.orchestra.run.smithi178.stdout: "map_inc": 4, 2024-03-19T03:42:48.651 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_sessions": 11, 2024-03-19T03:42:48.651 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_session_open": 11, 2024-03-19T03:42:48.652 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_session_close": 0, 2024-03-19T03:42:48.652 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_laggy": 0, 2024-03-19T03:42:48.652 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_wr": 0, 2024-03-19T03:42:48.652 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_rd": 2, 2024-03-19T03:42:48.652 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_del": 0 2024-03-19T03:42:48.652 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.652 INFO:teuthology.orchestra.run.smithi178.stdout: "oft": { 2024-03-19T03:42:48.652 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_objs": 0, 2024-03-19T03:42:48.652 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_kv_pairs": 0, 2024-03-19T03:42:48.652 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_updates": 0, 2024-03-19T03:42:48.652 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_removes": 0 2024-03-19T03:42:48.652 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.652 INFO:teuthology.orchestra.run.smithi178.stdout: "osd-slow-ops": { 2024-03-19T03:42:48.653 INFO:teuthology.orchestra.run.smithi178.stdout: "slow_ops_count": 0 2024-03-19T03:42:48.653 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.653 INFO:teuthology.orchestra.run.smithi178.stdout: "purge_queue": { 2024-03-19T03:42:48.653 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing_ops": 0, 2024-03-19T03:42:48.653 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing_ops_high_water": 0, 2024-03-19T03:42:48.653 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing": 0, 2024-03-19T03:42:48.653 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing_high_water": 0, 2024-03-19T03:42:48.653 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executed_ops": 0, 2024-03-19T03:42:48.653 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executed": 0, 2024-03-19T03:42:48.653 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_item_in_journal": 0 2024-03-19T03:42:48.654 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.654 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T03:42:48.654 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-19T03:42:48.654 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 104857600, 2024-03-19T03:42:48.654 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-19T03:42:48.654 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 3294, 2024-03-19T03:42:48.654 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 523606770, 2024-03-19T03:42:48.654 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:48.654 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 3294, 2024-03-19T03:42:48.654 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 0, 2024-03-19T03:42:48.654 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 0, 2024-03-19T03:42:48.655 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 3294, 2024-03-19T03:42:48.655 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 523606770, 2024-03-19T03:42:48.655 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-19T03:42:48.655 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.655 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.655 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.655 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:48.655 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.655 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-objecter_bytes": { 2024-03-19T03:42:48.655 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-19T03:42:48.655 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 104857600, 2024-03-19T03:42:48.655 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-19T03:42:48.655 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-19T03:42:48.655 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-19T03:42:48.656 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:48.656 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:48.656 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 3055, 2024-03-19T03:42:48.656 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 522179151, 2024-03-19T03:42:48.656 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 3055, 2024-03-19T03:42:48.656 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 522179151, 2024-03-19T03:42:48.656 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-19T03:42:48.656 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.656 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.656 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.656 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:48.657 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.657 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-objecter_ops": { 2024-03-19T03:42:48.657 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-19T03:42:48.657 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 1024, 2024-03-19T03:42:48.657 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-19T03:42:48.657 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-19T03:42:48.657 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-19T03:42:48.657 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:48.657 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:48.657 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 3055, 2024-03-19T03:42:48.657 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 3055, 2024-03-19T03:42:48.657 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 3055, 2024-03-19T03:42:48.657 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 3055, 2024-03-19T03:42:48.658 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-19T03:42:48.658 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.658 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.658 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.658 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:48.658 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.658 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-write_buf_throttle": { 2024-03-19T03:42:48.658 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-19T03:42:48.658 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 3758096384, 2024-03-19T03:42:48.658 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-19T03:42:48.658 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-19T03:42:48.658 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-19T03:42:48.658 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:48.658 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:48.659 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 0, 2024-03-19T03:42:48.659 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 0, 2024-03-19T03:42:48.659 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 0, 2024-03-19T03:42:48.659 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 0, 2024-03-19T03:42:48.659 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-19T03:42:48.659 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.659 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.659 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.659 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:48.659 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:48.659 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-write_buf_throttle-0x55e69e9a06e0": { 2024-03-19T03:42:48.659 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-19T03:42:48.659 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 3758096384, 2024-03-19T03:42:48.659 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-19T03:42:48.660 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-19T03:42:48.660 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-19T03:42:48.660 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:48.660 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:48.660 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 0, 2024-03-19T03:42:48.660 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 0, 2024-03-19T03:42:48.660 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 0, 2024-03-19T03:42:48.660 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 0, 2024-03-19T03:42:48.660 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-19T03:42:48.660 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:48.660 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:48.660 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:48.660 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:48.661 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:48.661 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-03-19T03:42:48.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:48 smithi119 ceph-mon[66861]: pgmap v688: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T03:42:49.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:48 smithi117 ceph-mon[31838]: pgmap v688: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T03:42:49.070 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:48 smithi178 ceph-mon[67578]: pgmap v688: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-19T03:42:49.388 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-19T03:42:49.388 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-03-19T03:42:49.388 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=0 2024-03-19T03:42:49.388 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=0 2024-03-19T03:42:49.388 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2024-03-19T03:42:49.389 DEBUG:tasks.check_counter:Getting stats from l 2024-03-19T03:42:49.389 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid a3a82456-e59f-11ee-95c9-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-19T03:42:49.624 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T03:42:50.740 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:50 smithi178 ceph-mon[67578]: pgmap v689: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:42:50.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:50 smithi119 ceph-mon[66861]: pgmap v689: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:42:51.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:50 smithi117 ceph-mon[31838]: pgmap v689: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T03:42:51.285 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2024-03-19T03:42:51.285 INFO:teuthology.orchestra.run.smithi178.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T03:42:51.285 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_messages": 776, 2024-03-19T03:42:51.285 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages": 778, 2024-03-19T03:42:51.286 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_bytes": 252941, 2024-03-19T03:42:51.286 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_bytes": 246769, 2024-03-19T03:42:51.286 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_created_connections": 6, 2024-03-19T03:42:51.286 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_active_connections": 2, 2024-03-19T03:42:51.286 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_total_time": 0.085659793, 2024-03-19T03:42:51.286 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_send_time": 0.025697083, 2024-03-19T03:42:51.286 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_recv_time": 0.022833493, 2024-03-19T03:42:51.286 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_fast_dispatch_time": 0.028499350, 2024-03-19T03:42:51.286 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:51.286 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 776, 2024-03-19T03:42:51.286 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.040257739, 2024-03-19T03:42:51.286 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000051878 2024-03-19T03:42:51.286 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.287 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:51.287 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.287 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.287 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.287 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.287 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T03:42:51.287 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T03:42:51.287 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.287 INFO:teuthology.orchestra.run.smithi178.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T03:42:51.287 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_messages": 1273, 2024-03-19T03:42:51.287 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages": 1430, 2024-03-19T03:42:51.287 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_bytes": 9965865, 2024-03-19T03:42:51.287 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_bytes": 823203, 2024-03-19T03:42:51.287 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_created_connections": 6, 2024-03-19T03:42:51.288 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_active_connections": 4, 2024-03-19T03:42:51.288 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_total_time": 0.097393657, 2024-03-19T03:42:51.288 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_send_time": 0.034187895, 2024-03-19T03:42:51.288 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_recv_time": 0.027383359, 2024-03-19T03:42:51.288 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_fast_dispatch_time": 0.027704433, 2024-03-19T03:42:51.288 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:51.288 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 1429, 2024-03-19T03:42:51.288 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.055306896, 2024-03-19T03:42:51.288 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000038703 2024-03-19T03:42:51.288 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.288 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:51.288 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.288 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.288 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.289 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.289 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-03-19T03:42:51.289 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_encrypted_bytes": 420240 2024-03-19T03:42:51.289 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.289 INFO:teuthology.orchestra.run.smithi178.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T03:42:51.289 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_messages": 243, 2024-03-19T03:42:51.289 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages": 221, 2024-03-19T03:42:51.289 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_bytes": 884196, 2024-03-19T03:42:51.289 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_bytes": 131045, 2024-03-19T03:42:51.289 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_created_connections": 4, 2024-03-19T03:42:51.289 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_active_connections": 3, 2024-03-19T03:42:51.289 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_total_time": 0.037413414, 2024-03-19T03:42:51.289 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_send_time": 0.012974584, 2024-03-19T03:42:51.289 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_recv_time": 0.022346179, 2024-03-19T03:42:51.289 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_running_fast_dispatch_time": 0.005190494, 2024-03-19T03:42:51.290 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T03:42:51.290 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 220, 2024-03-19T03:42:51.290 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.011053780, 2024-03-19T03:42:51.290 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000050244 2024-03-19T03:42:51.290 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.290 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_handle_ack_lat": { 2024-03-19T03:42:51.290 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.290 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.290 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.290 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.290 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_recv_encrypted_bytes": 882848, 2024-03-19T03:42:51.290 INFO:teuthology.orchestra.run.smithi178.stdout: "msgr_send_encrypted_bytes": 130080 2024-03-19T03:42:51.290 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.291 INFO:teuthology.orchestra.run.smithi178.stdout: "cct": { 2024-03-19T03:42:51.291 INFO:teuthology.orchestra.run.smithi178.stdout: "total_workers": 1, 2024-03-19T03:42:51.291 INFO:teuthology.orchestra.run.smithi178.stdout: "unhealthy_workers": 0 2024-03-19T03:42:51.291 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.291 INFO:teuthology.orchestra.run.smithi178.stdout: "finisher-MDSRank": { 2024-03-19T03:42:51.291 INFO:teuthology.orchestra.run.smithi178.stdout: "queue_len": 0, 2024-03-19T03:42:51.291 INFO:teuthology.orchestra.run.smithi178.stdout: "complete_latency": { 2024-03-19T03:42:51.291 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 3526, 2024-03-19T03:42:51.291 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.090082520, 2024-03-19T03:42:51.291 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000025548 2024-03-19T03:42:51.291 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:51.291 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.291 INFO:teuthology.orchestra.run.smithi178.stdout: "finisher-PurgeQueue": { 2024-03-19T03:42:51.291 INFO:teuthology.orchestra.run.smithi178.stdout: "queue_len": 0, 2024-03-19T03:42:51.292 INFO:teuthology.orchestra.run.smithi178.stdout: "complete_latency": { 2024-03-19T03:42:51.292 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.292 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.292 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.292 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:51.292 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.292 INFO:teuthology.orchestra.run.smithi178.stdout: "mds": { 2024-03-19T03:42:51.292 INFO:teuthology.orchestra.run.smithi178.stdout: "request": 0, 2024-03-19T03:42:51.292 INFO:teuthology.orchestra.run.smithi178.stdout: "reply": 0, 2024-03-19T03:42:51.292 INFO:teuthology.orchestra.run.smithi178.stdout: "reply_latency": { 2024-03-19T03:42:51.292 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.292 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.292 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.292 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.293 INFO:teuthology.orchestra.run.smithi178.stdout: "slow_reply": 0, 2024-03-19T03:42:51.293 INFO:teuthology.orchestra.run.smithi178.stdout: "forward": 0, 2024-03-19T03:42:51.293 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_fetch_complete": 0, 2024-03-19T03:42:51.293 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_fetch_keys": 0, 2024-03-19T03:42:51.293 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_commit": 0, 2024-03-19T03:42:51.293 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_split": 0, 2024-03-19T03:42:51.293 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_merge": 0, 2024-03-19T03:42:51.293 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes": 1204, 2024-03-19T03:42:51.293 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_top": 0, 2024-03-19T03:42:51.293 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_bottom": 1204, 2024-03-19T03:42:51.293 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_pin_tail": 0, 2024-03-19T03:42:51.293 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_pinned": 1204, 2024-03-19T03:42:51.293 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_expired": 0, 2024-03-19T03:42:51.294 INFO:teuthology.orchestra.run.smithi178.stdout: "inodes_with_caps": 0, 2024-03-19T03:42:51.294 INFO:teuthology.orchestra.run.smithi178.stdout: "caps": 0, 2024-03-19T03:42:51.294 INFO:teuthology.orchestra.run.smithi178.stdout: "subtrees": 1, 2024-03-19T03:42:51.294 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse": 0, 2024-03-19T03:42:51.294 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_hit": 0, 2024-03-19T03:42:51.294 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_forward": 0, 2024-03-19T03:42:51.294 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_discover": 0, 2024-03-19T03:42:51.294 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_dir_fetch": 0, 2024-03-19T03:42:51.294 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_remote_ino": 0, 2024-03-19T03:42:51.294 INFO:teuthology.orchestra.run.smithi178.stdout: "traverse_lock": 0, 2024-03-19T03:42:51.294 INFO:teuthology.orchestra.run.smithi178.stdout: "load_cent": 0, 2024-03-19T03:42:51.294 INFO:teuthology.orchestra.run.smithi178.stdout: "q": 0, 2024-03-19T03:42:51.294 INFO:teuthology.orchestra.run.smithi178.stdout: "exported": 0, 2024-03-19T03:42:51.295 INFO:teuthology.orchestra.run.smithi178.stdout: "exported_inodes": 0, 2024-03-19T03:42:51.295 INFO:teuthology.orchestra.run.smithi178.stdout: "imported": 0, 2024-03-19T03:42:51.295 INFO:teuthology.orchestra.run.smithi178.stdout: "imported_inodes": 0, 2024-03-19T03:42:51.295 INFO:teuthology.orchestra.run.smithi178.stdout: "openino_dir_fetch": 0, 2024-03-19T03:42:51.295 INFO:teuthology.orchestra.run.smithi178.stdout: "openino_backtrace_fetch": 0, 2024-03-19T03:42:51.295 INFO:teuthology.orchestra.run.smithi178.stdout: "openino_peer_discover": 0, 2024-03-19T03:42:51.295 INFO:teuthology.orchestra.run.smithi178.stdout: "root_rfiles": 0, 2024-03-19T03:42:51.295 INFO:teuthology.orchestra.run.smithi178.stdout: "root_rbytes": 0, 2024-03-19T03:42:51.295 INFO:teuthology.orchestra.run.smithi178.stdout: "root_rsnaps": 0, 2024-03-19T03:42:51.295 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T03:42:51.295 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_set_tag": 0, 2024-03-19T03:42:51.295 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T03:42:51.295 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_inotable_repaired": 0, 2024-03-19T03:42:51.295 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_dir_inodes": 0, 2024-03-19T03:42:51.295 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T03:42:51.296 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T03:42:51.296 INFO:teuthology.orchestra.run.smithi178.stdout: "scrub_file_inodes": 0, 2024-03-19T03:42:51.296 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_inode_file_caps": 0, 2024-03-19T03:42:51.296 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_revoke": 0, 2024-03-19T03:42:51.296 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_grant": 0, 2024-03-19T03:42:51.296 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T03:42:51.296 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T03:42:51.296 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T03:42:51.296 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_caps": 0, 2024-03-19T03:42:51.296 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_caps_dirty": 0, 2024-03-19T03:42:51.296 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_cap_release": 0, 2024-03-19T03:42:51.296 INFO:teuthology.orchestra.run.smithi178.stdout: "process_request_cap_release": 0 2024-03-19T03:42:51.296 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.296 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_cache": { 2024-03-19T03:42:51.297 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_update": 0, 2024-03-19T03:42:51.297 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_update_receipt": 0, 2024-03-19T03:42:51.297 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_try_discover": 0, 2024-03-19T03:42:51.297 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_send_discover": 0, 2024-03-19T03:42:51.297 INFO:teuthology.orchestra.run.smithi178.stdout: "dir_handle_discover": 0, 2024-03-19T03:42:51.297 INFO:teuthology.orchestra.run.smithi178.stdout: "num_strays": 0, 2024-03-19T03:42:51.297 INFO:teuthology.orchestra.run.smithi178.stdout: "num_strays_delayed": 0, 2024-03-19T03:42:51.297 INFO:teuthology.orchestra.run.smithi178.stdout: "num_strays_enqueuing": 0, 2024-03-19T03:42:51.297 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_created": 0, 2024-03-19T03:42:51.297 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_enqueued": 0, 2024-03-19T03:42:51.297 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_reintegrated": 0, 2024-03-19T03:42:51.297 INFO:teuthology.orchestra.run.smithi178.stdout: "strays_migrated": 0, 2024-03-19T03:42:51.297 INFO:teuthology.orchestra.run.smithi178.stdout: "num_recovering_processing": 0, 2024-03-19T03:42:51.297 INFO:teuthology.orchestra.run.smithi178.stdout: "num_recovering_enqueued": 0, 2024-03-19T03:42:51.297 INFO:teuthology.orchestra.run.smithi178.stdout: "num_recovering_prioritized": 0, 2024-03-19T03:42:51.298 INFO:teuthology.orchestra.run.smithi178.stdout: "recovery_started": 0, 2024-03-19T03:42:51.298 INFO:teuthology.orchestra.run.smithi178.stdout: "recovery_completed": 0, 2024-03-19T03:42:51.298 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_quiesce_path": 0, 2024-03-19T03:42:51.298 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_quiesce_inode": 0, 2024-03-19T03:42:51.298 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T03:42:51.298 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_exportdir": 0, 2024-03-19T03:42:51.298 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_flush": 0, 2024-03-19T03:42:51.298 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_fragmentdir": 0, 2024-03-19T03:42:51.298 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_fragstats": 0, 2024-03-19T03:42:51.298 INFO:teuthology.orchestra.run.smithi178.stdout: "ireq_inodestats": 0 2024-03-19T03:42:51.298 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.298 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_log": { 2024-03-19T03:42:51.298 INFO:teuthology.orchestra.run.smithi178.stdout: "evlrg": 0, 2024-03-19T03:42:51.298 INFO:teuthology.orchestra.run.smithi178.stdout: "evadd": 0, 2024-03-19T03:42:51.299 INFO:teuthology.orchestra.run.smithi178.stdout: "evex": 0, 2024-03-19T03:42:51.299 INFO:teuthology.orchestra.run.smithi178.stdout: "evtrm": 0, 2024-03-19T03:42:51.299 INFO:teuthology.orchestra.run.smithi178.stdout: "ev": 6494, 2024-03-19T03:42:51.299 INFO:teuthology.orchestra.run.smithi178.stdout: "evexg": 0, 2024-03-19T03:42:51.299 INFO:teuthology.orchestra.run.smithi178.stdout: "evexd": 0, 2024-03-19T03:42:51.299 INFO:teuthology.orchestra.run.smithi178.stdout: "segadd": 0, 2024-03-19T03:42:51.299 INFO:teuthology.orchestra.run.smithi178.stdout: "segex": 0, 2024-03-19T03:42:51.299 INFO:teuthology.orchestra.run.smithi178.stdout: "segtrm": 0, 2024-03-19T03:42:51.299 INFO:teuthology.orchestra.run.smithi178.stdout: "seg": 8, 2024-03-19T03:42:51.299 INFO:teuthology.orchestra.run.smithi178.stdout: "segmjr": 2, 2024-03-19T03:42:51.299 INFO:teuthology.orchestra.run.smithi178.stdout: "segexg": 0, 2024-03-19T03:42:51.299 INFO:teuthology.orchestra.run.smithi178.stdout: "segexd": 0, 2024-03-19T03:42:51.299 INFO:teuthology.orchestra.run.smithi178.stdout: "expos": 4194304, 2024-03-19T03:42:51.299 INFO:teuthology.orchestra.run.smithi178.stdout: "wrpos": 13833607, 2024-03-19T03:42:51.300 INFO:teuthology.orchestra.run.smithi178.stdout: "rdpos": 13832469, 2024-03-19T03:42:51.300 INFO:teuthology.orchestra.run.smithi178.stdout: "jlat": { 2024-03-19T03:42:51.300 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.300 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.300 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.300 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.300 INFO:teuthology.orchestra.run.smithi178.stdout: "replayed": 6494 2024-03-19T03:42:51.300 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.300 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_mem": { 2024-03-19T03:42:51.300 INFO:teuthology.orchestra.run.smithi178.stdout: "ino": 10, 2024-03-19T03:42:51.300 INFO:teuthology.orchestra.run.smithi178.stdout: "ino+": 1231, 2024-03-19T03:42:51.300 INFO:teuthology.orchestra.run.smithi178.stdout: "ino-": 1221, 2024-03-19T03:42:51.300 INFO:teuthology.orchestra.run.smithi178.stdout: "dir": 8, 2024-03-19T03:42:51.300 INFO:teuthology.orchestra.run.smithi178.stdout: "dir+": 75, 2024-03-19T03:42:51.300 INFO:teuthology.orchestra.run.smithi178.stdout: "dir-": 67, 2024-03-19T03:42:51.301 INFO:teuthology.orchestra.run.smithi178.stdout: "dn": 1204, 2024-03-19T03:42:51.301 INFO:teuthology.orchestra.run.smithi178.stdout: "dn+": 2894, 2024-03-19T03:42:51.301 INFO:teuthology.orchestra.run.smithi178.stdout: "dn-": 1690, 2024-03-19T03:42:51.301 INFO:teuthology.orchestra.run.smithi178.stdout: "cap": 0, 2024-03-19T03:42:51.301 INFO:teuthology.orchestra.run.smithi178.stdout: "cap+": 0, 2024-03-19T03:42:51.301 INFO:teuthology.orchestra.run.smithi178.stdout: "cap-": 0, 2024-03-19T03:42:51.301 INFO:teuthology.orchestra.run.smithi178.stdout: "rss": 52648, 2024-03-19T03:42:51.301 INFO:teuthology.orchestra.run.smithi178.stdout: "heap": 215420 2024-03-19T03:42:51.301 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.301 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_server": { 2024-03-19T03:42:51.301 INFO:teuthology.orchestra.run.smithi178.stdout: "dispatch_client_request": 0, 2024-03-19T03:42:51.301 INFO:teuthology.orchestra.run.smithi178.stdout: "dispatch_server_request": 0, 2024-03-19T03:42:51.301 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_request": 0, 2024-03-19T03:42:51.301 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_client_session": 0, 2024-03-19T03:42:51.302 INFO:teuthology.orchestra.run.smithi178.stdout: "handle_peer_request": 0, 2024-03-19T03:42:51.302 INFO:teuthology.orchestra.run.smithi178.stdout: "req_create_latency": { 2024-03-19T03:42:51.302 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.302 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.302 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.302 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.302 INFO:teuthology.orchestra.run.smithi178.stdout: "req_getattr_latency": { 2024-03-19T03:42:51.302 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.302 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.302 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.302 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.302 INFO:teuthology.orchestra.run.smithi178.stdout: "req_getfilelock_latency": { 2024-03-19T03:42:51.302 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.302 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.302 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.303 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.303 INFO:teuthology.orchestra.run.smithi178.stdout: "req_link_latency": { 2024-03-19T03:42:51.303 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.303 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.303 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.303 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.303 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookup_latency": { 2024-03-19T03:42:51.303 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.303 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.303 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.303 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.303 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookuphash_latency": { 2024-03-19T03:42:51.303 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.303 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.303 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.304 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.304 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupino_latency": { 2024-03-19T03:42:51.304 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.304 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.304 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.304 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.304 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupname_latency": { 2024-03-19T03:42:51.304 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.304 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.304 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.304 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.304 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupparent_latency": { 2024-03-19T03:42:51.304 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.304 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.304 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.305 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.305 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lookupsnap_latency": { 2024-03-19T03:42:51.305 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.305 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.305 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.305 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.305 INFO:teuthology.orchestra.run.smithi178.stdout: "req_lssnap_latency": { 2024-03-19T03:42:51.305 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.305 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.305 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.305 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.305 INFO:teuthology.orchestra.run.smithi178.stdout: "req_mkdir_latency": { 2024-03-19T03:42:51.306 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.306 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.306 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.306 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.306 INFO:teuthology.orchestra.run.smithi178.stdout: "req_mknod_latency": { 2024-03-19T03:42:51.306 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.306 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.307 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.307 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.307 INFO:teuthology.orchestra.run.smithi178.stdout: "req_mksnap_latency": { 2024-03-19T03:42:51.307 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.307 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.307 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.307 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.307 INFO:teuthology.orchestra.run.smithi178.stdout: "req_open_latency": { 2024-03-19T03:42:51.307 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.307 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.307 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.307 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.307 INFO:teuthology.orchestra.run.smithi178.stdout: "req_readdir_latency": { 2024-03-19T03:42:51.307 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.308 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.308 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.308 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.308 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rename_latency": { 2024-03-19T03:42:51.308 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.308 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.308 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.308 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.308 INFO:teuthology.orchestra.run.smithi178.stdout: "req_renamesnap_latency": { 2024-03-19T03:42:51.308 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.308 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.308 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.308 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.308 INFO:teuthology.orchestra.run.smithi178.stdout: "req_snapdiff_latency": { 2024-03-19T03:42:51.309 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.309 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.309 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.309 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.309 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rmdir_latency": { 2024-03-19T03:42:51.309 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.309 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.309 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.309 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.309 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rmsnap_latency": { 2024-03-19T03:42:51.309 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.309 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.309 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.310 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.310 INFO:teuthology.orchestra.run.smithi178.stdout: "req_rmxattr_latency": { 2024-03-19T03:42:51.310 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.310 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.310 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.310 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.310 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setattr_latency": { 2024-03-19T03:42:51.310 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.310 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.310 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.310 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.310 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setdirlayout_latency": { 2024-03-19T03:42:51.310 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.310 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.311 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.311 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.311 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setfilelock_latency": { 2024-03-19T03:42:51.311 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.311 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.311 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.311 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.311 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setlayout_latency": { 2024-03-19T03:42:51.311 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.311 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.311 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.311 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.311 INFO:teuthology.orchestra.run.smithi178.stdout: "req_setxattr_latency": { 2024-03-19T03:42:51.311 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.312 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.312 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.312 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.312 INFO:teuthology.orchestra.run.smithi178.stdout: "req_symlink_latency": { 2024-03-19T03:42:51.312 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.312 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.312 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.312 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.312 INFO:teuthology.orchestra.run.smithi178.stdout: "req_unlink_latency": { 2024-03-19T03:42:51.312 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.312 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.312 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.312 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.312 INFO:teuthology.orchestra.run.smithi178.stdout: "cap_revoke_eviction": 0, 2024-03-19T03:42:51.313 INFO:teuthology.orchestra.run.smithi178.stdout: "cap_acquisition_throttle": 0, 2024-03-19T03:42:51.313 INFO:teuthology.orchestra.run.smithi178.stdout: "req_getvxattr_latency": { 2024-03-19T03:42:51.313 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.313 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.313 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.313 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:51.313 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.313 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_sessions": { 2024-03-19T03:42:51.313 INFO:teuthology.orchestra.run.smithi178.stdout: "session_count": 1, 2024-03-19T03:42:51.313 INFO:teuthology.orchestra.run.smithi178.stdout: "session_add": 2, 2024-03-19T03:42:51.313 INFO:teuthology.orchestra.run.smithi178.stdout: "session_remove": 1, 2024-03-19T03:42:51.313 INFO:teuthology.orchestra.run.smithi178.stdout: "sessions_open": 2, 2024-03-19T03:42:51.313 INFO:teuthology.orchestra.run.smithi178.stdout: "sessions_stale": 0, 2024-03-19T03:42:51.313 INFO:teuthology.orchestra.run.smithi178.stdout: "total_load": 0, 2024-03-19T03:42:51.314 INFO:teuthology.orchestra.run.smithi178.stdout: "average_load": 0, 2024-03-19T03:42:51.314 INFO:teuthology.orchestra.run.smithi178.stdout: "avg_session_uptime": 6873, 2024-03-19T03:42:51.314 INFO:teuthology.orchestra.run.smithi178.stdout: "mdthresh_evicted": 0 2024-03-19T03:42:51.314 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.314 INFO:teuthology.orchestra.run.smithi178.stdout: "mempool": { 2024-03-19T03:42:51.314 INFO:teuthology.orchestra.run.smithi178.stdout: "bloom_filter_bytes": 0, 2024-03-19T03:42:51.314 INFO:teuthology.orchestra.run.smithi178.stdout: "bloom_filter_items": 0, 2024-03-19T03:42:51.314 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T03:42:51.314 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_alloc_items": 0, 2024-03-19T03:42:51.314 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T03:42:51.314 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_data_items": 0, 2024-03-19T03:42:51.314 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T03:42:51.314 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T03:42:51.315 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T03:42:51.315 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T03:42:51.315 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T03:42:51.315 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_other_items": 0, 2024-03-19T03:42:51.315 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T03:42:51.315 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T03:42:51.315 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_extent_bytes": 0, 2024-03-19T03:42:51.315 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_extent_items": 0, 2024-03-19T03:42:51.315 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_blob_bytes": 0, 2024-03-19T03:42:51.316 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_blob_items": 0, 2024-03-19T03:42:51.316 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T03:42:51.316 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T03:42:51.316 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T03:42:51.316 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T03:42:51.316 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T03:42:51.316 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_fsck_items": 0, 2024-03-19T03:42:51.316 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_txc_bytes": 0, 2024-03-19T03:42:51.316 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_txc_items": 0, 2024-03-19T03:42:51.316 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T03:42:51.316 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T03:42:51.316 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_bytes": 0, 2024-03-19T03:42:51.316 INFO:teuthology.orchestra.run.smithi178.stdout: "bluestore_writing_items": 0, 2024-03-19T03:42:51.316 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_bytes": 0, 2024-03-19T03:42:51.317 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_items": 0, 2024-03-19T03:42:51.317 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T03:42:51.317 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_reader_items": 0, 2024-03-19T03:42:51.317 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T03:42:51.317 INFO:teuthology.orchestra.run.smithi178.stdout: "bluefs_file_writer_items": 0, 2024-03-19T03:42:51.317 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_anon_bytes": 197441, 2024-03-19T03:42:51.317 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_anon_items": 54, 2024-03-19T03:42:51.317 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_meta_bytes": 0, 2024-03-19T03:42:51.317 INFO:teuthology.orchestra.run.smithi178.stdout: "buffer_meta_items": 0, 2024-03-19T03:42:51.318 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_bytes": 0, 2024-03-19T03:42:51.318 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_items": 0, 2024-03-19T03:42:51.318 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mapbl_bytes": 0, 2024-03-19T03:42:51.318 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_mapbl_items": 0, 2024-03-19T03:42:51.318 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pglog_bytes": 0, 2024-03-19T03:42:51.318 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_pglog_items": 0, 2024-03-19T03:42:51.318 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_bytes": 9184, 2024-03-19T03:42:51.318 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_items": 177, 2024-03-19T03:42:51.318 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T03:42:51.318 INFO:teuthology.orchestra.run.smithi178.stdout: "osdmap_mapping_items": 0, 2024-03-19T03:42:51.318 INFO:teuthology.orchestra.run.smithi178.stdout: "pgmap_bytes": 0, 2024-03-19T03:42:51.318 INFO:teuthology.orchestra.run.smithi178.stdout: "pgmap_items": 0, 2024-03-19T03:42:51.318 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_co_bytes": 880536, 2024-03-19T03:42:51.318 INFO:teuthology.orchestra.run.smithi178.stdout: "mds_co_items": 4911, 2024-03-19T03:42:51.319 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_1_bytes": 0, 2024-03-19T03:42:51.319 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_1_items": 0, 2024-03-19T03:42:51.319 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_2_bytes": 0, 2024-03-19T03:42:51.319 INFO:teuthology.orchestra.run.smithi178.stdout: "unittest_2_items": 0 2024-03-19T03:42:51.319 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.319 INFO:teuthology.orchestra.run.smithi178.stdout: "objecter": { 2024-03-19T03:42:51.319 INFO:teuthology.orchestra.run.smithi178.stdout: "op_active": 0, 2024-03-19T03:42:51.319 INFO:teuthology.orchestra.run.smithi178.stdout: "op_laggy": 0, 2024-03-19T03:42:51.322 INFO:teuthology.orchestra.run.smithi178.stdout: "op_send": 2046, 2024-03-19T03:42:51.322 INFO:teuthology.orchestra.run.smithi178.stdout: "op_send_bytes": 21, 2024-03-19T03:42:51.322 INFO:teuthology.orchestra.run.smithi178.stdout: "op_resend": 3, 2024-03-19T03:42:51.322 INFO:teuthology.orchestra.run.smithi178.stdout: "op_reply": 2046, 2024-03-19T03:42:51.322 INFO:teuthology.orchestra.run.smithi178.stdout: "op_latency": { 2024-03-19T03:42:51.322 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 2046, 2024-03-19T03:42:51.322 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 12.257758600, 2024-03-19T03:42:51.322 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.005991084 2024-03-19T03:42:51.322 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.322 INFO:teuthology.orchestra.run.smithi178.stdout: "op_inflight": 0, 2024-03-19T03:42:51.322 INFO:teuthology.orchestra.run.smithi178.stdout: "oplen_avg": { 2024-03-19T03:42:51.323 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 2046, 2024-03-19T03:42:51.323 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 2047 2024-03-19T03:42:51.323 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.323 INFO:teuthology.orchestra.run.smithi178.stdout: "op": 2046, 2024-03-19T03:42:51.323 INFO:teuthology.orchestra.run.smithi178.stdout: "op_r": 2046, 2024-03-19T03:42:51.323 INFO:teuthology.orchestra.run.smithi178.stdout: "op_w": 0, 2024-03-19T03:42:51.323 INFO:teuthology.orchestra.run.smithi178.stdout: "op_rmw": 0, 2024-03-19T03:42:51.323 INFO:teuthology.orchestra.run.smithi178.stdout: "op_pg": 0, 2024-03-19T03:42:51.323 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_stat": 1239, 2024-03-19T03:42:51.323 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_create": 0, 2024-03-19T03:42:51.323 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_read": 805, 2024-03-19T03:42:51.323 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_write": 0, 2024-03-19T03:42:51.323 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_writefull": 0, 2024-03-19T03:42:51.323 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_writesame": 0, 2024-03-19T03:42:51.323 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_append": 0, 2024-03-19T03:42:51.324 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_zero": 0, 2024-03-19T03:42:51.324 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_truncate": 0, 2024-03-19T03:42:51.324 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_delete": 0, 2024-03-19T03:42:51.324 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_mapext": 0, 2024-03-19T03:42:51.324 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_sparse_read": 0, 2024-03-19T03:42:51.324 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_clonerange": 0, 2024-03-19T03:42:51.324 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_getxattr": 1, 2024-03-19T03:42:51.324 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_setxattr": 0, 2024-03-19T03:42:51.324 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_cmpxattr": 0, 2024-03-19T03:42:51.324 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_rmxattr": 0, 2024-03-19T03:42:51.324 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_resetxattrs": 0, 2024-03-19T03:42:51.324 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_call": 0, 2024-03-19T03:42:51.324 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_watch": 0, 2024-03-19T03:42:51.324 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_notify": 0, 2024-03-19T03:42:51.324 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T03:42:51.325 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_pgls": 0, 2024-03-19T03:42:51.325 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_pgls_filter": 0, 2024-03-19T03:42:51.325 INFO:teuthology.orchestra.run.smithi178.stdout: "osdop_other": 0, 2024-03-19T03:42:51.325 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_active": 0, 2024-03-19T03:42:51.325 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_send": 0, 2024-03-19T03:42:51.325 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_resend": 0, 2024-03-19T03:42:51.325 INFO:teuthology.orchestra.run.smithi178.stdout: "linger_ping": 0, 2024-03-19T03:42:51.325 INFO:teuthology.orchestra.run.smithi178.stdout: "poolop_active": 0, 2024-03-19T03:42:51.325 INFO:teuthology.orchestra.run.smithi178.stdout: "poolop_send": 0, 2024-03-19T03:42:51.325 INFO:teuthology.orchestra.run.smithi178.stdout: "poolop_resend": 0, 2024-03-19T03:42:51.325 INFO:teuthology.orchestra.run.smithi178.stdout: "poolstat_active": 0, 2024-03-19T03:42:51.325 INFO:teuthology.orchestra.run.smithi178.stdout: "poolstat_send": 0, 2024-03-19T03:42:51.325 INFO:teuthology.orchestra.run.smithi178.stdout: "poolstat_resend": 0, 2024-03-19T03:42:51.325 INFO:teuthology.orchestra.run.smithi178.stdout: "statfs_active": 0, 2024-03-19T03:42:51.325 INFO:teuthology.orchestra.run.smithi178.stdout: "statfs_send": 0, 2024-03-19T03:42:51.326 INFO:teuthology.orchestra.run.smithi178.stdout: "statfs_resend": 0, 2024-03-19T03:42:51.326 INFO:teuthology.orchestra.run.smithi178.stdout: "command_active": 0, 2024-03-19T03:42:51.326 INFO:teuthology.orchestra.run.smithi178.stdout: "command_send": 0, 2024-03-19T03:42:51.326 INFO:teuthology.orchestra.run.smithi178.stdout: "command_resend": 0, 2024-03-19T03:42:51.326 INFO:teuthology.orchestra.run.smithi178.stdout: "map_epoch": 93, 2024-03-19T03:42:51.326 INFO:teuthology.orchestra.run.smithi178.stdout: "map_full": 0, 2024-03-19T03:42:51.326 INFO:teuthology.orchestra.run.smithi178.stdout: "map_inc": 4, 2024-03-19T03:42:51.326 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_sessions": 6, 2024-03-19T03:42:51.326 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_session_open": 6, 2024-03-19T03:42:51.326 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_session_close": 0, 2024-03-19T03:42:51.326 INFO:teuthology.orchestra.run.smithi178.stdout: "osd_laggy": 0, 2024-03-19T03:42:51.326 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_wr": 0, 2024-03-19T03:42:51.326 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_rd": 2, 2024-03-19T03:42:51.326 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_del": 0 2024-03-19T03:42:51.327 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.327 INFO:teuthology.orchestra.run.smithi178.stdout: "oft": { 2024-03-19T03:42:51.327 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_objs": 0, 2024-03-19T03:42:51.327 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_kv_pairs": 0, 2024-03-19T03:42:51.327 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_updates": 0, 2024-03-19T03:42:51.327 INFO:teuthology.orchestra.run.smithi178.stdout: "omap_total_removes": 0 2024-03-19T03:42:51.327 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.327 INFO:teuthology.orchestra.run.smithi178.stdout: "osd-slow-ops": { 2024-03-19T03:42:51.327 INFO:teuthology.orchestra.run.smithi178.stdout: "slow_ops_count": 0 2024-03-19T03:42:51.327 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.327 INFO:teuthology.orchestra.run.smithi178.stdout: "purge_queue": { 2024-03-19T03:42:51.327 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing_ops": 0, 2024-03-19T03:42:51.327 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing_ops_high_water": 0, 2024-03-19T03:42:51.327 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing": 0, 2024-03-19T03:42:51.328 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executing_high_water": 0, 2024-03-19T03:42:51.328 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executed_ops": 0, 2024-03-19T03:42:51.328 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_executed": 0, 2024-03-19T03:42:51.328 INFO:teuthology.orchestra.run.smithi178.stdout: "pq_item_in_journal": 0 2024-03-19T03:42:51.328 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.328 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T03:42:51.328 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-19T03:42:51.328 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 104857600, 2024-03-19T03:42:51.328 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-19T03:42:51.328 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 2288, 2024-03-19T03:42:51.328 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 10881771, 2024-03-19T03:42:51.328 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:51.328 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 2288, 2024-03-19T03:42:51.328 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 0, 2024-03-19T03:42:51.329 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 0, 2024-03-19T03:42:51.329 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 2288, 2024-03-19T03:42:51.329 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 10881771, 2024-03-19T03:42:51.329 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-19T03:42:51.329 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.329 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.329 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.329 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:51.329 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.329 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-objecter_bytes": { 2024-03-19T03:42:51.329 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-19T03:42:51.329 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 104857600, 2024-03-19T03:42:51.329 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-19T03:42:51.330 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-19T03:42:51.330 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-19T03:42:51.330 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:51.330 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:51.330 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 2046, 2024-03-19T03:42:51.330 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 9639308, 2024-03-19T03:42:51.330 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 2046, 2024-03-19T03:42:51.330 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 9639308, 2024-03-19T03:42:51.330 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-19T03:42:51.330 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.330 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.330 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.330 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:51.330 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.331 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-objecter_ops": { 2024-03-19T03:42:51.331 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-19T03:42:51.331 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 1024, 2024-03-19T03:42:51.331 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-19T03:42:51.331 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-19T03:42:51.331 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-19T03:42:51.331 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:51.331 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:51.331 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 2046, 2024-03-19T03:42:51.331 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 2046, 2024-03-19T03:42:51.331 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 2046, 2024-03-19T03:42:51.331 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 2046, 2024-03-19T03:42:51.331 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-19T03:42:51.332 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.332 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.332 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.332 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:51.332 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.332 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-write_buf_throttle": { 2024-03-19T03:42:51.332 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-19T03:42:51.332 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 3758096384, 2024-03-19T03:42:51.332 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-19T03:42:51.332 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-19T03:42:51.332 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-19T03:42:51.332 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:51.332 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:51.332 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 0, 2024-03-19T03:42:51.333 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 0, 2024-03-19T03:42:51.333 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 0, 2024-03-19T03:42:51.333 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 0, 2024-03-19T03:42:51.333 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-19T03:42:51.333 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.333 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.333 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.333 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:51.333 INFO:teuthology.orchestra.run.smithi178.stdout: }, 2024-03-19T03:42:51.333 INFO:teuthology.orchestra.run.smithi178.stdout: "throttle-write_buf_throttle-0x564a806826e0": { 2024-03-19T03:42:51.333 INFO:teuthology.orchestra.run.smithi178.stdout: "val": 0, 2024-03-19T03:42:51.333 INFO:teuthology.orchestra.run.smithi178.stdout: "max": 3758096384, 2024-03-19T03:42:51.333 INFO:teuthology.orchestra.run.smithi178.stdout: "get_started": 0, 2024-03-19T03:42:51.333 INFO:teuthology.orchestra.run.smithi178.stdout: "get": 0, 2024-03-19T03:42:51.334 INFO:teuthology.orchestra.run.smithi178.stdout: "get_sum": 0, 2024-03-19T03:42:51.334 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_fail": 0, 2024-03-19T03:42:51.334 INFO:teuthology.orchestra.run.smithi178.stdout: "get_or_fail_success": 0, 2024-03-19T03:42:51.334 INFO:teuthology.orchestra.run.smithi178.stdout: "take": 0, 2024-03-19T03:42:51.334 INFO:teuthology.orchestra.run.smithi178.stdout: "take_sum": 0, 2024-03-19T03:42:51.334 INFO:teuthology.orchestra.run.smithi178.stdout: "put": 0, 2024-03-19T03:42:51.334 INFO:teuthology.orchestra.run.smithi178.stdout: "put_sum": 0, 2024-03-19T03:42:51.334 INFO:teuthology.orchestra.run.smithi178.stdout: "wait": { 2024-03-19T03:42:51.334 INFO:teuthology.orchestra.run.smithi178.stdout: "avgcount": 0, 2024-03-19T03:42:51.334 INFO:teuthology.orchestra.run.smithi178.stdout: "sum": 0.000000000, 2024-03-19T03:42:51.334 INFO:teuthology.orchestra.run.smithi178.stdout: "avgtime": 0.000000000 2024-03-19T03:42:51.334 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:51.334 INFO:teuthology.orchestra.run.smithi178.stdout: } 2024-03-19T03:42:51.334 INFO:teuthology.orchestra.run.smithi178.stdout:} 2024-03-19T03:42:52.062 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-03-19T03:42:52.062 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-03-19T03:42:52.062 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=0 2024-03-19T03:42:52.063 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=0 2024-03-19T03:42:52.063 INFO:tasks.check_counter:Daemon mds.l mds.dir_split=0 2024-03-19T03:42:52.063 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-19T03:42:52.073 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-19T03:42:52.073 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-19T03:42:52.073 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T03:42:52.133 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-19T03:42:52.133 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T03:42:52.181 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:42:52.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.425+0000 7fbc0b000640 1 -- 172.21.15.117:0/932222380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc04103a60 msgr2=0x7fbc04103e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:52.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.425+0000 7fbc0b000640 1 --2- 172.21.15.117:0/932222380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc04103a60 0x7fbc04103e80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4009860 tx=0x7fbbf402bb70 comp rx=0 tx=0).stop 2024-03-19T03:42:52.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.425+0000 7fbc0b000640 1 -- 172.21.15.117:0/932222380 shutdown_connections 2024-03-19T03:42:52.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.425+0000 7fbc0b000640 1 --2- 172.21.15.117:0/932222380 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc04105610 0x7fbc0410bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:52.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.425+0000 7fbc0b000640 1 --2- 172.21.15.117:0/932222380 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc04104c50 0x7fbc041050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:52.425 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.425+0000 7fbc0b000640 1 --2- 172.21.15.117:0/932222380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc04103a60 0x7fbc04103e80 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:52.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.425+0000 7fbc0b000640 1 -- 172.21.15.117:0/932222380 >> 172.21.15.117:0/932222380 conn(0x7fbc040ff010 msgr2=0x7fbc04101430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:52.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.425+0000 7fbc0b000640 1 -- 172.21.15.117:0/932222380 shutdown_connections 2024-03-19T03:42:52.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.425+0000 7fbc0b000640 1 -- 172.21.15.117:0/932222380 wait complete. 2024-03-19T03:42:52.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.426+0000 7fbc0b000640 1 Processor -- start 2024-03-19T03:42:52.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.427+0000 7fbc0b000640 1 -- start start 2024-03-19T03:42:52.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.427+0000 7fbc0b000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc04103a60 0x7fbc04199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:52.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.427+0000 7fbc0b000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc04104c50 0x7fbc0419a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:52.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.427+0000 7fbc0b000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc04105610 0x7fbc041a1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:52.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.427+0000 7fbc0b000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0410f6d0 con 0x7fbc04104c50 2024-03-19T03:42:52.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.427+0000 7fbc0b000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0410f550 con 0x7fbc04103a60 2024-03-19T03:42:52.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.427+0000 7fbc0b000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0410f850 con 0x7fbc04105610 2024-03-19T03:42:52.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.427+0000 7fbc03e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc04103a60 0x7fbc04199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:52.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.427+0000 7fbc03e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc04103a60 0x7fbc04199dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:50420/0 (socket says 172.21.15.117:50420) 2024-03-19T03:42:52.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.427+0000 7fbc03e00640 1 -- 172.21.15.117:0/2713750970 learned_addr learned my addr 172.21.15.117:0/2713750970 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:42:52.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.427+0000 7fbbfbe00640 1 --2- 172.21.15.117:0/2713750970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc04104c50 0x7fbc0419a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:52.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.428+0000 7fbc09000640 1 --2- 172.21.15.117:0/2713750970 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc04105610 0x7fbc041a1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:52.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.428+0000 7fbc03e00640 1 -- 172.21.15.117:0/2713750970 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc04105610 msgr2=0x7fbc041a1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:52.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.428+0000 7fbc03e00640 1 --2- 172.21.15.117:0/2713750970 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc04105610 0x7fbc041a1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:52.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.428+0000 7fbc03e00640 1 -- 172.21.15.117:0/2713750970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc04104c50 msgr2=0x7fbc0419a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:52.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.428+0000 7fbc03e00640 1 --2- 172.21.15.117:0/2713750970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc04104c50 0x7fbc0419a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:52.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.428+0000 7fbc03e00640 1 -- 172.21.15.117:0/2713750970 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc041a19f0 con 0x7fbc04103a60 2024-03-19T03:42:52.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.428+0000 7fbc09000640 1 --2- 172.21.15.117:0/2713750970 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc04105610 0x7fbc041a1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:42:52.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.428+0000 7fbc03e00640 1 --2- 172.21.15.117:0/2713750970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc04103a60 0x7fbc04199dd0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbbf402c0b0 tx=0x7fbbf4032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:52.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.428+0000 7fbc01600640 1 -- 172.21.15.117:0/2713750970 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf4041020 con 0x7fbc04103a60 2024-03-19T03:42:52.429 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.428+0000 7fbc01600640 1 -- 172.21.15.117:0/2713750970 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbf402c9f0 con 0x7fbc04103a60 2024-03-19T03:42:52.430 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.428+0000 7fbc01600640 1 -- 172.21.15.117:0/2713750970 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf4039040 con 0x7fbc04103a60 2024-03-19T03:42:52.430 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.428+0000 7fbc0b000640 1 -- 172.21.15.117:0/2713750970 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc041a1c80 con 0x7fbc04103a60 2024-03-19T03:42:52.430 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.429+0000 7fbc0b000640 1 -- 172.21.15.117:0/2713750970 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc041a21c0 con 0x7fbc04103a60 2024-03-19T03:42:52.430 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.430+0000 7fbc0b000640 1 -- 172.21.15.117:0/2713750970 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc0410adf0 con 0x7fbc04103a60 2024-03-19T03:42:52.433 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.433+0000 7fbc01600640 1 -- 172.21.15.117:0/2713750970 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 20) ==== 137291+0+0 (secure 0 0 0) 0x7fbbf4044020 con 0x7fbc04103a60 2024-03-19T03:42:52.434 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.434+0000 7fbc01600640 1 --2- 172.21.15.117:0/2713750970 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fbbe80a2880 0x7fbbe80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:52.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.435+0000 7fbc01600640 1 -- 172.21.15.117:0/2713750970 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fbbf40f0760 con 0x7fbc04103a60 2024-03-19T03:42:52.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.435+0000 7fbbfbe00640 1 --2- 172.21.15.117:0/2713750970 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fbbe80a2880 0x7fbbe80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:52.435 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.436+0000 7fbbfbe00640 1 --2- 172.21.15.117:0/2713750970 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fbbe80a2880 0x7fbbe80a4d40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbc0419b2e0 tx=0x7fbbec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:52.442 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.443+0000 7fbc01600640 1 -- 172.21.15.117:0/2713750970 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbbf404b020 con 0x7fbc04103a60 2024-03-19T03:42:52.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.643+0000 7fbc0b000640 1 -- 172.21.15.117:0/2713750970 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbc04062420 con 0x7fbc04103a60 2024-03-19T03:42:52.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.645+0000 7fbc01600640 1 -- 172.21.15.117:0/2713750970 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24989 (secure 0 0 0) 0x7fbbf40bd410 con 0x7fbc04103a60 2024-03-19T03:42:52.646 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:42:52.646 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":93,"fsid":"a3a82456-e59f-11ee-95c9-87774f69a715","created":"2024-03-19T03:21:30.427136+0000","modified":"2024-03-19T03:32:40.453890+0000","last_up_change":"2024-03-19T03:29:32.967505+0000","last_in_change":"2024-03-19T03:29:12.848740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:25:18.824270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T03:30:33.137170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T03:30:34.598838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T03:30:38.230023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b32eb0ee-9d4b-4748-87ec-eda02c907415","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6803","nonce":3014934900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6805","nonce":3014934900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6809","nonce":3014934900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6807","nonce":3014934900}]},"public_addr":"172.21.15.117:6803/3014934900","cluster_addr":"172.21.15.117:6805/3014934900","heartbeat_back_addr":"172.21.15.117:6809/3014934900","heartbeat_front_addr":"172.21.15.117:6807/3014934900","state":["exists","up"]},{"osd":1,"uuid":"e07585be-c016-4466-b07c-abbc22a58de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6801","nonce":2425927341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6803","nonce":2425927341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6807","nonce":2425927341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6805","nonce":2425927341}]},"public_addr":"172.21.15.119:6801/2425927341","cluster_addr":"172.21.15.119:6803/2425927341","heartbeat_back_addr":"172.21.15.119:6807/2425927341","heartbeat_front_addr":"172.21.15.119:6805/2425927341","state":["exists","up"]},{"osd":2,"uuid":"ede9edda-3b9f-41ca-ad04-5dcb738ab757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6801","nonce":3733812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6803","nonce":3733812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6807","nonce":3733812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6805","nonce":3733812989}]},"public_addr":"172.21.15.178:6801/3733812989","cluster_addr":"172.21.15.178:6803/3733812989","heartbeat_back_addr":"172.21.15.178:6807/3733812989","heartbeat_front_addr":"172.21.15.178:6805/3733812989","state":["exists","up"]},{"osd":3,"uuid":"f81c41e1-3a69-40db-a90f-58151286b081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6811","nonce":2178310135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6813","nonce":2178310135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6817","nonce":2178310135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6815","nonce":2178310135}]},"public_addr":"172.21.15.117:6811/2178310135","cluster_addr":"172.21.15.117:6813/2178310135","heartbeat_back_addr":"172.21.15.117:6817/2178310135","heartbeat_front_addr":"172.21.15.117:6815/2178310135","state":["exists","up"]},{"osd":4,"uuid":"caf7ce86-caf0-4eaf-bbab-bd2d93e63c33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6809","nonce":3455398086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6811","nonce":3455398086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6815","nonce":3455398086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6813","nonce":3455398086}]},"public_addr":"172.21.15.119:6809/3455398086","cluster_addr":"172.21.15.119:6811/3455398086","heartbeat_back_addr":"172.21.15.119:6815/3455398086","heartbeat_front_addr":"172.21.15.119:6813/3455398086","state":["exists","up"]},{"osd":5,"uuid":"20ba0b8a-2e0f-4ea7-b435-430e44766200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6809","nonce":1259631912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6811","nonce":1259631912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6815","nonce":1259631912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6813","nonce":1259631912}]},"public_addr":"172.21.15.178:6809/1259631912","cluster_addr":"172.21.15.178:6811/1259631912","heartbeat_back_addr":"172.21.15.178:6815/1259631912","heartbeat_front_addr":"172.21.15.178:6813/1259631912","state":["exists","up"]},{"osd":6,"uuid":"75c2ddd7-f837-42e0-ac47-8cfbcf97a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6819","nonce":864146757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6821","nonce":864146757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6825","nonce":864146757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6823","nonce":864146757}]},"public_addr":"172.21.15.117:6819/864146757","cluster_addr":"172.21.15.117:6821/864146757","heartbeat_back_addr":"172.21.15.117:6825/864146757","heartbeat_front_addr":"172.21.15.117:6823/864146757","state":["exists","up"]},{"osd":7,"uuid":"929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6817","nonce":1273365545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6819","nonce":1273365545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6823","nonce":1273365545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6821","nonce":1273365545}]},"public_addr":"172.21.15.119:6817/1273365545","cluster_addr":"172.21.15.119:6819/1273365545","heartbeat_back_addr":"172.21.15.119:6823/1273365545","heartbeat_front_addr":"172.21.15.119:6821/1273365545","state":["exists","up"]},{"osd":8,"uuid":"fd371461-a538-4781-92e1-5eead4ca3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6817","nonce":1059587501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6819","nonce":1059587501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6823","nonce":1059587501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6821","nonce":1059587501}]},"public_addr":"172.21.15.178:6817/1059587501","cluster_addr":"172.21.15.178:6819/1059587501","heartbeat_back_addr":"172.21.15.178:6823/1059587501","heartbeat_front_addr":"172.21.15.178:6821/1059587501","state":["exists","up"]},{"osd":9,"uuid":"0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6827","nonce":3653875600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6829","nonce":3653875600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6833","nonce":3653875600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6831","nonce":3653875600}]},"public_addr":"172.21.15.117:6827/3653875600","cluster_addr":"172.21.15.117:6829/3653875600","heartbeat_back_addr":"172.21.15.117:6833/3653875600","heartbeat_front_addr":"172.21.15.117:6831/3653875600","state":["exists","up"]},{"osd":10,"uuid":"de2049da-d19e-4386-afbf-fd6b5fddbcba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6825","nonce":1050976194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6827","nonce":1050976194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6831","nonce":1050976194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6829","nonce":1050976194}]},"public_addr":"172.21.15.119:6825/1050976194","cluster_addr":"172.21.15.119:6827/1050976194","heartbeat_back_addr":"172.21.15.119:6831/1050976194","heartbeat_front_addr":"172.21.15.119:6829/1050976194","state":["exists","up"]},{"osd":11,"uuid":"3cccc0f9-d273-4b6b-9b8e-c926371472dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6825","nonce":2915080368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6827","nonce":2915080368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6831","nonce":2915080368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6829","nonce":2915080368}]},"public_addr":"172.21.15.178:6825/2915080368","cluster_addr":"172.21.15.178:6827/2915080368","heartbeat_back_addr":"172.21.15.178:6831/2915080368","heartbeat_front_addr":"172.21.15.178:6829/2915080368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:27.876274+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:53.094706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:16.598813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:48.160419+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:14.048925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:38.814304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:12.542328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:38.942747+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:04.488963+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:38.196053+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:05.254728+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:30.874915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7},{"from":5,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.117:0/185684447":"2024-03-20T03:22:38.134578+0000","172.21.15.117:0/361367101":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/2788067603":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3506200675":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6800/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3283471541":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3750920169":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/2780813298":"2024-03-20T03:22:03.905461+0000","172.21.15.117:6800/2788067603":"2024-03-20T03:22:38.134578+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:42:52.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.649+0000 7fbc0b000640 1 -- 172.21.15.117:0/2713750970 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fbbe80a2880 msgr2=0x7fbbe80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:52.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.649+0000 7fbc0b000640 1 --2- 172.21.15.117:0/2713750970 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fbbe80a2880 0x7fbbe80a4d40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbc0419b2e0 tx=0x7fbbec005f90 comp rx=0 tx=0).stop 2024-03-19T03:42:52.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.649+0000 7fbc0b000640 1 -- 172.21.15.117:0/2713750970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc04103a60 msgr2=0x7fbc04199dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:52.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.649+0000 7fbc0b000640 1 --2- 172.21.15.117:0/2713750970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc04103a60 0x7fbc04199dd0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbbf402c0b0 tx=0x7fbbf4032430 comp rx=0 tx=0).stop 2024-03-19T03:42:52.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.649+0000 7fbc0b000640 1 -- 172.21.15.117:0/2713750970 shutdown_connections 2024-03-19T03:42:52.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.649+0000 7fbc0b000640 1 --2- 172.21.15.117:0/2713750970 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc04105610 0x7fbc041a1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:52.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.649+0000 7fbc0b000640 1 --2- 172.21.15.117:0/2713750970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc04104c50 0x7fbc0419a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:52.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.649+0000 7fbc0b000640 1 --2- 172.21.15.117:0/2713750970 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fbbe80a2880 0x7fbbe80a4d40 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:52.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.649+0000 7fbc0b000640 1 --2- 172.21.15.117:0/2713750970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc04103a60 0x7fbc04199dd0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:52.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.649+0000 7fbc0b000640 1 -- 172.21.15.117:0/2713750970 >> 172.21.15.117:0/2713750970 conn(0x7fbc040ff010 msgr2=0x7fbc04107fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:52.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.649+0000 7fbc0b000640 1 -- 172.21.15.117:0/2713750970 shutdown_connections 2024-03-19T03:42:52.650 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.649+0000 7fbc0b000640 1 -- 172.21.15.117:0/2713750970 wait complete. 2024-03-19T03:42:52.669 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T03:25:18.824270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T03:30:33.137170+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T03:30:34.598838+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-19T03:30:38.230023+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-19T03:42:52.669 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T03:42:52.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.868+0000 7f5856800640 1 -- 172.21.15.117:0/2495804089 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f58501032d0 msgr2=0x7f5850109b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:52.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.868+0000 7f5856800640 1 --2- 172.21.15.117:0/2495804089 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f58501032d0 0x7f5850109b60 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5848009ca0 tx=0x7f584802c0f0 comp rx=0 tx=0).stop 2024-03-19T03:42:52.868 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.868+0000 7f5856800640 1 -- 172.21.15.117:0/2495804089 shutdown_connections 2024-03-19T03:42:52.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.868+0000 7f5856800640 1 --2- 172.21.15.117:0/2495804089 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f58501032d0 0x7f5850109b60 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:52.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.868+0000 7f5856800640 1 --2- 172.21.15.117:0/2495804089 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5850102910 0x7f5850102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:52.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.868+0000 7f5856800640 1 --2- 172.21.15.117:0/2495804089 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5850101720 0x7f5850101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:52.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.868+0000 7f5856800640 1 -- 172.21.15.117:0/2495804089 >> 172.21.15.117:0/2495804089 conn(0x7f58500fcc90 msgr2=0x7f58500ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:52.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.868+0000 7f5856800640 1 -- 172.21.15.117:0/2495804089 shutdown_connections 2024-03-19T03:42:52.869 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.868+0000 7f5856800640 1 -- 172.21.15.117:0/2495804089 wait complete. 2024-03-19T03:42:52.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.869+0000 7f5856800640 1 Processor -- start 2024-03-19T03:42:52.871 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.869+0000 7f5856800640 1 -- start start 2024-03-19T03:42:52.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.869+0000 7f5856800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5850101720 0x7f58500755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:52.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.869+0000 7f5856800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5850102910 0x7f5850073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:52.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.869+0000 7f5856800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f58501032d0 0x7f5850074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:52.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.869+0000 7f5856800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f585010d380 con 0x7f58501032d0 2024-03-19T03:42:52.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.869+0000 7f5856800640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f585010d200 con 0x7f5850101720 2024-03-19T03:42:52.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.869+0000 7f5856800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f585010d500 con 0x7f5850102910 2024-03-19T03:42:52.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.869+0000 7f5855400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5850101720 0x7f58500755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:52.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.870+0000 7f5855400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5850101720 0x7f58500755d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:50442/0 (socket says 172.21.15.117:50442) 2024-03-19T03:42:52.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.870+0000 7f5855400640 1 -- 172.21.15.117:0/3205909288 learned_addr learned my addr 172.21.15.117:0/3205909288 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:42:52.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.870+0000 7f5855e00640 1 --2- 172.21.15.117:0/3205909288 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f58501032d0 0x7f5850074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:52.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.870+0000 7f5854a00640 1 --2- 172.21.15.117:0/3205909288 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5850102910 0x7f5850073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:52.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.870+0000 7f5855400640 1 -- 172.21.15.117:0/3205909288 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5850102910 msgr2=0x7f5850073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:52.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.870+0000 7f5855400640 1 --2- 172.21.15.117:0/3205909288 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5850102910 0x7f5850073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:52.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.870+0000 7f5855400640 1 -- 172.21.15.117:0/3205909288 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f58501032d0 msgr2=0x7f5850074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:52.872 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.870+0000 7f5855400640 1 --2- 172.21.15.117:0/3205909288 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f58501032d0 0x7f5850074150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:52.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.870+0000 7f5855400640 1 -- 172.21.15.117:0/3205909288 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58500749e0 con 0x7f5850101720 2024-03-19T03:42:52.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.870+0000 7f5854a00640 1 --2- 172.21.15.117:0/3205909288 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5850102910 0x7f5850073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:42:52.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.870+0000 7f5855400640 1 --2- 172.21.15.117:0/3205909288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5850101720 0x7f58500755d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f584400d950 tx=0x7f584400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:52.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.870+0000 7f584e000640 1 -- 172.21.15.117:0/3205909288 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5844017020 con 0x7f5850101720 2024-03-19T03:42:52.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.871+0000 7f584e000640 1 -- 172.21.15.117:0/3205909288 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5844009c30 con 0x7f5850101720 2024-03-19T03:42:52.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.871+0000 7f5856800640 1 -- 172.21.15.117:0/3205909288 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58501a1a60 con 0x7f5850101720 2024-03-19T03:42:52.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.871+0000 7f584e000640 1 -- 172.21.15.117:0/3205909288 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584400f720 con 0x7f5850101720 2024-03-19T03:42:52.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.871+0000 7f5856800640 1 -- 172.21.15.117:0/3205909288 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58501a2000 con 0x7f5850101720 2024-03-19T03:42:52.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.872+0000 7f5856800640 1 -- 172.21.15.117:0/3205909288 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5850108a40 con 0x7f5850101720 2024-03-19T03:42:52.874 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.873+0000 7f584e000640 1 -- 172.21.15.117:0/3205909288 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 20) ==== 137291+0+0 (secure 0 0 0) 0x7f584401a020 con 0x7f5850101720 2024-03-19T03:42:52.875 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.876+0000 7f584e000640 1 --2- 172.21.15.117:0/3205909288 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f58380a2880 0x7f58380a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:52.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.876+0000 7f584e000640 1 -- 172.21.15.117:0/3205909288 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f58440cedc0 con 0x7f5850101720 2024-03-19T03:42:52.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.876+0000 7f5854a00640 1 --2- 172.21.15.117:0/3205909288 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f58380a2880 0x7f58380a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:52.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.876+0000 7f5854a00640 1 --2- 172.21.15.117:0/3205909288 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f58380a2880 0x7f58380a4d40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5850075340 tx=0x7f583c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:52.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:52.879+0000 7f584e000640 1 -- 172.21.15.117:0/3205909288 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5844021020 con 0x7f5850101720 2024-03-19T03:42:52.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:52 smithi119 ceph-mon[66861]: pgmap v690: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-19T03:42:52.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:52 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2713750970' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:42:53.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:52 smithi117 ceph-mon[31838]: pgmap v690: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-19T03:42:53.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:52 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2713750970' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:42:53.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.087+0000 7f5856800640 1 -- 172.21.15.117:0/3205909288 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f585002d1f0 con 0x7f5850101720 2024-03-19T03:42:53.087 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.088+0000 7f584e000640 1 -- 172.21.15.117:0/3205909288 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f5844009de0 con 0x7f5850101720 2024-03-19T03:42:53.088 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 1 2024-03-19T03:42:53.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.091+0000 7f5856800640 1 -- 172.21.15.117:0/3205909288 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f58380a2880 msgr2=0x7f58380a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:53.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.091+0000 7f5856800640 1 --2- 172.21.15.117:0/3205909288 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f58380a2880 0x7f58380a4d40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5850075340 tx=0x7f583c005f90 comp rx=0 tx=0).stop 2024-03-19T03:42:53.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.091+0000 7f5856800640 1 -- 172.21.15.117:0/3205909288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5850101720 msgr2=0x7f58500755d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:53.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.091+0000 7f5856800640 1 --2- 172.21.15.117:0/3205909288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5850101720 0x7f58500755d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f584400d950 tx=0x7f584400de10 comp rx=0 tx=0).stop 2024-03-19T03:42:53.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.091+0000 7f5856800640 1 -- 172.21.15.117:0/3205909288 shutdown_connections 2024-03-19T03:42:53.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.091+0000 7f5856800640 1 --2- 172.21.15.117:0/3205909288 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f58501032d0 0x7f5850074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:53.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.091+0000 7f5856800640 1 --2- 172.21.15.117:0/3205909288 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5850102910 0x7f5850073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:53.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.091+0000 7f5856800640 1 --2- 172.21.15.117:0/3205909288 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f58380a2880 0x7f58380a4d40 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:53.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.091+0000 7f5856800640 1 --2- 172.21.15.117:0/3205909288 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5850101720 0x7f58500755d0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:53.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.091+0000 7f5856800640 1 -- 172.21.15.117:0/3205909288 >> 172.21.15.117:0/3205909288 conn(0x7f58500fcc90 msgr2=0x7f5850105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:53.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.091+0000 7f5856800640 1 -- 172.21.15.117:0/3205909288 shutdown_connections 2024-03-19T03:42:53.092 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.091+0000 7f5856800640 1 -- 172.21.15.117:0/3205909288 wait complete. 2024-03-19T03:42:53.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:52 smithi178 ceph-mon[67578]: pgmap v690: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-19T03:42:53.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:52 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2713750970' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:42:53.110 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T03:42:53.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.349+0000 7f521fe00640 1 -- 172.21.15.117:0/408586673 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5220103920 msgr2=0x7f5220103d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:53.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.349+0000 7f521fe00640 1 --2- 172.21.15.117:0/408586673 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5220103920 0x7f5220103d40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5210009900 tx=0x7f521002bc30 comp rx=0 tx=0).stop 2024-03-19T03:42:53.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.349+0000 7f521fe00640 1 -- 172.21.15.117:0/408586673 shutdown_connections 2024-03-19T03:42:53.350 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.349+0000 7f521fe00640 1 --2- 172.21.15.117:0/408586673 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f52201054d0 0x7f522010bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:53.350 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.349+0000 7f521fe00640 1 --2- 172.21.15.117:0/408586673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5220104b10 0x7f5220104f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:53.350 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.349+0000 7f521fe00640 1 --2- 172.21.15.117:0/408586673 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5220103920 0x7f5220103d40 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:53.350 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.349+0000 7f521fe00640 1 -- 172.21.15.117:0/408586673 >> 172.21.15.117:0/408586673 conn(0x7f52200fee90 msgr2=0x7f52201012d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:53.350 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.349+0000 7f521fe00640 1 -- 172.21.15.117:0/408586673 shutdown_connections 2024-03-19T03:42:53.350 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.349+0000 7f521fe00640 1 -- 172.21.15.117:0/408586673 wait complete. 2024-03-19T03:42:53.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.350+0000 7f521fe00640 1 Processor -- start 2024-03-19T03:42:53.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.351+0000 7f521fe00640 1 -- start start 2024-03-19T03:42:53.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.351+0000 7f521fe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5220103920 0x7f5220199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:53.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.351+0000 7f521fe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5220104b10 0x7f522019a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:53.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.351+0000 7f521fe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f52201054d0 0x7f52201a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:53.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.351+0000 7f521fe00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f522010f5f0 con 0x7f5220104b10 2024-03-19T03:42:53.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.351+0000 7f521fe00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f522010f470 con 0x7f5220103920 2024-03-19T03:42:53.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.351+0000 7f521fe00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f522010f770 con 0x7f52201054d0 2024-03-19T03:42:53.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.351+0000 7f521ea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5220103920 0x7f5220199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:53.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.351+0000 7f521ea00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5220103920 0x7f5220199c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:50456/0 (socket says 172.21.15.117:50456) 2024-03-19T03:42:53.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.351+0000 7f521ea00640 1 -- 172.21.15.117:0/2021117909 learned_addr learned my addr 172.21.15.117:0/2021117909 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:42:53.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.351+0000 7f5217e00640 1 --2- 172.21.15.117:0/2021117909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5220104b10 0x7f522019a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:53.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.351+0000 7f521f400640 1 --2- 172.21.15.117:0/2021117909 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f52201054d0 0x7f52201a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:53.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.352+0000 7f521ea00640 1 -- 172.21.15.117:0/2021117909 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f52201054d0 msgr2=0x7f52201a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:53.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.352+0000 7f521ea00640 1 --2- 172.21.15.117:0/2021117909 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f52201054d0 0x7f52201a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:53.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.352+0000 7f521ea00640 1 -- 172.21.15.117:0/2021117909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5220104b10 msgr2=0x7f522019a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:53.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.352+0000 7f521ea00640 1 --2- 172.21.15.117:0/2021117909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5220104b10 0x7f522019a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:53.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.352+0000 7f521ea00640 1 -- 172.21.15.117:0/2021117909 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52201a1840 con 0x7f5220103920 2024-03-19T03:42:53.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.352+0000 7f521ea00640 1 --2- 172.21.15.117:0/2021117909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5220103920 0x7f5220199c30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5210000c00 tx=0x7f521002c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:53.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.352+0000 7f5217400640 1 -- 172.21.15.117:0/2021117909 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5210043020 con 0x7f5220103920 2024-03-19T03:42:53.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.352+0000 7f5217400640 1 -- 172.21.15.117:0/2021117909 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52100054e0 con 0x7f5220103920 2024-03-19T03:42:53.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.352+0000 7f5217400640 1 -- 172.21.15.117:0/2021117909 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5210003ce0 con 0x7f5220103920 2024-03-19T03:42:53.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.352+0000 7f521fe00640 1 -- 172.21.15.117:0/2021117909 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52201a1ad0 con 0x7f5220103920 2024-03-19T03:42:53.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.353+0000 7f521fe00640 1 -- 172.21.15.117:0/2021117909 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52201a2040 con 0x7f5220103920 2024-03-19T03:42:53.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.354+0000 7f521fe00640 1 -- 172.21.15.117:0/2021117909 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5220068530 con 0x7f5220103920 2024-03-19T03:42:53.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.357+0000 7f5217400640 1 -- 172.21.15.117:0/2021117909 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 20) ==== 137291+0+0 (secure 0 0 0) 0x7f5210046020 con 0x7f5220103920 2024-03-19T03:42:53.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.359+0000 7f5217400640 1 --2- 172.21.15.117:0/2021117909 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f52040a2880 0x7f52040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:53.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.359+0000 7f5217400640 1 -- 172.21.15.117:0/2021117909 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f52100f0f90 con 0x7f5220103920 2024-03-19T03:42:53.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.359+0000 7f5217e00640 1 --2- 172.21.15.117:0/2021117909 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f52040a2880 0x7f52040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:53.360 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.360+0000 7f5217e00640 1 --2- 172.21.15.117:0/2021117909 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f52040a2880 0x7f52040a4d40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f52080095f0 tx=0x7f5208005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:53.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.362+0000 7f5217400640 1 -- 172.21.15.117:0/2021117909 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f521004d020 con 0x7f5220103920 2024-03-19T03:42:53.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.567+0000 7f521fe00640 1 -- 172.21.15.117:0/2021117909 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5220062420 con 0x7f5220103920 2024-03-19T03:42:53.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.567+0000 7f5217400640 1 -- 172.21.15.117:0/2021117909 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f52100bdbe0 con 0x7f5220103920 2024-03-19T03:42:53.567 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 32 2024-03-19T03:42:53.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.571+0000 7f521fe00640 1 -- 172.21.15.117:0/2021117909 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f52040a2880 msgr2=0x7f52040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:53.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.571+0000 7f521fe00640 1 --2- 172.21.15.117:0/2021117909 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f52040a2880 0x7f52040a4d40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f52080095f0 tx=0x7f5208005f90 comp rx=0 tx=0).stop 2024-03-19T03:42:53.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.571+0000 7f521fe00640 1 -- 172.21.15.117:0/2021117909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5220103920 msgr2=0x7f5220199c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:53.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.571+0000 7f521fe00640 1 --2- 172.21.15.117:0/2021117909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5220103920 0x7f5220199c30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5210000c00 tx=0x7f521002c6c0 comp rx=0 tx=0).stop 2024-03-19T03:42:53.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.571+0000 7f521fe00640 1 -- 172.21.15.117:0/2021117909 shutdown_connections 2024-03-19T03:42:53.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.571+0000 7f521fe00640 1 --2- 172.21.15.117:0/2021117909 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f52201054d0 0x7f52201a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:53.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.571+0000 7f521fe00640 1 --2- 172.21.15.117:0/2021117909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5220104b10 0x7f522019a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:53.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.571+0000 7f521fe00640 1 --2- 172.21.15.117:0/2021117909 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f52040a2880 0x7f52040a4d40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:53.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.571+0000 7f521fe00640 1 --2- 172.21.15.117:0/2021117909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5220103920 0x7f5220199c30 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:53.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.571+0000 7f521fe00640 1 -- 172.21.15.117:0/2021117909 >> 172.21.15.117:0/2021117909 conn(0x7f52200fee90 msgr2=0x7f5220107e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:53.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.571+0000 7f521fe00640 1 -- 172.21.15.117:0/2021117909 shutdown_connections 2024-03-19T03:42:53.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.571+0000 7f521fe00640 1 -- 172.21.15.117:0/2021117909 wait complete. 2024-03-19T03:42:53.588 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T03:42:53.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.783+0000 7f54ed000640 1 -- 172.21.15.117:0/752271986 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54e8074220 msgr2=0x7f54e8074680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:53.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.783+0000 7f54ed000640 1 --2- 172.21.15.117:0/752271986 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54e8074220 0x7f54e8074680 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f54d0009900 tx=0x7f54d002bcc0 comp rx=0 tx=0).stop 2024-03-19T03:42:53.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.783+0000 7f54ed000640 1 -- 172.21.15.117:0/752271986 shutdown_connections 2024-03-19T03:42:53.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.783+0000 7f54ed000640 1 --2- 172.21.15.117:0/752271986 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54e81096c0 0x7f54e810bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:53.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.783+0000 7f54ed000640 1 --2- 172.21.15.117:0/752271986 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54e8074220 0x7f54e8074680 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:53.784 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.784+0000 7f54ed000640 1 --2- 172.21.15.117:0/752271986 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54e8075800 0x7f54e8073c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:53.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.784+0000 7f54ed000640 1 -- 172.21.15.117:0/752271986 >> 172.21.15.117:0/752271986 conn(0x7f54e80fcdc0 msgr2=0x7f54e80ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:53.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.784+0000 7f54ed000640 1 -- 172.21.15.117:0/752271986 shutdown_connections 2024-03-19T03:42:53.785 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.784+0000 7f54ed000640 1 -- 172.21.15.117:0/752271986 wait complete. 2024-03-19T03:42:53.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.784+0000 7f54ed000640 1 Processor -- start 2024-03-19T03:42:53.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.785+0000 7f54ed000640 1 -- start start 2024-03-19T03:42:53.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.785+0000 7f54ed000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54e8074220 0x7f54e8199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:53.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.785+0000 7f54ed000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54e8075800 0x7f54e819a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:53.789 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.785+0000 7f54ed000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54e81096c0 0x7f54e81a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:53.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.785+0000 7f54ed000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f54e810f6e0 con 0x7f54e8075800 2024-03-19T03:42:53.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.785+0000 7f54ed000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f54e810f560 con 0x7f54e81096c0 2024-03-19T03:42:53.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.785+0000 7f54ed000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f54e810f860 con 0x7f54e8074220 2024-03-19T03:42:53.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.785+0000 7f54e5e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54e8074220 0x7f54e8199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:53.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.785+0000 7f54e5e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54e8074220 0x7f54e8199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:43036/0 (socket says 172.21.15.117:43036) 2024-03-19T03:42:53.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.785+0000 7f54e6800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54e81096c0 0x7f54e81a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:53.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.785+0000 7f54e5e00640 1 -- 172.21.15.117:0/3969874535 learned_addr learned my addr 172.21.15.117:0/3969874535 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:42:53.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.785+0000 7f54dfe00640 1 --2- 172.21.15.117:0/3969874535 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54e8075800 0x7f54e819a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:53.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.786+0000 7f54e5e00640 1 -- 172.21.15.117:0/3969874535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54e81096c0 msgr2=0x7f54e81a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:53.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.786+0000 7f54e5e00640 1 --2- 172.21.15.117:0/3969874535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54e81096c0 0x7f54e81a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:53.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.786+0000 7f54e5e00640 1 -- 172.21.15.117:0/3969874535 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54e8075800 msgr2=0x7f54e819a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:53.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.786+0000 7f54e5e00640 1 --2- 172.21.15.117:0/3969874535 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54e8075800 0x7f54e819a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:53.790 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.786+0000 7f54e5e00640 1 -- 172.21.15.117:0/3969874535 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54e81a19c0 con 0x7f54e8074220 2024-03-19T03:42:53.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.786+0000 7f54e6800640 1 --2- 172.21.15.117:0/3969874535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54e81096c0 0x7f54e81a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:42:53.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.786+0000 7f54e5e00640 1 --2- 172.21.15.117:0/3969874535 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54e8074220 0x7f54e8199db0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f54d800d950 tx=0x7f54d800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:53.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.786+0000 7f54dea00640 1 -- 172.21.15.117:0/3969874535 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d8017030 con 0x7f54e8074220 2024-03-19T03:42:53.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.787+0000 7f54dea00640 1 -- 172.21.15.117:0/3969874535 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54d8004020 con 0x7f54e8074220 2024-03-19T03:42:53.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.787+0000 7f54ed000640 1 -- 172.21.15.117:0/3969874535 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54e81a1cb0 con 0x7f54e8074220 2024-03-19T03:42:53.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.787+0000 7f54dea00640 1 -- 172.21.15.117:0/3969874535 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d80047a0 con 0x7f54e8074220 2024-03-19T03:42:53.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.787+0000 7f54ed000640 1 -- 172.21.15.117:0/3969874535 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54e81a21a0 con 0x7f54e8074220 2024-03-19T03:42:53.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.788+0000 7f54ed000640 1 -- 172.21.15.117:0/3969874535 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54e8108480 con 0x7f54e8074220 2024-03-19T03:42:53.791 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.791+0000 7f54dea00640 1 -- 172.21.15.117:0/3969874535 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 20) ==== 137291+0+0 (secure 0 0 0) 0x7f54d80041d0 con 0x7f54e8074220 2024-03-19T03:42:53.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.793+0000 7f54dea00640 1 --2- 172.21.15.117:0/3969874535 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f54cc0a2880 0x7f54cc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:53.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.793+0000 7f54dea00640 1 -- 172.21.15.117:0/3969874535 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f54d80ce8b0 con 0x7f54e8074220 2024-03-19T03:42:53.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.793+0000 7f54dfe00640 1 --2- 172.21.15.117:0/3969874535 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f54cc0a2880 0x7f54cc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:53.793 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.794+0000 7f54dea00640 1 -- 172.21.15.117:0/3969874535 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f54d809b580 con 0x7f54e8074220 2024-03-19T03:42:53.794 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.794+0000 7f54dfe00640 1 --2- 172.21.15.117:0/3969874535 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f54cc0a2880 0x7f54cc0a4d40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f54d002c2d0 tx=0x7f54d002c910 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:53.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:53 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3205909288' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:42:53.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:53 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2021117909' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:42:53.993 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.992+0000 7f54ed000640 1 -- 172.21.15.117:0/3969874535 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f54e802d450 con 0x7f54e8074220 2024-03-19T03:42:53.994 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.993+0000 7f54dea00640 1 -- 172.21.15.117:0/3969874535 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f54d809ae80 con 0x7f54e8074220 2024-03-19T03:42:53.994 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 64 2024-03-19T03:42:53.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.995+0000 7f54ed000640 1 -- 172.21.15.117:0/3969874535 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f54cc0a2880 msgr2=0x7f54cc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:53.996 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.995+0000 7f54ed000640 1 --2- 172.21.15.117:0/3969874535 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f54cc0a2880 0x7f54cc0a4d40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f54d002c2d0 tx=0x7f54d002c910 comp rx=0 tx=0).stop 2024-03-19T03:42:53.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.995+0000 7f54ed000640 1 -- 172.21.15.117:0/3969874535 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54e8074220 msgr2=0x7f54e8199db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:53.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.995+0000 7f54ed000640 1 --2- 172.21.15.117:0/3969874535 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54e8074220 0x7f54e8199db0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f54d800d950 tx=0x7f54d800de10 comp rx=0 tx=0).stop 2024-03-19T03:42:53.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.995+0000 7f54ed000640 1 -- 172.21.15.117:0/3969874535 shutdown_connections 2024-03-19T03:42:53.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.995+0000 7f54ed000640 1 --2- 172.21.15.117:0/3969874535 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f54cc0a2880 0x7f54cc0a4d40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:53.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.995+0000 7f54ed000640 1 --2- 172.21.15.117:0/3969874535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54e81096c0 0x7f54e81a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:53.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.995+0000 7f54ed000640 1 --2- 172.21.15.117:0/3969874535 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54e8075800 0x7f54e819a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:53.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.995+0000 7f54ed000640 1 --2- 172.21.15.117:0/3969874535 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54e8074220 0x7f54e8199db0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:53.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.995+0000 7f54ed000640 1 -- 172.21.15.117:0/3969874535 >> 172.21.15.117:0/3969874535 conn(0x7f54e80fcdc0 msgr2=0x7f54e8105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:53.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.996+0000 7f54ed000640 1 -- 172.21.15.117:0/3969874535 shutdown_connections 2024-03-19T03:42:53.997 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:53.996+0000 7f54ed000640 1 -- 172.21.15.117:0/3969874535 wait complete. 2024-03-19T03:42:54.013 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:53 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3205909288' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:42:54.013 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:53 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2021117909' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:42:54.016 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-19T03:42:54.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:53 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3205909288' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T03:42:54.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:53 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2021117909' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T03:42:54.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.227+0000 7f8238e00640 1 -- 172.21.15.117:0/1780067359 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82341095b0 msgr2=0x7f823410ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:54.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.227+0000 7f8238e00640 1 --2- 172.21.15.117:0/1780067359 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82341095b0 0x7f823410ba50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f822c009ca0 tx=0x7f822c02c180 comp rx=0 tx=0).stop 2024-03-19T03:42:54.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.228+0000 7f8238e00640 1 -- 172.21.15.117:0/1780067359 shutdown_connections 2024-03-19T03:42:54.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.228+0000 7f8238e00640 1 --2- 172.21.15.117:0/1780067359 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82341095b0 0x7f823410ba50 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:54.229 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.228+0000 7f8238e00640 1 --2- 172.21.15.117:0/1780067359 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8234074220 0x7f8234074680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:54.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.228+0000 7f8238e00640 1 --2- 172.21.15.117:0/1780067359 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8234075800 0x7f8234073c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:54.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.228+0000 7f8238e00640 1 -- 172.21.15.117:0/1780067359 >> 172.21.15.117:0/1780067359 conn(0x7f82340fcc90 msgr2=0x7f82340ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:54.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.228+0000 7f8238e00640 1 -- 172.21.15.117:0/1780067359 shutdown_connections 2024-03-19T03:42:54.230 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.228+0000 7f8238e00640 1 -- 172.21.15.117:0/1780067359 wait complete. 2024-03-19T03:42:54.232 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.229+0000 7f8238e00640 1 Processor -- start 2024-03-19T03:42:54.232 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.229+0000 7f8238e00640 1 -- start start 2024-03-19T03:42:54.232 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.229+0000 7f8238e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8234074220 0x7f8234199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:54.232 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.229+0000 7f8238e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8234075800 0x7f823419a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:54.232 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.229+0000 7f8238e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82341095b0 0x7f82341a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:54.232 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.229+0000 7f8238e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f823410f580 con 0x7f82341095b0 2024-03-19T03:42:54.232 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.229+0000 7f8238e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f823410f400 con 0x7f8234075800 2024-03-19T03:42:54.232 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.229+0000 7f8238e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f823410f700 con 0x7f8234074220 2024-03-19T03:42:54.232 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.230+0000 7f822be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8234075800 0x7f823419a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:54.232 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.230+0000 7f822be00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8234075800 0x7f823419a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:50476/0 (socket says 172.21.15.117:50476) 2024-03-19T03:42:54.232 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.230+0000 7f822be00640 1 -- 172.21.15.117:0/1176886045 learned_addr learned my addr 172.21.15.117:0/1176886045 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:42:54.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.230+0000 7f8233400640 1 --2- 172.21.15.117:0/1176886045 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8234074220 0x7f8234199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:54.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.230+0000 7f8233e00640 1 --2- 172.21.15.117:0/1176886045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82341095b0 0x7f82341a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:54.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.230+0000 7f822be00640 1 -- 172.21.15.117:0/1176886045 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8234074220 msgr2=0x7f8234199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:54.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.230+0000 7f822be00640 1 --2- 172.21.15.117:0/1176886045 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8234074220 0x7f8234199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:54.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.230+0000 7f822be00640 1 -- 172.21.15.117:0/1176886045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82341095b0 msgr2=0x7f82341a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:54.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.230+0000 7f822be00640 1 --2- 172.21.15.117:0/1176886045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82341095b0 0x7f82341a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:54.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.230+0000 7f822be00640 1 -- 172.21.15.117:0/1176886045 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82341a17e0 con 0x7f8234075800 2024-03-19T03:42:54.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.230+0000 7f8233400640 1 --2- 172.21.15.117:0/1176886045 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8234074220 0x7f8234199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:42:54.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.231+0000 7f822be00640 1 --2- 172.21.15.117:0/1176886045 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8234075800 0x7f823419a100 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f821c00e9f0 tx=0x7f821c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:54.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.231+0000 7f8230c00640 1 -- 172.21.15.117:0/1176886045 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821c00cd80 con 0x7f8234075800 2024-03-19T03:42:54.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.231+0000 7f8230c00640 1 -- 172.21.15.117:0/1176886045 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f821c010910 con 0x7f8234075800 2024-03-19T03:42:54.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.231+0000 7f8238e00640 1 -- 172.21.15.117:0/1176886045 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82341a1ad0 con 0x7f8234075800 2024-03-19T03:42:54.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.231+0000 7f8230c00640 1 -- 172.21.15.117:0/1176886045 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821c018990 con 0x7f8234075800 2024-03-19T03:42:54.233 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.231+0000 7f8238e00640 1 -- 172.21.15.117:0/1176886045 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82341a2020 con 0x7f8234075800 2024-03-19T03:42:54.234 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.232+0000 7f8238e00640 1 -- 172.21.15.117:0/1176886045 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8234108370 con 0x7f8234075800 2024-03-19T03:42:54.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.235+0000 7f8230c00640 1 -- 172.21.15.117:0/1176886045 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 20) ==== 137291+0+0 (secure 0 0 0) 0x7f821c010ac0 con 0x7f8234075800 2024-03-19T03:42:54.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.237+0000 7f8230c00640 1 --2- 172.21.15.117:0/1176886045 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f82180a2800 0x7f82180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:54.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.237+0000 7f8230c00640 1 -- 172.21.15.117:0/1176886045 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f821c0cf340 con 0x7f8234075800 2024-03-19T03:42:54.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.237+0000 7f8233400640 1 --2- 172.21.15.117:0/1176886045 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f82180a2800 0x7f82180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:54.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.238+0000 7f8233400640 1 --2- 172.21.15.117:0/1176886045 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f82180a2800 0x7f82180a4cc0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8224009c20 tx=0x7f8224005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:54.241 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.241+0000 7f8230c00640 1 -- 172.21.15.117:0/1176886045 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f821c09bf90 con 0x7f8234075800 2024-03-19T03:42:54.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.439+0000 7f8238e00640 1 -- 172.21.15.117:0/1176886045 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8234062420 con 0x7f8234075800 2024-03-19T03:42:54.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.440+0000 7f8230c00640 1 -- 172.21.15.117:0/1176886045 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f821c084e20 con 0x7f8234075800 2024-03-19T03:42:54.441 INFO:teuthology.orchestra.run.smithi117.stdout:pg_num: 32 2024-03-19T03:42:54.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.442+0000 7f8238e00640 1 -- 172.21.15.117:0/1176886045 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f82180a2800 msgr2=0x7f82180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:54.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.442+0000 7f8238e00640 1 --2- 172.21.15.117:0/1176886045 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f82180a2800 0x7f82180a4cc0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8224009c20 tx=0x7f8224005f90 comp rx=0 tx=0).stop 2024-03-19T03:42:54.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.442+0000 7f8238e00640 1 -- 172.21.15.117:0/1176886045 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8234075800 msgr2=0x7f823419a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:54.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.442+0000 7f8238e00640 1 --2- 172.21.15.117:0/1176886045 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8234075800 0x7f823419a100 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f821c00e9f0 tx=0x7f821c00eeb0 comp rx=0 tx=0).stop 2024-03-19T03:42:54.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.443+0000 7f8238e00640 1 -- 172.21.15.117:0/1176886045 shutdown_connections 2024-03-19T03:42:54.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.443+0000 7f8238e00640 1 --2- 172.21.15.117:0/1176886045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82341095b0 0x7f82341a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:54.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.443+0000 7f8238e00640 1 --2- 172.21.15.117:0/1176886045 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f82180a2800 0x7f82180a4cc0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:54.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.443+0000 7f8238e00640 1 --2- 172.21.15.117:0/1176886045 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8234075800 0x7f823419a100 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:54.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.443+0000 7f8238e00640 1 --2- 172.21.15.117:0/1176886045 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8234074220 0x7f8234199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:54.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.443+0000 7f8238e00640 1 -- 172.21.15.117:0/1176886045 >> 172.21.15.117:0/1176886045 conn(0x7f82340fcc90 msgr2=0x7f8234105540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:54.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.443+0000 7f8238e00640 1 -- 172.21.15.117:0/1176886045 shutdown_connections 2024-03-19T03:42:54.445 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.443+0000 7f8238e00640 1 -- 172.21.15.117:0/1176886045 wait complete. 2024-03-19T03:42:54.470 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-19T03:42:54.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.682+0000 7fa7d5e00640 1 -- 172.21.15.117:0/3850960615 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7d0075860 msgr2=0x7fa7d0073cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:54.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.682+0000 7fa7d5e00640 1 --2- 172.21.15.117:0/3850960615 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7d0075860 0x7fa7d0073cb0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4009900 tx=0x7fa7c402bc30 comp rx=0 tx=0).stop 2024-03-19T03:42:54.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.682+0000 7fa7d5e00640 1 -- 172.21.15.117:0/3850960615 shutdown_connections 2024-03-19T03:42:54.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.682+0000 7fa7d5e00640 1 --2- 172.21.15.117:0/3850960615 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7d01094f0 0x7fa7d010b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:54.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.682+0000 7fa7d5e00640 1 --2- 172.21.15.117:0/3850960615 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7d0074280 0x7fa7d0074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:54.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.682+0000 7fa7d5e00640 1 --2- 172.21.15.117:0/3850960615 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7d0075860 0x7fa7d0073cb0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:54.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.682+0000 7fa7d5e00640 1 -- 172.21.15.117:0/3850960615 >> 172.21.15.117:0/3850960615 conn(0x7fa7d00fcb80 msgr2=0x7fa7d00fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:54.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.683+0000 7fa7d5e00640 1 -- 172.21.15.117:0/3850960615 shutdown_connections 2024-03-19T03:42:54.684 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.683+0000 7fa7d5e00640 1 -- 172.21.15.117:0/3850960615 wait complete. 2024-03-19T03:42:54.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.683+0000 7fa7d5e00640 1 Processor -- start 2024-03-19T03:42:54.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.684+0000 7fa7d5e00640 1 -- start start 2024-03-19T03:42:54.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.684+0000 7fa7d5e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7d0074280 0x7fa7d0199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:54.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.684+0000 7fa7d5e00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7d0075860 0x7fa7d019a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:54.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.684+0000 7fa7d5e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7d01094f0 0x7fa7d01a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:54.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.684+0000 7fa7d5e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d010f4e0 con 0x7fa7d0074280 2024-03-19T03:42:54.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.684+0000 7fa7d5e00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d010f360 con 0x7fa7d0075860 2024-03-19T03:42:54.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.684+0000 7fa7d5e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d010f660 con 0x7fa7d01094f0 2024-03-19T03:42:54.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.684+0000 7fa7d4a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7d0074280 0x7fa7d0199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:54.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.684+0000 7fa7cfe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7d0075860 0x7fa7d019a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:54.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.684+0000 7fa7cfe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7d0075860 0x7fa7d019a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:50490/0 (socket says 172.21.15.117:50490) 2024-03-19T03:42:54.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.684+0000 7fa7d4a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7d0074280 0x7fa7d0199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:49542/0 (socket says 172.21.15.117:49542) 2024-03-19T03:42:54.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.684+0000 7fa7cfe00640 1 -- 172.21.15.117:0/360290101 learned_addr learned my addr 172.21.15.117:0/360290101 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:42:54.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.684+0000 7fa7d5400640 1 --2- 172.21.15.117:0/360290101 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7d01094f0 0x7fa7d01a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:54.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.685+0000 7fa7cfe00640 1 -- 172.21.15.117:0/360290101 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7d01094f0 msgr2=0x7fa7d01a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:54.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.685+0000 7fa7cfe00640 1 --2- 172.21.15.117:0/360290101 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7d01094f0 0x7fa7d01a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:54.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.685+0000 7fa7cfe00640 1 -- 172.21.15.117:0/360290101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7d0074280 msgr2=0x7fa7d0199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:54.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.685+0000 7fa7cfe00640 1 --2- 172.21.15.117:0/360290101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7d0074280 0x7fa7d0199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:54.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.685+0000 7fa7cfe00640 1 -- 172.21.15.117:0/360290101 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d01a1710 con 0x7fa7d0075860 2024-03-19T03:42:54.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.685+0000 7fa7cfe00640 1 --2- 172.21.15.117:0/360290101 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7d0075860 0x7fa7d019a030 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc00d950 tx=0x7fa7bc00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:54.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.685+0000 7fa7cd600640 1 -- 172.21.15.117:0/360290101 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc017030 con 0x7fa7d0075860 2024-03-19T03:42:54.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.685+0000 7fa7cd600640 1 -- 172.21.15.117:0/360290101 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7bc004020 con 0x7fa7d0075860 2024-03-19T03:42:54.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.685+0000 7fa7cd600640 1 -- 172.21.15.117:0/360290101 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc0047a0 con 0x7fa7d0075860 2024-03-19T03:42:54.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.685+0000 7fa7d5e00640 1 -- 172.21.15.117:0/360290101 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7d01a1a00 con 0x7fa7d0075860 2024-03-19T03:42:54.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.685+0000 7fa7d5e00640 1 -- 172.21.15.117:0/360290101 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7d01a1f50 con 0x7fa7d0075860 2024-03-19T03:42:54.689 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.686+0000 7fa7d5e00640 1 -- 172.21.15.117:0/360290101 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7d0108220 con 0x7fa7d0075860 2024-03-19T03:42:54.689 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.689+0000 7fa7cd600640 1 -- 172.21.15.117:0/360290101 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 20) ==== 137291+0+0 (secure 0 0 0) 0x7fa7bc0041d0 con 0x7fa7d0075860 2024-03-19T03:42:54.691 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.690+0000 7fa7cd600640 1 --2- 172.21.15.117:0/360290101 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa7b80a2800 0x7fa7b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:54.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.691+0000 7fa7cd600640 1 -- 172.21.15.117:0/360290101 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7fa7bc0ce930 con 0x7fa7d0075860 2024-03-19T03:42:54.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.691+0000 7fa7d4a00640 1 --2- 172.21.15.117:0/360290101 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa7b80a2800 0x7fa7b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:54.692 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.692+0000 7fa7d4a00640 1 --2- 172.21.15.117:0/360290101 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa7b80a2800 0x7fa7b80a4cc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa7c402c240 tx=0x7fa7c402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:54.697 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.696+0000 7fa7cd600640 1 -- 172.21.15.117:0/360290101 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa7bc09b580 con 0x7fa7d0075860 2024-03-19T03:42:54.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:54 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3969874535' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:42:54.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:54 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1176886045' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:42:54.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:54 smithi117 ceph-mon[31838]: pgmap v691: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-19T03:42:54.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.932+0000 7fa7d5e00640 1 -- 172.21.15.117:0/360290101 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa7d002d1f0 con 0x7fa7d0075860 2024-03-19T03:42:54.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.933+0000 7fa7cd600640 1 -- 172.21.15.117:0/360290101 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v93) ==== 81+0+600 (secure 0 0 0) 0x7fa7bc09ae80 con 0x7fa7d0075860 2024-03-19T03:42:54.934 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:0/185684447 2024-03-20T03:22:38.134578+0000 2024-03-19T03:42:54.935 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:0/361367101 2024-03-20T03:22:38.134578+0000 2024-03-19T03:42:54.935 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:6801/2788067603 2024-03-20T03:22:38.134578+0000 2024-03-19T03:42:54.935 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:6801/1570899077 2024-03-20T03:22:03.905461+0000 2024-03-19T03:42:54.935 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:0/3506200675 2024-03-20T03:22:38.134578+0000 2024-03-19T03:42:54.935 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:6800/1570899077 2024-03-20T03:22:03.905461+0000 2024-03-19T03:42:54.935 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:0/3283471541 2024-03-20T03:22:03.905461+0000 2024-03-19T03:42:54.935 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:0/3750920169 2024-03-20T03:22:03.905461+0000 2024-03-19T03:42:54.935 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:0/2780813298 2024-03-20T03:22:03.905461+0000 2024-03-19T03:42:54.935 INFO:teuthology.orchestra.run.smithi117.stdout:172.21.15.117:6800/2788067603 2024-03-20T03:22:38.134578+0000 2024-03-19T03:42:54.935 INFO:teuthology.orchestra.run.smithi117.stderr:listed 10 entries 2024-03-19T03:42:54.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.936+0000 7fa7d5e00640 1 -- 172.21.15.117:0/360290101 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa7b80a2800 msgr2=0x7fa7b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:54.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.936+0000 7fa7d5e00640 1 --2- 172.21.15.117:0/360290101 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa7b80a2800 0x7fa7b80a4cc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa7c402c240 tx=0x7fa7c402df90 comp rx=0 tx=0).stop 2024-03-19T03:42:54.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.936+0000 7fa7d5e00640 1 -- 172.21.15.117:0/360290101 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7d0075860 msgr2=0x7fa7d019a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:54.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.936+0000 7fa7d5e00640 1 --2- 172.21.15.117:0/360290101 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7d0075860 0x7fa7d019a030 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc00d950 tx=0x7fa7bc00de10 comp rx=0 tx=0).stop 2024-03-19T03:42:54.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.936+0000 7fa7d5e00640 1 -- 172.21.15.117:0/360290101 shutdown_connections 2024-03-19T03:42:54.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.936+0000 7fa7d5e00640 1 --2- 172.21.15.117:0/360290101 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7d01094f0 0x7fa7d01a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:54.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.936+0000 7fa7d5e00640 1 --2- 172.21.15.117:0/360290101 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa7b80a2800 0x7fa7b80a4cc0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:54.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.936+0000 7fa7d5e00640 1 --2- 172.21.15.117:0/360290101 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7d0075860 0x7fa7d019a030 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:54.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.936+0000 7fa7d5e00640 1 --2- 172.21.15.117:0/360290101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7d0074280 0x7fa7d0199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:54.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.936+0000 7fa7d5e00640 1 -- 172.21.15.117:0/360290101 >> 172.21.15.117:0/360290101 conn(0x7fa7d00fcb80 msgr2=0x7fa7d01053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:54.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.936+0000 7fa7d5e00640 1 -- 172.21.15.117:0/360290101 shutdown_connections 2024-03-19T03:42:54.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:54.936+0000 7fa7d5e00640 1 -- 172.21.15.117:0/360290101 wait complete. 2024-03-19T03:42:54.955 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-19T03:42:54.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:42:54.955 DEBUG:teuthology.orchestra.run.smithi117:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-19T03:42:54.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:54 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3969874535' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:42:54.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:54 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1176886045' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:42:54.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:54 smithi119 ceph-mon[66861]: pgmap v691: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-19T03:42:55.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:54 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3969874535' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T03:42:55.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:54 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1176886045' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-19T03:42:55.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:54 smithi178 ceph-mon[67578]: pgmap v691: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-19T03:42:55.384 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi117.front.sepia.ceph.com 2024-03-19T03:42:55.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:42:55.385 DEBUG:teuthology.orchestra.run.smithi117:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-19T03:42:55.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:42:55.465 DEBUG:teuthology.orchestra.run.smithi117:> ip netns list 2024-03-19T03:42:55.527 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-19T03:42:55.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:42:55.528 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-19T03:42:55.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T03:42:55.600 DEBUG:teuthology.orchestra.run.smithi117:> sudo ip link delete ceph-brx 2024-03-19T03:42:55.718 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-19T03:42:55.728 INFO:tasks.cephadm:Teardown begin 2024-03-19T03:42:55.728 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-19T03:42:55.728 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:42:55.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.921+0000 7f9ff7200640 1 -- 172.21.15.117:0/3205741291 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ff8103340 msgr2=0x7f9ff8109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:55.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.921+0000 7f9ff7200640 1 --2- 172.21.15.117:0/3205741291 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ff8103340 0x7f9ff8109bd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9ff0009cd0 tx=0x7f9ff002c0f0 comp rx=0 tx=0).stop 2024-03-19T03:42:55.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.921+0000 7f9ff7200640 1 -- 172.21.15.117:0/3205741291 shutdown_connections 2024-03-19T03:42:55.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.921+0000 7f9ff7200640 1 --2- 172.21.15.117:0/3205741291 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ff8103340 0x7f9ff8109bd0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:55.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.921+0000 7f9ff7200640 1 --2- 172.21.15.117:0/3205741291 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ff8102980 0x7f9ff8102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:55.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.921+0000 7f9ff7200640 1 --2- 172.21.15.117:0/3205741291 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ff80ff9b0 0x7f9ff80ffdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:55.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.921+0000 7f9ff7200640 1 -- 172.21.15.117:0/3205741291 >> 172.21.15.117:0/3205741291 conn(0x7f9ff80fb2e0 msgr2=0x7f9ff80fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:55.922 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.921+0000 7f9ff7200640 1 -- 172.21.15.117:0/3205741291 shutdown_connections 2024-03-19T03:42:55.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.921+0000 7f9ff7200640 1 -- 172.21.15.117:0/3205741291 wait complete. 2024-03-19T03:42:55.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.922+0000 7f9ff7200640 1 Processor -- start 2024-03-19T03:42:55.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.922+0000 7f9ff7200640 1 -- start start 2024-03-19T03:42:55.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.923+0000 7f9ff7200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ff80ff9b0 0x7f9ff81979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:55.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.923+0000 7f9ff7200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ff8102980 0x7f9ff8197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:55.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.923+0000 7f9ff7200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ff8103340 0x7f9ff819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:55.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.923+0000 7f9ff7200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff810d300 con 0x7f9ff8103340 2024-03-19T03:42:55.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.923+0000 7f9ff7200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff810d180 con 0x7f9ff8102980 2024-03-19T03:42:55.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.923+0000 7f9ff7200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff810d480 con 0x7f9ff80ff9b0 2024-03-19T03:42:55.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.923+0000 7f9ff5e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ff80ff9b0 0x7f9ff81979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:55.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.923+0000 7f9ff5400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ff8102980 0x7f9ff8197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:55.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.923+0000 7f9ff5e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ff80ff9b0 0x7f9ff81979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:45422/0 (socket says 172.21.15.117:45422) 2024-03-19T03:42:55.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.923+0000 7f9ff5400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ff8102980 0x7f9ff8197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:44116/0 (socket says 172.21.15.117:44116) 2024-03-19T03:42:55.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.923+0000 7f9ff5e00640 1 -- 172.21.15.117:0/2679157926 learned_addr learned my addr 172.21.15.117:0/2679157926 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:42:55.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.923+0000 7f9ff6800640 1 --2- 172.21.15.117:0/2679157926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ff8103340 0x7f9ff819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:55.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.923+0000 7f9ff5e00640 1 -- 172.21.15.117:0/2679157926 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ff8102980 msgr2=0x7f9ff8197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:55.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.923+0000 7f9ff5e00640 1 --2- 172.21.15.117:0/2679157926 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ff8102980 0x7f9ff8197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:55.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.923+0000 7f9ff5e00640 1 -- 172.21.15.117:0/2679157926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ff8103340 msgr2=0x7f9ff819ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:55.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.923+0000 7f9ff5e00640 1 --2- 172.21.15.117:0/2679157926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ff8103340 0x7f9ff819ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:55.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.923+0000 7f9ff5e00640 1 -- 172.21.15.117:0/2679157926 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ff819f5d0 con 0x7f9ff80ff9b0 2024-03-19T03:42:55.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.923+0000 7f9ff6800640 1 --2- 172.21.15.117:0/2679157926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ff8103340 0x7f9ff819ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T03:42:55.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.923+0000 7f9ff5400640 1 --2- 172.21.15.117:0/2679157926 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ff8102980 0x7f9ff8197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:42:55.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.924+0000 7f9ff5e00640 1 --2- 172.21.15.117:0/2679157926 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ff80ff9b0 0x7f9ff81979b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9fe800e9c0 tx=0x7f9fe800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:55.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.924+0000 7f9feea00640 1 -- 172.21.15.117:0/2679157926 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe800cde0 con 0x7f9ff80ff9b0 2024-03-19T03:42:55.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.924+0000 7f9feea00640 1 -- 172.21.15.117:0/2679157926 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fe8010910 con 0x7f9ff80ff9b0 2024-03-19T03:42:55.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.924+0000 7f9feea00640 1 -- 172.21.15.117:0/2679157926 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe8018a00 con 0x7f9ff80ff9b0 2024-03-19T03:42:55.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.924+0000 7f9ff7200640 1 -- 172.21.15.117:0/2679157926 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ff81007d0 con 0x7f9ff80ff9b0 2024-03-19T03:42:55.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.924+0000 7f9ff7200640 1 -- 172.21.15.117:0/2679157926 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ff8100d70 con 0x7f9ff80ff9b0 2024-03-19T03:42:55.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.925+0000 7f9ff7200640 1 -- 172.21.15.117:0/2679157926 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ff8108a50 con 0x7f9ff80ff9b0 2024-03-19T03:42:55.928 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.928+0000 7f9feea00640 1 -- 172.21.15.117:0/2679157926 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 20) ==== 137291+0+0 (secure 0 0 0) 0x7f9fe801f020 con 0x7f9ff80ff9b0 2024-03-19T03:42:55.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.929+0000 7f9feea00640 1 --2- 172.21.15.117:0/2679157926 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f9fdc0a2880 0x7f9fdc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:55.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.929+0000 7f9feea00640 1 -- 172.21.15.117:0/2679157926 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f9fe80cf0c0 con 0x7f9ff80ff9b0 2024-03-19T03:42:55.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.929+0000 7f9ff5400640 1 --2- 172.21.15.117:0/2679157926 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f9fdc0a2880 0x7f9fdc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:55.931 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.930+0000 7f9ff5400640 1 --2- 172.21.15.117:0/2679157926 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f9fdc0a2880 0x7f9fdc0a4d40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8198ec0 tx=0x7f9fe4012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:55.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:55.933+0000 7f9feea00640 1 -- 172.21.15.117:0/2679157926 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9fe809bd10 con 0x7f9ff80ff9b0 2024-03-19T03:42:55.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:55 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/360290101' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T03:42:56.041 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:55 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/360290101' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T03:42:56.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:55 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/360290101' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T03:42:56.234 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.233+0000 7f9ff7200640 1 -- 172.21.15.117:0/2679157926 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ff802c070 con 0x7f9ff80ff9b0 2024-03-19T03:42:56.235 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.234+0000 7f9feea00640 1 -- 172.21.15.117:0/2679157926 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+11961 (secure 0 0 0) 0x7f9fe809b610 con 0x7f9ff80ff9b0 2024-03-19T03:42:56.235 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:42:56.235 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/54638259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":54638259},{"type":"v1","addr":"172.21.15.117:6835","nonce":54638259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3608185039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3608185039},{"type":"v1","addr":"172.21.15.117:6837","nonce":3608185039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":45,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T03:30:35.081740+0000","modified":"2024-03-19T03:41:58.439645+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24599,"mds_1":24581,"mds_2":24569,"mds_3":24545,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":20,"addr":"172.21.15.119:6833/1291833214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1291833214},{"type":"v1","addr":"172.21.15.119:6833","nonce":1291833214}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6835/1579860804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1579860804},{"type":"v1","addr":"172.21.15.178:6835","nonce":1579860804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6835/1686938941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1686938941},{"type":"v1","addr":"172.21.15.119:6835","nonce":1686938941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6837/1770913538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":1770913538},{"type":"v1","addr":"172.21.15.178:6837","nonce":1770913538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6839/2655593383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2655593383},{"type":"v1","addr":"172.21.15.178:6839","nonce":2655593383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.119:6837/2054992738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2054992738},{"type":"v1","addr":"172.21.15.119:6837","nonce":2054992738}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"g","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.117:6839/684335078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":684335078},{"type":"v1","addr":"172.21.15.117:6839","nonce":684335078}]},"join_fscid":-1,"export_targets":[0,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.117:6841/1329933892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6840","nonce":1329933892},{"type":"v1","addr":"172.21.15.117:6841","nonce":1329933892}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/2383354833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2383354833},{"type":"v1","addr":"172.21.15.119:6839","nonce":2383354833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24467,24545,24569,24581,24599]},"id":1}]} 2024-03-19T03:42:56.235 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 45 2024-03-19T03:42:56.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.236+0000 7f9ff7200640 1 -- 172.21.15.117:0/2679157926 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f9fdc0a2880 msgr2=0x7f9fdc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:56.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.236+0000 7f9ff7200640 1 --2- 172.21.15.117:0/2679157926 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f9fdc0a2880 0x7f9fdc0a4d40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9ff8198ec0 tx=0x7f9fe4012040 comp rx=0 tx=0).stop 2024-03-19T03:42:56.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.236+0000 7f9ff7200640 1 -- 172.21.15.117:0/2679157926 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ff80ff9b0 msgr2=0x7f9ff81979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:56.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.236+0000 7f9ff7200640 1 --2- 172.21.15.117:0/2679157926 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ff80ff9b0 0x7f9ff81979b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9fe800e9c0 tx=0x7f9fe800ee80 comp rx=0 tx=0).stop 2024-03-19T03:42:56.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.237+0000 7f9ff7200640 1 -- 172.21.15.117:0/2679157926 shutdown_connections 2024-03-19T03:42:56.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.237+0000 7f9ff7200640 1 --2- 172.21.15.117:0/2679157926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ff8103340 0x7f9ff819ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:56.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.237+0000 7f9ff7200640 1 --2- 172.21.15.117:0/2679157926 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f9fdc0a2880 0x7f9fdc0a4d40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:56.238 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.237+0000 7f9ff7200640 1 --2- 172.21.15.117:0/2679157926 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ff8102980 0x7f9ff8197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:56.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.237+0000 7f9ff7200640 1 --2- 172.21.15.117:0/2679157926 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ff80ff9b0 0x7f9ff81979b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:56.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.237+0000 7f9ff7200640 1 -- 172.21.15.117:0/2679157926 >> 172.21.15.117:0/2679157926 conn(0x7f9ff80fb2e0 msgr2=0x7f9ff8105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:56.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.237+0000 7f9ff7200640 1 -- 172.21.15.117:0/2679157926 shutdown_connections 2024-03-19T03:42:56.239 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.237+0000 7f9ff7200640 1 -- 172.21.15.117:0/2679157926 wait complete. 2024-03-19T03:42:56.254 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T03:42:56.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.462+0000 7ffb6ac00640 1 -- 172.21.15.117:0/487591455 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb64100570 msgr2=0x7ffb64100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:56.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.462+0000 7ffb6ac00640 1 --2- 172.21.15.117:0/487591455 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb64100570 0x7ffb64100990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ffb58009900 tx=0x7ffb5802bc40 comp rx=0 tx=0).stop 2024-03-19T03:42:56.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.462+0000 7ffb6ac00640 1 -- 172.21.15.117:0/487591455 shutdown_connections 2024-03-19T03:42:56.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.462+0000 7ffb6ac00640 1 --2- 172.21.15.117:0/487591455 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffb64102120 0x7ffb641089b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:56.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.462+0000 7ffb6ac00640 1 --2- 172.21.15.117:0/487591455 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb64101760 0x7ffb64101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:56.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.462+0000 7ffb6ac00640 1 --2- 172.21.15.117:0/487591455 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb64100570 0x7ffb64100990 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:56.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.462+0000 7ffb6ac00640 1 -- 172.21.15.117:0/487591455 >> 172.21.15.117:0/487591455 conn(0x7ffb640fbb00 msgr2=0x7ffb640fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:56.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.463+0000 7ffb6ac00640 1 -- 172.21.15.117:0/487591455 shutdown_connections 2024-03-19T03:42:56.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.463+0000 7ffb6ac00640 1 -- 172.21.15.117:0/487591455 wait complete. 2024-03-19T03:42:56.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.464+0000 7ffb6ac00640 1 Processor -- start 2024-03-19T03:42:56.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.464+0000 7ffb6ac00640 1 -- start start 2024-03-19T03:42:56.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.464+0000 7ffb6ac00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb64100570 0x7ffb641044b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:56.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.464+0000 7ffb6ac00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb64101760 0x7ffb64102b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:56.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.464+0000 7ffb6ac00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffb64102120 0x7ffb64103080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:56.467 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.464+0000 7ffb6ac00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6410c200 con 0x7ffb64101760 2024-03-19T03:42:56.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.464+0000 7ffb6ac00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6410c080 con 0x7ffb64100570 2024-03-19T03:42:56.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.464+0000 7ffb6ac00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6410c380 con 0x7ffb64102120 2024-03-19T03:42:56.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.464+0000 7ffb69800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb64100570 0x7ffb641044b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:56.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.464+0000 7ffb68e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb64101760 0x7ffb64102b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:56.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.465+0000 7ffb69800640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb64100570 0x7ffb641044b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:44130/0 (socket says 172.21.15.117:44130) 2024-03-19T03:42:56.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.465+0000 7ffb69800640 1 -- 172.21.15.117:0/1159976059 learned_addr learned my addr 172.21.15.117:0/1159976059 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:42:56.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.465+0000 7ffb6a200640 1 --2- 172.21.15.117:0/1159976059 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffb64102120 0x7ffb64103080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:56.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.465+0000 7ffb69800640 1 -- 172.21.15.117:0/1159976059 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffb64102120 msgr2=0x7ffb64103080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:56.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.465+0000 7ffb69800640 1 --2- 172.21.15.117:0/1159976059 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffb64102120 0x7ffb64103080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:56.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.465+0000 7ffb69800640 1 -- 172.21.15.117:0/1159976059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb64101760 msgr2=0x7ffb64102b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:56.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.465+0000 7ffb69800640 1 --2- 172.21.15.117:0/1159976059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb64101760 0x7ffb64102b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:56.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.465+0000 7ffb69800640 1 -- 172.21.15.117:0/1159976059 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb64103940 con 0x7ffb64100570 2024-03-19T03:42:56.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.465+0000 7ffb69800640 1 --2- 172.21.15.117:0/1159976059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb64100570 0x7ffb641044b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ffb58000c00 tx=0x7ffb58005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:56.468 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.465+0000 7ffb62000640 1 -- 172.21.15.117:0/1159976059 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb58041070 con 0x7ffb64100570 2024-03-19T03:42:56.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.466+0000 7ffb6ac00640 1 -- 172.21.15.117:0/1159976059 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb641a1aa0 con 0x7ffb64100570 2024-03-19T03:42:56.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.466+0000 7ffb62000640 1 -- 172.21.15.117:0/1159976059 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb580044d0 con 0x7ffb64100570 2024-03-19T03:42:56.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.466+0000 7ffb62000640 1 -- 172.21.15.117:0/1159976059 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb58034440 con 0x7ffb64100570 2024-03-19T03:42:56.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.466+0000 7ffb6ac00640 1 -- 172.21.15.117:0/1159976059 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb641a1fb0 con 0x7ffb64100570 2024-03-19T03:42:56.469 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.467+0000 7ffb6ac00640 1 -- 172.21.15.117:0/1159976059 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb641a2180 con 0x7ffb64100570 2024-03-19T03:42:56.471 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.470+0000 7ffb62000640 1 -- 172.21.15.117:0/1159976059 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 20) ==== 137291+0+0 (secure 0 0 0) 0x7ffb58044070 con 0x7ffb64100570 2024-03-19T03:42:56.473 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.472+0000 7ffb62000640 1 --2- 172.21.15.117:0/1159976059 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7ffb4c0a2880 0x7ffb4c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:56.473 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.472+0000 7ffb62000640 1 -- 172.21.15.117:0/1159976059 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7ffb580f0b00 con 0x7ffb64100570 2024-03-19T03:42:56.473 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.472+0000 7ffb68e00640 1 --2- 172.21.15.117:0/1159976059 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7ffb4c0a2880 0x7ffb4c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:56.473 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.473+0000 7ffb68e00640 1 --2- 172.21.15.117:0/1159976059 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7ffb4c0a2880 0x7ffb4c0a4d40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ffb64104220 tx=0x7ffb5000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:56.478 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.477+0000 7ffb62000640 1 -- 172.21.15.117:0/1159976059 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ffb580bd750 con 0x7ffb64100570 2024-03-19T03:42:56.711 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.711+0000 7ffb6ac00640 1 -- 172.21.15.117:0/1159976059 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb641a2470 con 0x7ffb64100570 2024-03-19T03:42:56.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.713+0000 7ffb62000640 1 -- 172.21.15.117:0/1159976059 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+11961 (secure 0 0 0) 0x7ffb580bd050 con 0x7ffb64100570 2024-03-19T03:42:56.715 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:42:56.715 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/54638259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":54638259},{"type":"v1","addr":"172.21.15.117:6835","nonce":54638259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14409,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/3608185039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":3608185039},{"type":"v1","addr":"172.21.15.117:6837","nonce":3608185039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":45,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T03:30:35.081740+0000","modified":"2024-03-19T03:41:58.439645+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24599,"mds_1":24581,"mds_2":24569,"mds_3":24545,"mds_4":24467},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6833/3751720259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3751720259},{"type":"v1","addr":"172.21.15.178:6833","nonce":3751720259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":20,"addr":"172.21.15.119:6833/1291833214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1291833214},{"type":"v1","addr":"172.21.15.119:6833","nonce":1291833214}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6835/1579860804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1579860804},{"type":"v1","addr":"172.21.15.178:6835","nonce":1579860804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.119:6835/1686938941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1686938941},{"type":"v1","addr":"172.21.15.119:6835","nonce":1686938941}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6837/1770913538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":1770913538},{"type":"v1","addr":"172.21.15.178:6837","nonce":1770913538}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6839/2655593383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2655593383},{"type":"v1","addr":"172.21.15.178:6839","nonce":2655593383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24545":{"gid":24545,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.119:6837/2054992738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6836","nonce":2054992738},{"type":"v1","addr":"172.21.15.119:6837","nonce":2054992738}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24569":{"gid":24569,"name":"g","rank":2,"incarnation":22,"state":"up:active","state_seq":13,"addr":"172.21.15.117:6839/684335078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":684335078},{"type":"v1","addr":"172.21.15.117:6839","nonce":684335078}]},"join_fscid":-1,"export_targets":[0,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24581":{"gid":24581,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.117:6841/1329933892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6840","nonce":1329933892},{"type":"v1","addr":"172.21.15.117:6841","nonce":1329933892}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24599":{"gid":24599,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6839/2383354833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6838","nonce":2383354833},{"type":"v1","addr":"172.21.15.119:6839","nonce":2383354833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24599,"qdb_cluster":[24467,24545,24569,24581,24599]},"id":1}]} 2024-03-19T03:42:56.715 INFO:teuthology.orchestra.run.smithi117.stderr:dumped fsmap epoch 45 2024-03-19T03:42:56.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.717+0000 7ffb6ac00640 1 -- 172.21.15.117:0/1159976059 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7ffb4c0a2880 msgr2=0x7ffb4c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:56.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.717+0000 7ffb6ac00640 1 --2- 172.21.15.117:0/1159976059 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7ffb4c0a2880 0x7ffb4c0a4d40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ffb64104220 tx=0x7ffb5000b390 comp rx=0 tx=0).stop 2024-03-19T03:42:56.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.717+0000 7ffb6ac00640 1 -- 172.21.15.117:0/1159976059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb64100570 msgr2=0x7ffb641044b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:56.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.717+0000 7ffb6ac00640 1 --2- 172.21.15.117:0/1159976059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb64100570 0x7ffb641044b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ffb58000c00 tx=0x7ffb58005d50 comp rx=0 tx=0).stop 2024-03-19T03:42:56.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.717+0000 7ffb6ac00640 1 -- 172.21.15.117:0/1159976059 shutdown_connections 2024-03-19T03:42:56.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.717+0000 7ffb6ac00640 1 --2- 172.21.15.117:0/1159976059 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffb64102120 0x7ffb64103080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:56.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.717+0000 7ffb6ac00640 1 --2- 172.21.15.117:0/1159976059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb64101760 0x7ffb64102b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:56.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.717+0000 7ffb6ac00640 1 --2- 172.21.15.117:0/1159976059 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7ffb4c0a2880 0x7ffb4c0a4d40 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:56.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.717+0000 7ffb6ac00640 1 --2- 172.21.15.117:0/1159976059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb64100570 0x7ffb641044b0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:56.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.717+0000 7ffb6ac00640 1 -- 172.21.15.117:0/1159976059 >> 172.21.15.117:0/1159976059 conn(0x7ffb640fbb00 msgr2=0x7ffb640fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:56.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.717+0000 7ffb6ac00640 1 -- 172.21.15.117:0/1159976059 shutdown_connections 2024-03-19T03:42:56.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.717+0000 7ffb6ac00640 1 -- 172.21.15.117:0/1159976059 wait complete. 2024-03-19T03:42:56.738 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T03:42:56.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.940+0000 7f4bbb200640 1 -- 172.21.15.117:0/2655596596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4bb41022b0 msgr2=0x7f4bb41046a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:56.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.940+0000 7f4bbb200640 1 --2- 172.21.15.117:0/2655596596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4bb41022b0 0x7f4bb41046a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4009900 tx=0x7f4ba402bc30 comp rx=0 tx=0).stop 2024-03-19T03:42:56.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.941+0000 7f4bbb200640 1 -- 172.21.15.117:0/2655596596 shutdown_connections 2024-03-19T03:42:56.942 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.941+0000 7f4bbb200640 1 --2- 172.21.15.117:0/2655596596 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4bb4107510 0x7f4bb4109960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:56.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.941+0000 7f4bbb200640 1 --2- 172.21.15.117:0/2655596596 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4bb4104be0 0x7f4bb4106fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:56.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.941+0000 7f4bbb200640 1 --2- 172.21.15.117:0/2655596596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4bb41022b0 0x7f4bb41046a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:56.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.941+0000 7f4bbb200640 1 -- 172.21.15.117:0/2655596596 >> 172.21.15.117:0/2655596596 conn(0x7f4bb40fbca0 msgr2=0x7f4bb40fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:56.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.941+0000 7f4bbb200640 1 -- 172.21.15.117:0/2655596596 shutdown_connections 2024-03-19T03:42:56.943 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.941+0000 7f4bbb200640 1 -- 172.21.15.117:0/2655596596 wait complete. 2024-03-19T03:42:56.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.942+0000 7f4bbb200640 1 Processor -- start 2024-03-19T03:42:56.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.942+0000 7f4bbb200640 1 -- start start 2024-03-19T03:42:56.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.942+0000 7f4bbb200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4bb41022b0 0x7f4bb4101c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:56.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.942+0000 7f4bbb200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4bb4104be0 0x7f4bb4100260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:56.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.942+0000 7f4bbb200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4bb4107510 0x7f4bb41007a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:56.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.942+0000 7f4bbb200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb410d530 con 0x7f4bb4107510 2024-03-19T03:42:56.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.942+0000 7f4bbb200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb410d3b0 con 0x7f4bb4104be0 2024-03-19T03:42:56.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.942+0000 7f4bbb200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb410d6b0 con 0x7f4bb41022b0 2024-03-19T03:42:56.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.942+0000 7f4bb3400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4bb4104be0 0x7f4bb4100260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:56.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.943+0000 7f4bb3400640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4bb4104be0 0x7f4bb4100260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:44136/0 (socket says 172.21.15.117:44136) 2024-03-19T03:42:56.945 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.943+0000 7f4bb3400640 1 -- 172.21.15.117:0/539826350 learned_addr learned my addr 172.21.15.117:0/539826350 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:42:56.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.943+0000 7f4bb9200640 1 --2- 172.21.15.117:0/539826350 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4bb4107510 0x7f4bb41007a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:56.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.943+0000 7f4bb3e00640 1 --2- 172.21.15.117:0/539826350 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4bb41022b0 0x7f4bb4101c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:56.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.943+0000 7f4bb3400640 1 -- 172.21.15.117:0/539826350 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4bb41022b0 msgr2=0x7f4bb4101c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:56.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.943+0000 7f4bb3400640 1 --2- 172.21.15.117:0/539826350 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4bb41022b0 0x7f4bb4101c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:56.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.943+0000 7f4bb3400640 1 -- 172.21.15.117:0/539826350 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4bb4107510 msgr2=0x7f4bb41007a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:56.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.943+0000 7f4bb3400640 1 --2- 172.21.15.117:0/539826350 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4bb4107510 0x7f4bb41007a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:56.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.943+0000 7f4bb3400640 1 -- 172.21.15.117:0/539826350 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb4101030 con 0x7f4bb4104be0 2024-03-19T03:42:56.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.943+0000 7f4bb3400640 1 --2- 172.21.15.117:0/539826350 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4bb4104be0 0x7f4bb4100260 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000e9f0 tx=0x7f4ba000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:56.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.944+0000 7f4bb0c00640 1 -- 172.21.15.117:0/539826350 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba000ccc0 con 0x7f4bb4104be0 2024-03-19T03:42:56.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.944+0000 7f4bb0c00640 1 -- 172.21.15.117:0/539826350 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ba0005500 con 0x7f4bb4104be0 2024-03-19T03:42:56.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.944+0000 7f4bbb200640 1 -- 172.21.15.117:0/539826350 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bb41a1cf0 con 0x7f4bb4104be0 2024-03-19T03:42:56.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.944+0000 7f4bb0c00640 1 -- 172.21.15.117:0/539826350 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0010690 con 0x7f4bb4104be0 2024-03-19T03:42:56.946 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.944+0000 7f4bbb200640 1 -- 172.21.15.117:0/539826350 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bb41a2190 con 0x7f4bb4104be0 2024-03-19T03:42:56.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.945+0000 7f4bbb200640 1 -- 172.21.15.117:0/539826350 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bb40fd790 con 0x7f4bb4104be0 2024-03-19T03:42:56.948 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.947+0000 7f4bb0c00640 1 -- 172.21.15.117:0/539826350 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 20) ==== 137291+0+0 (secure 0 0 0) 0x7f4ba00046c0 con 0x7f4bb4104be0 2024-03-19T03:42:56.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.949+0000 7f4bb0c00640 1 --2- 172.21.15.117:0/539826350 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4b980a2800 0x7f4b980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:56.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.949+0000 7f4bb0c00640 1 -- 172.21.15.117:0/539826350 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f4ba00cf200 con 0x7f4bb4104be0 2024-03-19T03:42:56.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.949+0000 7f4bb3e00640 1 --2- 172.21.15.117:0/539826350 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4b980a2800 0x7f4b980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:56.951 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.950+0000 7f4bb3e00640 1 --2- 172.21.15.117:0/539826350 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4b980a2800 0x7f4b980a4cc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4ba402c240 tx=0x7f4ba402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:56.958 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:56.957+0000 7f4bb0c00640 1 -- 172.21.15.117:0/539826350 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4ba009beb0 con 0x7f4bb4104be0 2024-03-19T03:42:56.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:56 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2679157926' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:42:56.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:56 smithi119 ceph-mon[66861]: pgmap v692: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-19T03:42:57.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:56 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2679157926' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:42:57.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:56 smithi117 ceph-mon[31838]: pgmap v692: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-19T03:42:57.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:56 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2679157926' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:42:57.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:56 smithi178 ceph-mon[67578]: pgmap v692: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-19T03:42:57.247 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.246+0000 7f4bbb200640 1 -- 172.21.15.117:0/539826350 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4bb4062420 con 0x7f4bb4104be0 2024-03-19T03:42:57.249 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.248+0000 7f4bb0c00640 1 -- 172.21.15.117:0/539826350 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24989 (secure 0 0 0) 0x7f4ba009b7b0 con 0x7f4bb4104be0 2024-03-19T03:42:57.250 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-19T03:42:57.250 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":93,"fsid":"a3a82456-e59f-11ee-95c9-87774f69a715","created":"2024-03-19T03:21:30.427136+0000","modified":"2024-03-19T03:32:40.453890+0000","last_up_change":"2024-03-19T03:29:32.967505+0000","last_in_change":"2024-03-19T03:29:12.848740+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T03:25:18.824270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T03:30:33.137170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T03:30:34.598838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-19T03:30:38.230023+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b32eb0ee-9d4b-4748-87ec-eda02c907415","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6803","nonce":3014934900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6805","nonce":3014934900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6809","nonce":3014934900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3014934900},{"type":"v1","addr":"172.21.15.117:6807","nonce":3014934900}]},"public_addr":"172.21.15.117:6803/3014934900","cluster_addr":"172.21.15.117:6805/3014934900","heartbeat_back_addr":"172.21.15.117:6809/3014934900","heartbeat_front_addr":"172.21.15.117:6807/3014934900","state":["exists","up"]},{"osd":1,"uuid":"e07585be-c016-4466-b07c-abbc22a58de4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6801","nonce":2425927341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6803","nonce":2425927341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6807","nonce":2425927341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2425927341},{"type":"v1","addr":"172.21.15.119:6805","nonce":2425927341}]},"public_addr":"172.21.15.119:6801/2425927341","cluster_addr":"172.21.15.119:6803/2425927341","heartbeat_back_addr":"172.21.15.119:6807/2425927341","heartbeat_front_addr":"172.21.15.119:6805/2425927341","state":["exists","up"]},{"osd":2,"uuid":"ede9edda-3b9f-41ca-ad04-5dcb738ab757","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6801","nonce":3733812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6803","nonce":3733812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6807","nonce":3733812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":3733812989},{"type":"v1","addr":"172.21.15.178:6805","nonce":3733812989}]},"public_addr":"172.21.15.178:6801/3733812989","cluster_addr":"172.21.15.178:6803/3733812989","heartbeat_back_addr":"172.21.15.178:6807/3733812989","heartbeat_front_addr":"172.21.15.178:6805/3733812989","state":["exists","up"]},{"osd":3,"uuid":"f81c41e1-3a69-40db-a90f-58151286b081","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6811","nonce":2178310135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6813","nonce":2178310135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6817","nonce":2178310135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":2178310135},{"type":"v1","addr":"172.21.15.117:6815","nonce":2178310135}]},"public_addr":"172.21.15.117:6811/2178310135","cluster_addr":"172.21.15.117:6813/2178310135","heartbeat_back_addr":"172.21.15.117:6817/2178310135","heartbeat_front_addr":"172.21.15.117:6815/2178310135","state":["exists","up"]},{"osd":4,"uuid":"caf7ce86-caf0-4eaf-bbab-bd2d93e63c33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6809","nonce":3455398086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6811","nonce":3455398086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6815","nonce":3455398086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3455398086},{"type":"v1","addr":"172.21.15.119:6813","nonce":3455398086}]},"public_addr":"172.21.15.119:6809/3455398086","cluster_addr":"172.21.15.119:6811/3455398086","heartbeat_back_addr":"172.21.15.119:6815/3455398086","heartbeat_front_addr":"172.21.15.119:6813/3455398086","state":["exists","up"]},{"osd":5,"uuid":"20ba0b8a-2e0f-4ea7-b435-430e44766200","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6809","nonce":1259631912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6811","nonce":1259631912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6815","nonce":1259631912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":1259631912},{"type":"v1","addr":"172.21.15.178:6813","nonce":1259631912}]},"public_addr":"172.21.15.178:6809/1259631912","cluster_addr":"172.21.15.178:6811/1259631912","heartbeat_back_addr":"172.21.15.178:6815/1259631912","heartbeat_front_addr":"172.21.15.178:6813/1259631912","state":["exists","up"]},{"osd":6,"uuid":"75c2ddd7-f837-42e0-ac47-8cfbcf97a52a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6819","nonce":864146757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6821","nonce":864146757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6825","nonce":864146757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":864146757},{"type":"v1","addr":"172.21.15.117:6823","nonce":864146757}]},"public_addr":"172.21.15.117:6819/864146757","cluster_addr":"172.21.15.117:6821/864146757","heartbeat_back_addr":"172.21.15.117:6825/864146757","heartbeat_front_addr":"172.21.15.117:6823/864146757","state":["exists","up"]},{"osd":7,"uuid":"929fd0a4-2cc7-4b54-a1b6-e32cc501ea9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6817","nonce":1273365545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6819","nonce":1273365545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6823","nonce":1273365545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1273365545},{"type":"v1","addr":"172.21.15.119:6821","nonce":1273365545}]},"public_addr":"172.21.15.119:6817/1273365545","cluster_addr":"172.21.15.119:6819/1273365545","heartbeat_back_addr":"172.21.15.119:6823/1273365545","heartbeat_front_addr":"172.21.15.119:6821/1273365545","state":["exists","up"]},{"osd":8,"uuid":"fd371461-a538-4781-92e1-5eead4ca3864","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6817","nonce":1059587501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6819","nonce":1059587501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6823","nonce":1059587501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":1059587501},{"type":"v1","addr":"172.21.15.178:6821","nonce":1059587501}]},"public_addr":"172.21.15.178:6817/1059587501","cluster_addr":"172.21.15.178:6819/1059587501","heartbeat_back_addr":"172.21.15.178:6823/1059587501","heartbeat_front_addr":"172.21.15.178:6821/1059587501","state":["exists","up"]},{"osd":9,"uuid":"0f2bff95-b14e-4d0e-ac1e-54d94fe3d2f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6827","nonce":3653875600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6829","nonce":3653875600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6833","nonce":3653875600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3653875600},{"type":"v1","addr":"172.21.15.117:6831","nonce":3653875600}]},"public_addr":"172.21.15.117:6827/3653875600","cluster_addr":"172.21.15.117:6829/3653875600","heartbeat_back_addr":"172.21.15.117:6833/3653875600","heartbeat_front_addr":"172.21.15.117:6831/3653875600","state":["exists","up"]},{"osd":10,"uuid":"de2049da-d19e-4386-afbf-fd6b5fddbcba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6825","nonce":1050976194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6827","nonce":1050976194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6831","nonce":1050976194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1050976194},{"type":"v1","addr":"172.21.15.119:6829","nonce":1050976194}]},"public_addr":"172.21.15.119:6825/1050976194","cluster_addr":"172.21.15.119:6827/1050976194","heartbeat_back_addr":"172.21.15.119:6831/1050976194","heartbeat_front_addr":"172.21.15.119:6829/1050976194","state":["exists","up"]},{"osd":11,"uuid":"3cccc0f9-d273-4b6b-9b8e-c926371472dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6825","nonce":2915080368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6827","nonce":2915080368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6831","nonce":2915080368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":2915080368},{"type":"v1","addr":"172.21.15.178:6829","nonce":2915080368}]},"public_addr":"172.21.15.178:6825/2915080368","cluster_addr":"172.21.15.178:6827/2915080368","heartbeat_back_addr":"172.21.15.178:6831/2915080368","heartbeat_front_addr":"172.21.15.178:6829/2915080368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:27.876274+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:24:53.094706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:16.598813+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:25:48.160419+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:14.048925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:26:38.814304+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:12.542328+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:27:38.942747+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:04.488963+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:28:38.196053+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:05.254728+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T03:29:30.874915+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7},{"from":5,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.117:0/185684447":"2024-03-20T03:22:38.134578+0000","172.21.15.117:0/361367101":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/2788067603":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6801/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3506200675":"2024-03-20T03:22:38.134578+0000","172.21.15.117:6800/1570899077":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3283471541":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/3750920169":"2024-03-20T03:22:03.905461+0000","172.21.15.117:0/2780813298":"2024-03-20T03:22:03.905461+0000","172.21.15.117:6800/2788067603":"2024-03-20T03:22:38.134578+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T03:42:57.252 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.251+0000 7f4bbb200640 1 -- 172.21.15.117:0/539826350 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4b980a2800 msgr2=0x7f4b980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:57.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.251+0000 7f4bbb200640 1 --2- 172.21.15.117:0/539826350 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4b980a2800 0x7f4b980a4cc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4ba402c240 tx=0x7f4ba402df90 comp rx=0 tx=0).stop 2024-03-19T03:42:57.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.251+0000 7f4bbb200640 1 -- 172.21.15.117:0/539826350 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4bb4104be0 msgr2=0x7f4bb4100260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:57.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.251+0000 7f4bbb200640 1 --2- 172.21.15.117:0/539826350 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4bb4104be0 0x7f4bb4100260 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000e9f0 tx=0x7f4ba000eeb0 comp rx=0 tx=0).stop 2024-03-19T03:42:57.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.251+0000 7f4bbb200640 1 -- 172.21.15.117:0/539826350 shutdown_connections 2024-03-19T03:42:57.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.252+0000 7f4bbb200640 1 --2- 172.21.15.117:0/539826350 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4bb4107510 0x7f4bb41007a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:57.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.252+0000 7f4bbb200640 1 --2- 172.21.15.117:0/539826350 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f4b980a2800 0x7f4b980a4cc0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:57.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.252+0000 7f4bbb200640 1 --2- 172.21.15.117:0/539826350 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4bb4104be0 0x7f4bb4100260 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:57.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.252+0000 7f4bbb200640 1 --2- 172.21.15.117:0/539826350 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4bb41022b0 0x7f4bb4101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:57.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.252+0000 7f4bbb200640 1 -- 172.21.15.117:0/539826350 >> 172.21.15.117:0/539826350 conn(0x7f4bb40fbca0 msgr2=0x7f4bb4102bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:57.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.252+0000 7f4bbb200640 1 -- 172.21.15.117:0/539826350 shutdown_connections 2024-03-19T03:42:57.253 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.252+0000 7f4bbb200640 1 -- 172.21.15.117:0/539826350 wait complete. 2024-03-19T03:42:57.274 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-19T03:42:57.475 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.474+0000 7f38fbe00640 1 -- 172.21.15.117:0/1686472519 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38fc104b10 msgr2=0x7f38fc104f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:57.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.474+0000 7f38fbe00640 1 --2- 172.21.15.117:0/1686472519 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38fc104b10 0x7f38fc104f90 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f38e4009ca0 tx=0x7f38e402c100 comp rx=0 tx=0).stop 2024-03-19T03:42:57.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.474+0000 7f38fbe00640 1 -- 172.21.15.117:0/1686472519 shutdown_connections 2024-03-19T03:42:57.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.474+0000 7f38fbe00640 1 --2- 172.21.15.117:0/1686472519 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38fc1054d0 0x7f38fc10bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:57.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.474+0000 7f38fbe00640 1 --2- 172.21.15.117:0/1686472519 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38fc104b10 0x7f38fc104f90 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:57.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.474+0000 7f38fbe00640 1 --2- 172.21.15.117:0/1686472519 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38fc103920 0x7f38fc103d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:57.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.474+0000 7f38fbe00640 1 -- 172.21.15.117:0/1686472519 >> 172.21.15.117:0/1686472519 conn(0x7f38fc0fee90 msgr2=0x7f38fc1012d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:57.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.474+0000 7f38fbe00640 1 -- 172.21.15.117:0/1686472519 shutdown_connections 2024-03-19T03:42:57.476 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.474+0000 7f38fbe00640 1 -- 172.21.15.117:0/1686472519 wait complete. 2024-03-19T03:42:57.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.475+0000 7f38fbe00640 1 Processor -- start 2024-03-19T03:42:57.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.475+0000 7f38fbe00640 1 -- start start 2024-03-19T03:42:57.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.476+0000 7f38fbe00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38fc103920 0x7f38fc199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:57.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.476+0000 7f38fbe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38fc104b10 0x7f38fc19a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:57.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.476+0000 7f38fbe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38fc1054d0 0x7f38fc1a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:57.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.476+0000 7f38fbe00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f38fc10f5a0 con 0x7f38fc104b10 2024-03-19T03:42:57.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.476+0000 7f38fbe00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f38fc10f420 con 0x7f38fc103920 2024-03-19T03:42:57.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.476+0000 7f38fbe00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f38fc10f720 con 0x7f38fc1054d0 2024-03-19T03:42:57.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.476+0000 7f38f2000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38fc104b10 0x7f38fc19a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:57.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.476+0000 7f38faa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38fc103920 0x7f38fc199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:57.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.476+0000 7f38faa00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38fc103920 0x7f38fc199bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:44158/0 (socket says 172.21.15.117:44158) 2024-03-19T03:42:57.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.476+0000 7f38faa00640 1 -- 172.21.15.117:0/1627759250 learned_addr learned my addr 172.21.15.117:0/1627759250 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:42:57.479 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.476+0000 7f38fb400640 1 --2- 172.21.15.117:0/1627759250 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38fc1054d0 0x7f38fc1a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:57.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.476+0000 7f38faa00640 1 -- 172.21.15.117:0/1627759250 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38fc1054d0 msgr2=0x7f38fc1a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:57.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.476+0000 7f38faa00640 1 --2- 172.21.15.117:0/1627759250 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38fc1054d0 0x7f38fc1a1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:57.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.476+0000 7f38faa00640 1 -- 172.21.15.117:0/1627759250 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38fc104b10 msgr2=0x7f38fc19a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:57.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.476+0000 7f38faa00640 1 --2- 172.21.15.117:0/1627759250 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38fc104b10 0x7f38fc19a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:57.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.476+0000 7f38faa00640 1 -- 172.21.15.117:0/1627759250 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38fc1a17d0 con 0x7f38fc103920 2024-03-19T03:42:57.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.477+0000 7f38faa00640 1 --2- 172.21.15.117:0/1627759250 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38fc103920 0x7f38fc199bb0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f38ec00e9c0 tx=0x7f38ec00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:57.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.477+0000 7f3900a00640 1 -- 172.21.15.117:0/1627759250 <== mon.2 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ec00cde0 con 0x7f38fc103920 2024-03-19T03:42:57.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.477+0000 7f3900a00640 1 -- 172.21.15.117:0/1627759250 <== mon.2 v2:172.21.15.119:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38ec010910 con 0x7f38fc103920 2024-03-19T03:42:57.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.477+0000 7f38fbe00640 1 -- 172.21.15.117:0/1627759250 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38fc1a1ac0 con 0x7f38fc103920 2024-03-19T03:42:57.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.477+0000 7f3900a00640 1 -- 172.21.15.117:0/1627759250 <== mon.2 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ec018a00 con 0x7f38fc103920 2024-03-19T03:42:57.480 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.477+0000 7f38fbe00640 1 -- 172.21.15.117:0/1627759250 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38fc1a1fe0 con 0x7f38fc103920 2024-03-19T03:42:57.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.478+0000 7f38fbe00640 1 -- 172.21.15.117:0/1627759250 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38fc10ac40 con 0x7f38fc103920 2024-03-19T03:42:57.481 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.480+0000 7f3900a00640 1 -- 172.21.15.117:0/1627759250 <== mon.2 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 20) ==== 137291+0+0 (secure 0 0 0) 0x7f38ec01f020 con 0x7f38fc103920 2024-03-19T03:42:57.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.483+0000 7f3900a00640 1 --2- 172.21.15.117:0/1627759250 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f38e00a2800 0x7f38e00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:57.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.483+0000 7f3900a00640 1 -- 172.21.15.117:0/1627759250 <== mon.2 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9245+0+0 (secure 0 0 0) 0x7f38ec0cf1f0 con 0x7f38fc103920 2024-03-19T03:42:57.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.483+0000 7f38f2000640 1 --2- 172.21.15.117:0/1627759250 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f38e00a2800 0x7f38e00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:57.484 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.484+0000 7f38f2000640 1 --2- 172.21.15.117:0/1627759250 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f38e00a2800 0x7f38e00a4cc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f38e4006010 tx=0x7f38e4032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:57.489 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.488+0000 7f3900a00640 1 -- 172.21.15.117:0/1627759250 <== mon.2 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f38ec09be40 con 0x7f38fc103920 2024-03-19T03:42:57.863 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.862+0000 7f38fbe00640 1 -- 172.21.15.117:0/1627759250 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f38fc02d1f0 con 0x7f38fc103920 2024-03-19T03:42:57.873 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.872+0000 7f3900a00640 1 -- 172.21.15.117:0/1627759250 <== mon.2 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v46) ==== 161+0+0 (secure 0 0 0) 0x7f38ec09b740 con 0x7f38fc103920 2024-03-19T03:42:57.874 INFO:teuthology.orchestra.run.smithi117.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-19T03:42:57.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.875+0000 7f38fbe00640 1 -- 172.21.15.117:0/1627759250 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f38e00a2800 msgr2=0x7f38e00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:57.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.875+0000 7f38fbe00640 1 --2- 172.21.15.117:0/1627759250 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f38e00a2800 0x7f38e00a4cc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f38e4006010 tx=0x7f38e4032590 comp rx=0 tx=0).stop 2024-03-19T03:42:57.876 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.875+0000 7f38fbe00640 1 -- 172.21.15.117:0/1627759250 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38fc103920 msgr2=0x7f38fc199bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:57.877 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.875+0000 7f38fbe00640 1 --2- 172.21.15.117:0/1627759250 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38fc103920 0x7f38fc199bb0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f38ec00e9c0 tx=0x7f38ec00ee80 comp rx=0 tx=0).stop 2024-03-19T03:42:57.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.877+0000 7f38fbe00640 1 -- 172.21.15.117:0/1627759250 shutdown_connections 2024-03-19T03:42:57.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.877+0000 7f38fbe00640 1 --2- 172.21.15.117:0/1627759250 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38fc1054d0 0x7f38fc1a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:57.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.877+0000 7f38fbe00640 1 --2- 172.21.15.117:0/1627759250 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38fc104b10 0x7f38fc19a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:57.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.877+0000 7f38fbe00640 1 --2- 172.21.15.117:0/1627759250 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f38e00a2800 0x7f38e00a4cc0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:57.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.877+0000 7f38fbe00640 1 --2- 172.21.15.117:0/1627759250 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38fc103920 0x7f38fc199bb0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:57.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.877+0000 7f38fbe00640 1 -- 172.21.15.117:0/1627759250 >> 172.21.15.117:0/1627759250 conn(0x7f38fc0fee90 msgr2=0x7f38fc107e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:57.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.877+0000 7f38fbe00640 1 -- 172.21.15.117:0/1627759250 shutdown_connections 2024-03-19T03:42:57.878 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:57.877+0000 7f38fbe00640 1 -- 172.21.15.117:0/1627759250 wait complete. 2024-03-19T03:42:57.897 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-19T03:42:57.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:57 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1159976059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:42:57.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:57 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/539826350' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:42:57.985 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 19 03:42:57 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-b[97021]: ignoring --setuser ceph since I am not root 2024-03-19T03:42:57.985 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 19 03:42:57 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-b[97021]: ignoring --setgroup ceph since I am not root 2024-03-19T03:42:57.985 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 19 03:42:57 smithi119 ceph-mds[97025]: -- 172.21.15.119:0/94836981 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e6bc07ba40 con 0x55e6bc04ec00 2024-03-19T03:42:57.985 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 19 03:42:57 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-b[97021]: starting mds.b at 2024-03-19T03:42:57.986 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 19 03:42:57 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-e[96402]: ignoring --setuser ceph since I am not root 2024-03-19T03:42:57.986 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 19 03:42:57 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-e[96402]: ignoring --setgroup ceph since I am not root 2024-03-19T03:42:57.986 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 19 03:42:57 smithi119 ceph-mds[96406]: -- 172.21.15.119:0/3536654627 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b405e17a40 con 0x55b405deb000 2024-03-19T03:42:57.986 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 19 03:42:57 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-e[96402]: starting mds.e at 2024-03-19T03:42:57.986 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 19 03:42:57 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-h[96710]: ignoring --setuser ceph since I am not root 2024-03-19T03:42:57.986 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 19 03:42:57 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-h[96710]: ignoring --setgroup ceph since I am not root 2024-03-19T03:42:57.986 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 19 03:42:57 smithi119 ceph-mds[96714]: -- 172.21.15.119:0/903292637 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c291fd7a40 con 0x55c291faac00 2024-03-19T03:42:57.986 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 19 03:42:57 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-h[96710]: starting mds.h at 2024-03-19T03:42:57.987 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 19 03:42:57 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-k[96080]: ignoring --setuser ceph since I am not root 2024-03-19T03:42:57.987 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 19 03:42:57 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-k[96080]: ignoring --setgroup ceph since I am not root 2024-03-19T03:42:57.987 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 19 03:42:57 smithi119 ceph-mds[96084]: -- 172.21.15.119:0/2086787253 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56074aa01a40 con 0x56074a9d4c00 2024-03-19T03:42:57.987 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 19 03:42:57 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-k[96080]: starting mds.k at 2024-03-19T03:42:58.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:57 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1159976059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:42:58.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:57 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/539826350' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:42:58.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:57 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-a[31834]: 2024-03-19T03:42:57.864+0000 7f8a3f600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-19T03:42:58.042 INFO:journalctl@ceph.mds.a.smithi117.stdout:Mar 19 03:42:57 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-a[73998]: ignoring --setuser ceph since I am not root 2024-03-19T03:42:58.042 INFO:journalctl@ceph.mds.a.smithi117.stdout:Mar 19 03:42:57 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-a[73998]: ignoring --setgroup ceph since I am not root 2024-03-19T03:42:58.042 INFO:journalctl@ceph.mds.a.smithi117.stdout:Mar 19 03:42:57 smithi117 ceph-mds[74018]: -- 172.21.15.117:0/183510261 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55772dabda40 con 0x55772da90c00 2024-03-19T03:42:58.042 INFO:journalctl@ceph.mds.a.smithi117.stdout:Mar 19 03:42:57 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-a[73998]: starting mds.a at 2024-03-19T03:42:58.043 INFO:journalctl@ceph.mds.g.smithi117.stdout:Mar 19 03:42:57 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-g[73651]: ignoring --setuser ceph since I am not root 2024-03-19T03:42:58.043 INFO:journalctl@ceph.mds.g.smithi117.stdout:Mar 19 03:42:57 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-g[73651]: ignoring --setgroup ceph since I am not root 2024-03-19T03:42:58.043 INFO:journalctl@ceph.mds.g.smithi117.stdout:Mar 19 03:42:57 smithi117 ceph-mds[73655]: -- 172.21.15.117:0/1609570650 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5610a2893a40 con 0x5610a2866800 2024-03-19T03:42:58.043 INFO:journalctl@ceph.mds.g.smithi117.stdout:Mar 19 03:42:57 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-g[73651]: starting mds.g at 2024-03-19T03:42:58.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.084+0000 7f455bc00640 1 -- 172.21.15.117:0/443282622 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4554108fb0 msgr2=0x7f4554109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:58.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.084+0000 7f455bc00640 1 --2- 172.21.15.117:0/443282622 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4554108fb0 0x7f4554109430 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f453c009860 tx=0x7f453c02bb70 comp rx=0 tx=0).stop 2024-03-19T03:42:58.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.084+0000 7f455bc00640 1 -- 172.21.15.117:0/443282622 shutdown_connections 2024-03-19T03:42:58.085 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.084+0000 7f455bc00640 1 --2- 172.21.15.117:0/443282622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4554109970 0x7f4554110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:58.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.084+0000 7f455bc00640 1 --2- 172.21.15.117:0/443282622 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4554108fb0 0x7f4554109430 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:58.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.084+0000 7f455bc00640 1 --2- 172.21.15.117:0/443282622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4554107e60 0x7f4554108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:58.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.084+0000 7f455bc00640 1 -- 172.21.15.117:0/443282622 >> 172.21.15.117:0/443282622 conn(0x7f4554076e10 msgr2=0x7f4554079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:58.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.084+0000 7f455bc00640 1 -- 172.21.15.117:0/443282622 shutdown_connections 2024-03-19T03:42:58.086 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.084+0000 7f455bc00640 1 -- 172.21.15.117:0/443282622 wait complete. 2024-03-19T03:42:58.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.085+0000 7f455bc00640 1 Processor -- start 2024-03-19T03:42:58.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.085+0000 7f455bc00640 1 -- start start 2024-03-19T03:42:58.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.085+0000 7f455bc00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4554107e60 0x7f455419e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:58.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.085+0000 7f455bc00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4554108fb0 0x7f455419e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:58.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.085+0000 7f455bc00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4554109970 0x7f45541a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:58.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.085+0000 7f455bc00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4554113b00 con 0x7f4554108fb0 2024-03-19T03:42:58.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.085+0000 7f455bc00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f4554113980 con 0x7f4554107e60 2024-03-19T03:42:58.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.085+0000 7f455bc00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f4554113c80 con 0x7f4554109970 2024-03-19T03:42:58.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.086+0000 7f4559c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4554109970 0x7f45541a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:58.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.086+0000 7f454be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4554108fb0 0x7f455419e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:58.088 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.086+0000 7f4559200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4554107e60 0x7f455419e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:58.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.086+0000 7f4559c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4554109970 0x7f45541a5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:45550/0 (socket says 172.21.15.117:45550) 2024-03-19T03:42:58.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.086+0000 7f4559200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4554107e60 0x7f455419e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.117:44228/0 (socket says 172.21.15.117:44228) 2024-03-19T03:42:58.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.086+0000 7f4559c00640 1 -- 172.21.15.117:0/1329982212 learned_addr learned my addr 172.21.15.117:0/1329982212 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:42:58.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.086+0000 7f4559c00640 1 -- 172.21.15.117:0/1329982212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4554107e60 msgr2=0x7f455419e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:58.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.086+0000 7f4559c00640 1 --2- 172.21.15.117:0/1329982212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4554107e60 0x7f455419e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:58.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.086+0000 7f4559c00640 1 -- 172.21.15.117:0/1329982212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4554108fb0 msgr2=0x7f455419e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:58.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.086+0000 7f4559c00640 1 --2- 172.21.15.117:0/1329982212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4554108fb0 0x7f455419e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:58.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.086+0000 7f4559c00640 1 -- 172.21.15.117:0/1329982212 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45541a5d70 con 0x7f4554109970 2024-03-19T03:42:58.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.086+0000 7f454be00640 1 --2- 172.21.15.117:0/1329982212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4554108fb0 0x7f455419e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:42:58.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.086+0000 7f4559c00640 1 --2- 172.21.15.117:0/1329982212 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4554109970 0x7f45541a5700 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f454c00efc0 tx=0x7f454c00c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:58.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.087+0000 7f4552000640 1 -- 172.21.15.117:0/1329982212 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454c003970 con 0x7f4554109970 2024-03-19T03:42:58.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.087+0000 7f4552000640 1 -- 172.21.15.117:0/1329982212 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f454c004020 con 0x7f4554109970 2024-03-19T03:42:58.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.087+0000 7f455bc00640 1 -- 172.21.15.117:0/1329982212 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45541a6060 con 0x7f4554109970 2024-03-19T03:42:58.089 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.087+0000 7f4552000640 1 -- 172.21.15.117:0/1329982212 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454c00b9f0 con 0x7f4554109970 2024-03-19T03:42:58.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.087+0000 7f455bc00640 1 -- 172.21.15.117:0/1329982212 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45541a6580 con 0x7f4554109970 2024-03-19T03:42:58.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.088+0000 7f455bc00640 1 -- 172.21.15.117:0/1329982212 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f455410f110 con 0x7f4554109970 2024-03-19T03:42:58.090 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.090+0000 7f4552000640 1 -- 172.21.15.117:0/1329982212 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 20) ==== 137291+0+0 (secure 0 0 0) 0x7f454c01d020 con 0x7f4554109970 2024-03-19T03:42:58.091 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.091+0000 7f4552000640 1 --2- 172.21.15.117:0/1329982212 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f45380a2880 0x7f45380a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:58.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.091+0000 7f4552000640 1 -- 172.21.15.117:0/1329982212 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9675+0+0 (secure 0 0 0) 0x7f454c0cee60 con 0x7f4554109970 2024-03-19T03:42:58.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.091+0000 7f4559200640 1 --2- 172.21.15.117:0/1329982212 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f45380a2880 0x7f45380a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:58.093 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.092+0000 7f4559200640 1 --2- 172.21.15.117:0/1329982212 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f45380a2880 0x7f45380a4d40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4544009bd0 tx=0x7f4544005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:58.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:57 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1159976059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T03:42:58.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:57 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/539826350' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T03:42:58.096 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 19 03:42:57 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-c[98443]: ignoring --setuser ceph since I am not root 2024-03-19T03:42:58.096 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 19 03:42:57 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-c[98443]: ignoring --setgroup ceph since I am not root 2024-03-19T03:42:58.096 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 19 03:42:57 smithi178 ceph-mds[98447]: -- 172.21.15.178:0/2928692738 <== mon.2 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561cd4ae5a40 con 0x561cd4ab8c00 2024-03-19T03:42:58.096 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 19 03:42:57 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-c[98443]: starting mds.c at 2024-03-19T03:42:58.096 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 19 03:42:57 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-l[97823]: ignoring --setuser ceph since I am not root 2024-03-19T03:42:58.097 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 19 03:42:57 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-l[97823]: ignoring --setgroup ceph since I am not root 2024-03-19T03:42:58.097 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 19 03:42:57 smithi178 ceph-mds[97827]: -- 172.21.15.178:0/2531361311 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55a6853044e0 con 0x55a6852dac00 2024-03-19T03:42:58.097 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 19 03:42:57 smithi178 ceph-mds[97827]: -- 172.21.15.178:0/2531361311 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a685307a40 con 0x55a6852dac00 2024-03-19T03:42:58.097 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 19 03:42:57 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-l[97823]: starting mds.l at 2024-03-19T03:42:58.097 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 19 03:42:57 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-f[98131]: ignoring --setuser ceph since I am not root 2024-03-19T03:42:58.097 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 19 03:42:57 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-f[98131]: ignoring --setgroup ceph since I am not root 2024-03-19T03:42:58.097 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 19 03:42:57 smithi178 ceph-mds[98135]: -- 172.21.15.178:0/3689095532 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x565013fb3a40 con 0x565013f86c00 2024-03-19T03:42:58.097 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 19 03:42:57 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-f[98131]: starting mds.f at 2024-03-19T03:42:58.098 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 19 03:42:57 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-i[98749]: ignoring --setuser ceph since I am not root 2024-03-19T03:42:58.098 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 19 03:42:57 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-i[98749]: ignoring --setgroup ceph since I am not root 2024-03-19T03:42:58.098 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 19 03:42:57 smithi178 ceph-mds[98753]: -- 172.21.15.178:0/3618488363 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557d7d8bc1a0 con 0x557d7d892c00 2024-03-19T03:42:58.098 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 19 03:42:57 smithi178 ceph-mds[98753]: -- 172.21.15.178:0/3618488363 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557d7d8bfa40 con 0x557d7d892c00 2024-03-19T03:42:58.098 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 19 03:42:57 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-i[98749]: starting mds.i at 2024-03-19T03:42:58.098 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.095+0000 7f4552000640 1 -- 172.21.15.117:0/1329982212 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f454c09b900 con 0x7f4554109970 2024-03-19T03:42:58.339 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.338+0000 7f455bc00640 1 -- 172.21.15.117:0/1329982212 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f4554062420 con 0x7f4554109970 2024-03-19T03:42:58.880 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.879+0000 7f4552000640 1 -- 172.21.15.117:0/1329982212 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v47) ==== 104+0+0 (secure 0 0 0) 0x7f454c09b200 con 0x7f4554109970 2024-03-19T03:42:58.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.883+0000 7f455bc00640 1 -- 172.21.15.117:0/1329982212 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f45380a2880 msgr2=0x7f45380a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:58.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.883+0000 7f455bc00640 1 --2- 172.21.15.117:0/1329982212 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f45380a2880 0x7f45380a4d40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4544009bd0 tx=0x7f4544005f90 comp rx=0 tx=0).stop 2024-03-19T03:42:58.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.883+0000 7f455bc00640 1 -- 172.21.15.117:0/1329982212 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4554109970 msgr2=0x7f45541a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:58.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.883+0000 7f455bc00640 1 --2- 172.21.15.117:0/1329982212 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4554109970 0x7f45541a5700 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f454c00efc0 tx=0x7f454c00c800 comp rx=0 tx=0).stop 2024-03-19T03:42:58.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.883+0000 7f455bc00640 1 -- 172.21.15.117:0/1329982212 shutdown_connections 2024-03-19T03:42:58.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.883+0000 7f455bc00640 1 --2- 172.21.15.117:0/1329982212 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4554109970 0x7f45541a5700 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:58.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.883+0000 7f455bc00640 1 --2- 172.21.15.117:0/1329982212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4554108fb0 0x7f455419e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:58.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.883+0000 7f455bc00640 1 --2- 172.21.15.117:0/1329982212 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f45380a2880 0x7f45380a4d40 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:58.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.883+0000 7f455bc00640 1 --2- 172.21.15.117:0/1329982212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4554107e60 0x7f455419e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:58.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.883+0000 7f455bc00640 1 -- 172.21.15.117:0/1329982212 >> 172.21.15.117:0/1329982212 conn(0x7f4554076e10 msgr2=0x7f455410c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:58.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.884+0000 7f455bc00640 1 -- 172.21.15.117:0/1329982212 shutdown_connections 2024-03-19T03:42:58.886 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:58.884+0000 7f455bc00640 1 -- 172.21.15.117:0/1329982212 wait complete. 2024-03-19T03:42:58.908 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-19T03:42:58.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:58 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1627759250' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-19T03:42:58.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:58 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-19T03:42:58.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:58 smithi119 ceph-mon[66861]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-19T03:42:58.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:58 smithi119 ceph-mon[66861]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-19T03:42:58.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:58 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-19T03:42:58.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:58 smithi119 ceph-mon[66861]: osdmap e94: 12 total, 12 up, 12 in 2024-03-19T03:42:58.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:58 smithi119 ceph-mon[66861]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-19T03:42:58.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:58 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/1329982212' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T03:42:58.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:58 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T03:42:58.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:58 smithi119 ceph-mon[66861]: pgmap v694: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-19T03:42:59.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:58 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1627759250' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-19T03:42:59.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:58 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-19T03:42:59.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:58 smithi117 ceph-mon[31838]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-19T03:42:59.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:58 smithi117 ceph-mon[31838]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-19T03:42:59.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:58 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-19T03:42:59.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:58 smithi117 ceph-mon[31838]: osdmap e94: 12 total, 12 up, 12 in 2024-03-19T03:42:59.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:58 smithi117 ceph-mon[31838]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-19T03:42:59.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:58 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/1329982212' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T03:42:59.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:58 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T03:42:59.043 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:58 smithi117 ceph-mon[31838]: pgmap v694: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-19T03:42:59.095 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:58 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1627759250' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-19T03:42:59.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:58 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-19T03:42:59.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:58 smithi178 ceph-mon[67578]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-19T03:42:59.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:58 smithi178 ceph-mon[67578]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-19T03:42:59.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:58 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-19T03:42:59.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:58 smithi178 ceph-mon[67578]: osdmap e94: 12 total, 12 up, 12 in 2024-03-19T03:42:59.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:58 smithi178 ceph-mon[67578]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-19T03:42:59.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:58 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/1329982212' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T03:42:59.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:58 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T03:42:59.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:58 smithi178 ceph-mon[67578]: pgmap v694: 129 pgs: 129 active+clean; 246 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-19T03:42:59.107 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.102+0000 7fa5ab200640 1 -- 172.21.15.117:0/2572951175 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5a4103a40 msgr2=0x7fa5a4103e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:59.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.102+0000 7fa5ab200640 1 --2- 172.21.15.117:0/2572951175 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5a4103a40 0x7fa5a4103e60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa594009900 tx=0x7fa59402bc30 comp rx=0 tx=0).stop 2024-03-19T03:42:59.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.103+0000 7fa5ab200640 1 -- 172.21.15.117:0/2572951175 shutdown_connections 2024-03-19T03:42:59.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.103+0000 7fa5ab200640 1 --2- 172.21.15.117:0/2572951175 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5a41055f0 0x7fa5a410be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:59.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.103+0000 7fa5ab200640 1 --2- 172.21.15.117:0/2572951175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5a4104c30 0x7fa5a41050b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:59.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.103+0000 7fa5ab200640 1 --2- 172.21.15.117:0/2572951175 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5a4103a40 0x7fa5a4103e60 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:59.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.103+0000 7fa5ab200640 1 -- 172.21.15.117:0/2572951175 >> 172.21.15.117:0/2572951175 conn(0x7fa5a40fefd0 msgr2=0x7fa5a41013f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:59.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.103+0000 7fa5ab200640 1 -- 172.21.15.117:0/2572951175 shutdown_connections 2024-03-19T03:42:59.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.103+0000 7fa5ab200640 1 -- 172.21.15.117:0/2572951175 wait complete. 2024-03-19T03:42:59.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.104+0000 7fa5ab200640 1 Processor -- start 2024-03-19T03:42:59.108 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.104+0000 7fa5ab200640 1 -- start start 2024-03-19T03:42:59.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.104+0000 7fa5ab200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5a4103a40 0x7fa5a4199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:59.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.104+0000 7fa5ab200640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5a4104c30 0x7fa5a419a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:59.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.104+0000 7fa5ab200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5a41055f0 0x7fa5a41a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:59.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.104+0000 7fa5ab200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a410f6c0 con 0x7fa5a4103a40 2024-03-19T03:42:59.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.104+0000 7fa5ab200640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a410f540 con 0x7fa5a4104c30 2024-03-19T03:42:59.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.104+0000 7fa5ab200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a410f840 con 0x7fa5a41055f0 2024-03-19T03:42:59.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.104+0000 7fa5a3e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5a4103a40 0x7fa5a4199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:59.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.104+0000 7fa5a3e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5a4103a40 0x7fa5a4199d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:41352/0 (socket says 172.21.15.117:41352) 2024-03-19T03:42:59.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.104+0000 7fa5a3e00640 1 -- 172.21.15.117:0/2146388029 learned_addr learned my addr 172.21.15.117:0/2146388029 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:42:59.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.105+0000 7fa59be00640 1 --2- 172.21.15.117:0/2146388029 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5a4104c30 0x7fa5a419a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:59.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.105+0000 7fa5a9200640 1 --2- 172.21.15.117:0/2146388029 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5a41055f0 0x7fa5a41a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:59.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.105+0000 7fa5a3e00640 1 -- 172.21.15.117:0/2146388029 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5a41055f0 msgr2=0x7fa5a41a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:59.109 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.105+0000 7fa5a3e00640 1 --2- 172.21.15.117:0/2146388029 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5a41055f0 0x7fa5a41a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:59.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.105+0000 7fa5a3e00640 1 -- 172.21.15.117:0/2146388029 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5a4104c30 msgr2=0x7fa5a419a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:59.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.105+0000 7fa5a3e00640 1 --2- 172.21.15.117:0/2146388029 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5a4104c30 0x7fa5a419a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:59.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.105+0000 7fa5a3e00640 1 -- 172.21.15.117:0/2146388029 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5a41a1980 con 0x7fa5a4103a40 2024-03-19T03:42:59.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.105+0000 7fa5a3e00640 1 --2- 172.21.15.117:0/2146388029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5a4103a40 0x7fa5a4199d60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa594000c00 tx=0x7fa59402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:59.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.105+0000 7fa5a1600640 1 -- 172.21.15.117:0/2146388029 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa594041070 con 0x7fa5a4103a40 2024-03-19T03:42:59.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.105+0000 7fa5a1600640 1 -- 172.21.15.117:0/2146388029 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa594004450 con 0x7fa5a4103a40 2024-03-19T03:42:59.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.105+0000 7fa5ab200640 1 -- 172.21.15.117:0/2146388029 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5a41a1c10 con 0x7fa5a4103a40 2024-03-19T03:42:59.110 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.106+0000 7fa5a1600640 1 -- 172.21.15.117:0/2146388029 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa594034440 con 0x7fa5a4103a40 2024-03-19T03:42:59.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.106+0000 7fa5ab200640 1 -- 172.21.15.117:0/2146388029 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5a41a2100 con 0x7fa5a4103a40 2024-03-19T03:42:59.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.107+0000 7fa5ab200640 1 -- 172.21.15.117:0/2146388029 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5a410ad80 con 0x7fa5a4103a40 2024-03-19T03:42:59.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.109+0000 7fa5a1600640 1 -- 172.21.15.117:0/2146388029 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 20) ==== 137291+0+0 (secure 0 0 0) 0x7fa594044070 con 0x7fa5a4103a40 2024-03-19T03:42:59.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.110+0000 7fa5a1600640 1 --2- 172.21.15.117:0/2146388029 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa5880a2800 0x7fa5880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:42:59.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.110+0000 7fa5a1600640 1 -- 172.21.15.117:0/2146388029 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9675+0+0 (secure 0 0 0) 0x7fa5940f0e90 con 0x7fa5a4103a40 2024-03-19T03:42:59.111 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.110+0000 7fa59be00640 1 --2- 172.21.15.117:0/2146388029 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa5880a2800 0x7fa5880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:42:59.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.111+0000 7fa59be00640 1 --2- 172.21.15.117:0/2146388029 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa5880a2800 0x7fa5880a4cc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa58c0095f0 tx=0x7fa58c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:42:59.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.113+0000 7fa5a1600640 1 -- 172.21.15.117:0/2146388029 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa5940bd9c0 con 0x7fa5a4103a40 2024-03-19T03:42:59.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.313+0000 7fa5ab200640 1 -- 172.21.15.117:0/2146388029 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa5a402d1f0 con 0x7fa5a4103a40 2024-03-19T03:42:59.880 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.879+0000 7fa5a1600640 1 -- 172.21.15.117:0/2146388029 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v95) ==== 181+0+0 (secure 0 0 0) 0x7fa5940bd2c0 con 0x7fa5a4103a40 2024-03-19T03:42:59.881 INFO:teuthology.orchestra.run.smithi117.stderr:pool 'cephfs_metadata' removed 2024-03-19T03:42:59.887 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.886+0000 7fa5ab200640 1 -- 172.21.15.117:0/2146388029 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa5880a2800 msgr2=0x7fa5880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:59.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.887+0000 7fa5ab200640 1 --2- 172.21.15.117:0/2146388029 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa5880a2800 0x7fa5880a4cc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa58c0095f0 tx=0x7fa58c005f90 comp rx=0 tx=0).stop 2024-03-19T03:42:59.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.887+0000 7fa5ab200640 1 -- 172.21.15.117:0/2146388029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5a4103a40 msgr2=0x7fa5a4199d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:42:59.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.887+0000 7fa5ab200640 1 --2- 172.21.15.117:0/2146388029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5a4103a40 0x7fa5a4199d60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa594000c00 tx=0x7fa59402c640 comp rx=0 tx=0).stop 2024-03-19T03:42:59.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.887+0000 7fa5ab200640 1 -- 172.21.15.117:0/2146388029 shutdown_connections 2024-03-19T03:42:59.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.887+0000 7fa5ab200640 1 --2- 172.21.15.117:0/2146388029 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5a41055f0 0x7fa5a41a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:59.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.887+0000 7fa5ab200640 1 --2- 172.21.15.117:0/2146388029 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7fa5880a2800 0x7fa5880a4cc0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:59.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.887+0000 7fa5ab200640 1 --2- 172.21.15.117:0/2146388029 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5a4104c30 0x7fa5a419a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:42:59.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.887+0000 7fa5ab200640 1 --2- 172.21.15.117:0/2146388029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5a4103a40 0x7fa5a4199d60 secure :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa594000c00 tx=0x7fa59402c640 comp rx=0 tx=0).stop 2024-03-19T03:42:59.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.887+0000 7fa5ab200640 1 -- 172.21.15.117:0/2146388029 >> 172.21.15.117:0/2146388029 conn(0x7fa5a40fefd0 msgr2=0x7fa5a4107f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:42:59.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.887+0000 7fa5ab200640 1 -- 172.21.15.117:0/2146388029 shutdown_connections 2024-03-19T03:42:59.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:42:59.887+0000 7fa5ab200640 1 -- 172.21.15.117:0/2146388029 wait complete. 2024-03-19T03:42:59.911 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-19T03:42:59.979 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-19T03:42:59.979 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-19T03:42:59.979 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: Cluster is now healthy 2024-03-19T03:42:59.979 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-19T03:42:59.979 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.119:6836/835367091,v1:172.21.15.119:6837/835367091] up:boot 2024-03-19T03:42:59.979 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.178:6834/1777128900,v1:172.21.15.178:6835/1777128900] up:boot 2024-03-19T03:42:59.980 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.119:6838/1683758926,v1:172.21.15.119:6839/1683758926] up:boot 2024-03-19T03:42:59.980 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.178:6838/1569891469,v1:172.21.15.178:6839/1569891469] up:boot 2024-03-19T03:42:59.980 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.119:6832/1272684646,v1:172.21.15.119:6834/1272684646] up:boot 2024-03-19T03:42:59.980 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.178:6832/1553448325,v1:172.21.15.178:6833/1553448325] up:boot 2024-03-19T03:42:59.980 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.119:6833/1823244672,v1:172.21.15.119:6835/1823244672] up:boot 2024-03-19T03:42:59.980 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.178:6836/3425420012,v1:172.21.15.178:6837/3425420012] up:boot 2024-03-19T03:42:59.980 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.117:6838/2385796423,v1:172.21.15.117:6839/2385796423] up:boot 2024-03-19T03:42:59.980 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: mds.? [v2:172.21.15.117:6840/2694746731,v1:172.21.15.117:6841/2694746731] up:boot 2024-03-19T03:42:59.980 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: fsmap 12 up:standby 2024-03-19T03:42:59.980 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T03:42:59.980 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T03:42:59.980 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T03:42:59.980 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T03:42:59.980 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T03:42:59.980 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T03:42:59.981 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T03:42:59.981 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T03:42:59.981 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T03:42:59.981 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T03:42:59.981 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:42:59 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2146388029' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T03:42:59.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-19T03:42:59.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-19T03:42:59.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: Cluster is now healthy 2024-03-19T03:42:59.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-19T03:42:59.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.119:6836/835367091,v1:172.21.15.119:6837/835367091] up:boot 2024-03-19T03:42:59.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.178:6834/1777128900,v1:172.21.15.178:6835/1777128900] up:boot 2024-03-19T03:42:59.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.119:6838/1683758926,v1:172.21.15.119:6839/1683758926] up:boot 2024-03-19T03:42:59.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.178:6838/1569891469,v1:172.21.15.178:6839/1569891469] up:boot 2024-03-19T03:42:59.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.119:6832/1272684646,v1:172.21.15.119:6834/1272684646] up:boot 2024-03-19T03:42:59.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.178:6832/1553448325,v1:172.21.15.178:6833/1553448325] up:boot 2024-03-19T03:42:59.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.119:6833/1823244672,v1:172.21.15.119:6835/1823244672] up:boot 2024-03-19T03:42:59.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.178:6836/3425420012,v1:172.21.15.178:6837/3425420012] up:boot 2024-03-19T03:42:59.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.117:6838/2385796423,v1:172.21.15.117:6839/2385796423] up:boot 2024-03-19T03:42:59.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: mds.? [v2:172.21.15.117:6840/2694746731,v1:172.21.15.117:6841/2694746731] up:boot 2024-03-19T03:42:59.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: fsmap 12 up:standby 2024-03-19T03:42:59.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T03:42:59.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T03:42:59.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T03:42:59.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T03:42:59.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T03:42:59.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T03:42:59.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T03:42:59.985 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T03:42:59.986 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T03:42:59.986 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T03:42:59.986 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:42:59 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2146388029' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T03:43:00.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-19T03:43:00.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-19T03:43:00.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: Cluster is now healthy 2024-03-19T03:43:00.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-19T03:43:00.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.119:6836/835367091,v1:172.21.15.119:6837/835367091] up:boot 2024-03-19T03:43:00.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.178:6834/1777128900,v1:172.21.15.178:6835/1777128900] up:boot 2024-03-19T03:43:00.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.119:6838/1683758926,v1:172.21.15.119:6839/1683758926] up:boot 2024-03-19T03:43:00.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.178:6838/1569891469,v1:172.21.15.178:6839/1569891469] up:boot 2024-03-19T03:43:00.096 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.119:6832/1272684646,v1:172.21.15.119:6834/1272684646] up:boot 2024-03-19T03:43:00.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.178:6832/1553448325,v1:172.21.15.178:6833/1553448325] up:boot 2024-03-19T03:43:00.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.119:6833/1823244672,v1:172.21.15.119:6835/1823244672] up:boot 2024-03-19T03:43:00.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.178:6836/3425420012,v1:172.21.15.178:6837/3425420012] up:boot 2024-03-19T03:43:00.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.117:6838/2385796423,v1:172.21.15.117:6839/2385796423] up:boot 2024-03-19T03:43:00.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: mds.? [v2:172.21.15.117:6840/2694746731,v1:172.21.15.117:6841/2694746731] up:boot 2024-03-19T03:43:00.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: fsmap 12 up:standby 2024-03-19T03:43:00.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T03:43:00.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T03:43:00.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T03:43:00.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T03:43:00.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T03:43:00.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T03:43:00.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T03:43:00.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T03:43:00.097 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T03:43:00.098 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: from='mgr.14152 172.21.15.117:0/3901654794' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T03:43:00.098 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:42:59 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2146388029' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T03:43:00.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.117+0000 7f8186000640 1 -- 172.21.15.117:0/1654063704 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8180102a20 msgr2=0x7f8180102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:43:00.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.117+0000 7f8186000640 1 --2- 172.21.15.117:0/1654063704 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8180102a20 0x7f8180102ea0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f816c009860 tx=0x7f816c02bb70 comp rx=0 tx=0).stop 2024-03-19T03:43:00.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.118+0000 7f8186000640 1 -- 172.21.15.117:0/1654063704 shutdown_connections 2024-03-19T03:43:00.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.118+0000 7f8186000640 1 --2- 172.21.15.117:0/1654063704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f81801033e0 0x7f8180109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:43:00.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.118+0000 7f8186000640 1 --2- 172.21.15.117:0/1654063704 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8180102a20 0x7f8180102ea0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:43:00.119 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.118+0000 7f8186000640 1 --2- 172.21.15.117:0/1654063704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8180101830 0x7f8180101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:43:00.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.118+0000 7f8186000640 1 -- 172.21.15.117:0/1654063704 >> 172.21.15.117:0/1654063704 conn(0x7f81800fcdc0 msgr2=0x7f81800ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:43:00.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.118+0000 7f8186000640 1 -- 172.21.15.117:0/1654063704 shutdown_connections 2024-03-19T03:43:00.120 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.118+0000 7f8186000640 1 -- 172.21.15.117:0/1654063704 wait complete. 2024-03-19T03:43:00.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.119+0000 7f8186000640 1 Processor -- start 2024-03-19T03:43:00.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.119+0000 7f8186000640 1 -- start start 2024-03-19T03:43:00.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.119+0000 7f8186000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8180101830 0x7f8180197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:43:00.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.119+0000 7f8186000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8180102a20 0x7f8180198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:43:00.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.119+0000 7f8186000640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f81801033e0 0x7f818019f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:43:00.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.119+0000 7f8186000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f818010d4b0 con 0x7f8180101830 2024-03-19T03:43:00.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.119+0000 7f8186000640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f818010d330 con 0x7f81801033e0 2024-03-19T03:43:00.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.119+0000 7f8186000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f818010d630 con 0x7f8180102a20 2024-03-19T03:43:00.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.119+0000 7f817f400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8180101830 0x7f8180197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:43:00.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.119+0000 7f817f400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8180101830 0x7f8180197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:41368/0 (socket says 172.21.15.117:41368) 2024-03-19T03:43:00.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.119+0000 7f817f400640 1 -- 172.21.15.117:0/3190126232 learned_addr learned my addr 172.21.15.117:0/3190126232 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:43:00.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.120+0000 7f817ea00640 1 --2- 172.21.15.117:0/3190126232 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8180102a20 0x7f8180198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:43:00.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.120+0000 7f817fe00640 1 --2- 172.21.15.117:0/3190126232 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f81801033e0 0x7f818019f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:43:00.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.120+0000 7f817f400640 1 -- 172.21.15.117:0/3190126232 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8180102a20 msgr2=0x7f8180198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:43:00.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.120+0000 7f817f400640 1 --2- 172.21.15.117:0/3190126232 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8180102a20 0x7f8180198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:43:00.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.120+0000 7f817f400640 1 -- 172.21.15.117:0/3190126232 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f81801033e0 msgr2=0x7f818019f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:43:00.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.120+0000 7f817f400640 1 --2- 172.21.15.117:0/3190126232 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f81801033e0 0x7f818019f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:43:00.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.120+0000 7f817f400640 1 -- 172.21.15.117:0/3190126232 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f818019f770 con 0x7f8180101830 2024-03-19T03:43:00.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.120+0000 7f817ea00640 1 --2- 172.21.15.117:0/3190126232 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8180102a20 0x7f8180198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T03:43:00.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.120+0000 7f817f400640 1 --2- 172.21.15.117:0/3190126232 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8180101830 0x7f8180197b50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f817000e9a0 tx=0x7f817000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:43:00.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.120+0000 7f8177e00640 1 -- 172.21.15.117:0/3190126232 <== mon.0 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817000cd80 con 0x7f8180101830 2024-03-19T03:43:00.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.120+0000 7f8186000640 1 -- 172.21.15.117:0/3190126232 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f818019fa60 con 0x7f8180101830 2024-03-19T03:43:00.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.121+0000 7f8177e00640 1 -- 172.21.15.117:0/3190126232 <== mon.0 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8170010910 con 0x7f8180101830 2024-03-19T03:43:00.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.121+0000 7f8177e00640 1 -- 172.21.15.117:0/3190126232 <== mon.0 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8170018a40 con 0x7f8180101830 2024-03-19T03:43:00.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.121+0000 7f8186000640 1 -- 172.21.15.117:0/3190126232 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f818019ffd0 con 0x7f8180101830 2024-03-19T03:43:00.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.122+0000 7f8186000640 1 -- 172.21.15.117:0/3190126232 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8180068a60 con 0x7f8180101830 2024-03-19T03:43:00.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.125+0000 7f8177e00640 1 -- 172.21.15.117:0/3190126232 <== mon.0 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 20) ==== 137291+0+0 (secure 0 0 0) 0x7f8170010ac0 con 0x7f8180101830 2024-03-19T03:43:00.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.126+0000 7f8177e00640 1 --2- 172.21.15.117:0/3190126232 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f81640a2800 0x7f81640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:43:00.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.126+0000 7f8177e00640 1 -- 172.21.15.117:0/3190126232 <== mon.0 v2:172.21.15.117:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9121+0+0 (secure 0 0 0) 0x7f81700cf730 con 0x7f8180101830 2024-03-19T03:43:00.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.127+0000 7f817ea00640 1 --2- 172.21.15.117:0/3190126232 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f81640a2800 0x7f81640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:43:00.128 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.127+0000 7f817ea00640 1 --2- 172.21.15.117:0/3190126232 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f81640a2800 0x7f81640a4cc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f816c02c180 tx=0x7f816c02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:43:00.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.129+0000 7f8177e00640 1 -- 172.21.15.117:0/3190126232 <== mon.0 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8170026020 con 0x7f8180101830 2024-03-19T03:43:00.332 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.330+0000 7f8186000640 1 -- 172.21.15.117:0/3190126232 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8180108b50 con 0x7f8180101830 2024-03-19T03:43:00.890 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.889+0000 7f8177e00640 1 -- 172.21.15.117:0/3190126232 <== mon.0 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v96) ==== 169+0+0 (secure 0 0 0) 0x7f817009c3f0 con 0x7f8180101830 2024-03-19T03:43:00.890 INFO:teuthology.orchestra.run.smithi117.stderr:pool 'cephfs_data' removed 2024-03-19T03:43:00.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.893+0000 7f8186000640 1 -- 172.21.15.117:0/3190126232 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f81640a2800 msgr2=0x7f81640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:43:00.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.893+0000 7f8186000640 1 --2- 172.21.15.117:0/3190126232 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f81640a2800 0x7f81640a4cc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f816c02c180 tx=0x7f816c02e040 comp rx=0 tx=0).stop 2024-03-19T03:43:00.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.893+0000 7f8186000640 1 -- 172.21.15.117:0/3190126232 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8180101830 msgr2=0x7f8180197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:43:00.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.893+0000 7f8186000640 1 --2- 172.21.15.117:0/3190126232 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8180101830 0x7f8180197b50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f817000e9a0 tx=0x7f817000ee60 comp rx=0 tx=0).stop 2024-03-19T03:43:00.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.893+0000 7f8186000640 1 -- 172.21.15.117:0/3190126232 shutdown_connections 2024-03-19T03:43:00.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.893+0000 7f8186000640 1 --2- 172.21.15.117:0/3190126232 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f81640a2800 0x7f81640a4cc0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:43:00.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.893+0000 7f8186000640 1 --2- 172.21.15.117:0/3190126232 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f81801033e0 0x7f818019f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:43:00.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.893+0000 7f8186000640 1 --2- 172.21.15.117:0/3190126232 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8180102a20 0x7f8180198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:43:00.894 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.893+0000 7f8186000640 1 --2- 172.21.15.117:0/3190126232 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8180101830 0x7f8180197b50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:43:00.895 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.893+0000 7f8186000640 1 -- 172.21.15.117:0/3190126232 >> 172.21.15.117:0/3190126232 conn(0x7f81800fcdc0 msgr2=0x7f8180105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:43:00.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.893+0000 7f8186000640 1 -- 172.21.15.117:0/3190126232 shutdown_connections 2024-03-19T03:43:00.896 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:00.893+0000 7f8186000640 1 -- 172.21.15.117:0/3190126232 wait complete. 2024-03-19T03:43:00.923 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-19T03:43:01.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.120+0000 7f8f9fc00640 1 -- 172.21.15.117:0/896201145 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f98105610 msgr2=0x7f8f9810bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:43:01.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.120+0000 7f8f9fc00640 1 --2- 172.21.15.117:0/896201145 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f98105610 0x7f8f9810bea0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8f90009be0 tx=0x7f8f9002bef0 comp rx=0 tx=0).stop 2024-03-19T03:43:01.121 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.120+0000 7f8f9fc00640 1 -- 172.21.15.117:0/896201145 shutdown_connections 2024-03-19T03:43:01.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.120+0000 7f8f9fc00640 1 --2- 172.21.15.117:0/896201145 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f98105610 0x7f8f9810bea0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:43:01.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.120+0000 7f8f9fc00640 1 --2- 172.21.15.117:0/896201145 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f98104c50 0x7f8f981050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:43:01.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.120+0000 7f8f9fc00640 1 --2- 172.21.15.117:0/896201145 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f98103a60 0x7f8f98103e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:43:01.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.120+0000 7f8f9fc00640 1 -- 172.21.15.117:0/896201145 >> 172.21.15.117:0/896201145 conn(0x7f8f980fefd0 msgr2=0x7f8f98101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:43:01.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.120+0000 7f8f9fc00640 1 -- 172.21.15.117:0/896201145 shutdown_connections 2024-03-19T03:43:01.122 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.120+0000 7f8f9fc00640 1 -- 172.21.15.117:0/896201145 wait complete. 2024-03-19T03:43:01.123 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.121+0000 7f8f9fc00640 1 Processor -- start 2024-03-19T03:43:01.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.121+0000 7f8f9fc00640 1 -- start start 2024-03-19T03:43:01.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.122+0000 7f8f9fc00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f98103a60 0x7f8f98199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:43:01.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.122+0000 7f8f9fc00640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f98104c50 0x7f8f9819a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:43:01.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.122+0000 7f8f9fc00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f98105610 0x7f8f981a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:43:01.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.122+0000 7f8f9fc00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9810f710 con 0x7f8f98105610 2024-03-19T03:43:01.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.122+0000 7f8f9fc00640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9810f590 con 0x7f8f98104c50 2024-03-19T03:43:01.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.122+0000 7f8f9fc00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9810f890 con 0x7f8f98103a60 2024-03-19T03:43:01.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.122+0000 7f8f9d200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f98103a60 0x7f8f98199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:43:01.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.122+0000 7f8f9d200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f98103a60 0x7f8f98199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.117:45610/0 (socket says 172.21.15.117:45610) 2024-03-19T03:43:01.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.122+0000 7f8f9d200640 1 -- 172.21.15.117:0/303345136 learned_addr learned my addr 172.21.15.117:0/303345136 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-19T03:43:01.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.122+0000 7f8f97e00640 1 --2- 172.21.15.117:0/303345136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f98104c50 0x7f8f9819a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:43:01.124 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.122+0000 7f8f9dc00640 1 --2- 172.21.15.117:0/303345136 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f98105610 0x7f8f981a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:43:01.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.122+0000 7f8f9d200640 1 -- 172.21.15.117:0/303345136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f98104c50 msgr2=0x7f8f9819a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:43:01.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.122+0000 7f8f9d200640 1 --2- 172.21.15.117:0/303345136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f98104c50 0x7f8f9819a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:43:01.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.122+0000 7f8f9d200640 1 -- 172.21.15.117:0/303345136 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f98105610 msgr2=0x7f8f981a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:43:01.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.122+0000 7f8f9d200640 1 --2- 172.21.15.117:0/303345136 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f98105610 0x7f8f981a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:43:01.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.122+0000 7f8f9d200640 1 -- 172.21.15.117:0/303345136 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f981a1920 con 0x7f8f98103a60 2024-03-19T03:43:01.125 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.123+0000 7f8f9d200640 1 --2- 172.21.15.117:0/303345136 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f98103a60 0x7f8f98199d10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c00e9f0 tx=0x7f8f8c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:43:01.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.123+0000 7f8f95600640 1 -- 172.21.15.117:0/303345136 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c00ccc0 con 0x7f8f98103a60 2024-03-19T03:43:01.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.123+0000 7f8f9fc00640 1 -- 172.21.15.117:0/303345136 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f981a1c10 con 0x7f8f98103a60 2024-03-19T03:43:01.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.123+0000 7f8f95600640 1 -- 172.21.15.117:0/303345136 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f8c0035f0 con 0x7f8f98103a60 2024-03-19T03:43:01.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.123+0000 7f8f95600640 1 -- 172.21.15.117:0/303345136 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c00ce60 con 0x7f8f98103a60 2024-03-19T03:43:01.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.123+0000 7f8f9fc00640 1 -- 172.21.15.117:0/303345136 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f981a21b0 con 0x7f8f98103a60 2024-03-19T03:43:01.126 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.124+0000 7f8f9fc00640 1 -- 172.21.15.117:0/303345136 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f9810ad80 con 0x7f8f98103a60 2024-03-19T03:43:01.127 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.127+0000 7f8f95600640 1 -- 172.21.15.117:0/303345136 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 20) ==== 137291+0+0 (secure 0 0 0) 0x7f8f8c0037a0 con 0x7f8f98103a60 2024-03-19T03:43:01.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.128+0000 7f8f95600640 1 --2- 172.21.15.117:0/303345136 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8f800a2880 0x7f8f800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T03:43:01.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.128+0000 7f8f95600640 1 -- 172.21.15.117:0/303345136 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8496+0+0 (secure 0 0 0) 0x7f8f8c0ceef0 con 0x7f8f98103a60 2024-03-19T03:43:01.129 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.128+0000 7f8f97e00640 1 --2- 172.21.15.117:0/303345136 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8f800a2880 0x7f8f800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T03:43:01.130 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.129+0000 7f8f97e00640 1 --2- 172.21.15.117:0/303345136 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8f800a2880 0x7f8f800a4d40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8f84009bd0 tx=0x7f8f84005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T03:43:01.131 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.130+0000 7f8f95600640 1 -- 172.21.15.117:0/303345136 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8f8c09be50 con 0x7f8f98103a60 2024-03-19T03:43:01.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:43:00 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/2146388029' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-19T03:43:01.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:43:00 smithi119 ceph-mon[66861]: osdmap e95: 12 total, 12 up, 12 in 2024-03-19T03:43:01.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:43:00 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3190126232' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T03:43:01.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:43:00 smithi119 ceph-mon[66861]: pgmap v696: 97 pgs: 97 active+clean; 577 KiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:43:01.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:43:00 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/2146388029' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-19T03:43:01.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:43:00 smithi117 ceph-mon[31838]: osdmap e95: 12 total, 12 up, 12 in 2024-03-19T03:43:01.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:43:00 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3190126232' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T03:43:01.292 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:43:00 smithi117 ceph-mon[31838]: pgmap v696: 97 pgs: 97 active+clean; 577 KiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:43:01.330 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.329+0000 7f8f9fc00640 1 -- 172.21.15.117:0/303345136 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8f9802d1f0 con 0x7f8f98103a60 2024-03-19T03:43:01.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:43:00 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/2146388029' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-19T03:43:01.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:43:00 smithi178 ceph-mon[67578]: osdmap e95: 12 total, 12 up, 12 in 2024-03-19T03:43:01.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:43:00 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3190126232' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T03:43:01.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:43:00 smithi178 ceph-mon[67578]: pgmap v696: 97 pgs: 97 active+clean; 577 KiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T03:43:01.906 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.904+0000 7f8f95600640 1 -- 172.21.15.117:0/303345136 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v97) ==== 178+0+0 (secure 0 0 0) 0x7f8f8c09b750 con 0x7f8f98103a60 2024-03-19T03:43:01.906 INFO:teuthology.orchestra.run.smithi117.stderr:pool 'cephfs_data_ec' removed 2024-03-19T03:43:01.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.910+0000 7f8f9fc00640 1 -- 172.21.15.117:0/303345136 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8f800a2880 msgr2=0x7f8f800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:43:01.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.910+0000 7f8f9fc00640 1 --2- 172.21.15.117:0/303345136 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8f800a2880 0x7f8f800a4d40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8f84009bd0 tx=0x7f8f84005f90 comp rx=0 tx=0).stop 2024-03-19T03:43:01.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.910+0000 7f8f9fc00640 1 -- 172.21.15.117:0/303345136 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f98103a60 msgr2=0x7f8f98199d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T03:43:01.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.910+0000 7f8f9fc00640 1 --2- 172.21.15.117:0/303345136 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f98103a60 0x7f8f98199d10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c00e9f0 tx=0x7f8f8c00eeb0 comp rx=0 tx=0).stop 2024-03-19T03:43:01.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.910+0000 7f8f9fc00640 1 -- 172.21.15.117:0/303345136 shutdown_connections 2024-03-19T03:43:01.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.910+0000 7f8f9fc00640 1 --2- 172.21.15.117:0/303345136 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f98105610 0x7f8f981a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:43:01.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.910+0000 7f8f9fc00640 1 --2- 172.21.15.117:0/303345136 >> [v2:172.21.15.117:6800/3144614841,v1:172.21.15.117:6801/3144614841] conn(0x7f8f800a2880 0x7f8f800a4d40 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:43:01.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.910+0000 7f8f9fc00640 1 --2- 172.21.15.117:0/303345136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f98104c50 0x7f8f9819a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:43:01.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.910+0000 7f8f9fc00640 1 --2- 172.21.15.117:0/303345136 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f98103a60 0x7f8f98199d10 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T03:43:01.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.910+0000 7f8f9fc00640 1 -- 172.21.15.117:0/303345136 >> 172.21.15.117:0/303345136 conn(0x7f8f980fefd0 msgr2=0x7f8f98107f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T03:43:01.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.911+0000 7f8f9fc00640 1 -- 172.21.15.117:0/303345136 shutdown_connections 2024-03-19T03:43:01.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-19T03:43:01.911+0000 7f8f9fc00640 1 -- 172.21.15.117:0/303345136 wait complete. 2024-03-19T03:43:01.936 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-19T03:43:02.015 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-19T03:43:02.055 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-19T03:43:02.086 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-19T03:43:02.086 DEBUG:teuthology.orchestra.run.smithi117:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-19T03:43:02.135 DEBUG:teuthology.orchestra.run.smithi119:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-19T03:43:02.150 DEBUG:teuthology.orchestra.run.smithi178:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-19T03:43:02.164 INFO:tasks.cephadm:Stopping all daemons... 2024-03-19T03:43:02.164 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-19T03:43:02.165 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mon.a 2024-03-19T03:43:02.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:43:01 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/3190126232' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-19T03:43:02.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:43:01 smithi119 ceph-mon[66861]: osdmap e96: 12 total, 12 up, 12 in 2024-03-19T03:43:02.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:43:01 smithi119 ceph-mon[66861]: from='client.? 172.21.15.117:0/303345136' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T03:43:02.234 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:43:01 smithi119 ceph-mon[66861]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T03:43:02.247 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:43:01 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/3190126232' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-19T03:43:02.247 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:43:01 smithi178 ceph-mon[67578]: osdmap e96: 12 total, 12 up, 12 in 2024-03-19T03:43:02.247 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:43:01 smithi178 ceph-mon[67578]: from='client.? 172.21.15.117:0/303345136' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T03:43:02.247 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:43:01 smithi178 ceph-mon[67578]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T03:43:02.259 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:43:01 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/3190126232' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-19T03:43:02.259 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:43:01 smithi117 ceph-mon[31838]: osdmap e96: 12 total, 12 up, 12 in 2024-03-19T03:43:02.259 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:43:01 smithi117 ceph-mon[31838]: from='client.? 172.21.15.117:0/303345136' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T03:43:02.259 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:43:01 smithi117 ceph-mon[31838]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T03:43:02.542 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:43:02 smithi117 systemd[1]: Stopping Ceph mon.a for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:43:03.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:43:02 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-a[31834]: 2024-03-19T03:43:02.737+0000 7f8a46400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:43:03.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:43:02 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-a[31834]: 2024-03-19T03:43:02.737+0000 7f8a46400700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-19T03:43:03.042 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:43:02 smithi117 podman[92527]: 2024-03-19 03:43:02.779790962 +0000 UTC m=+0.206510591 container died e14c366900a5afcce7dc85a3c5542c09342ca4aca90ffd51346751015653e315 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-a, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-19T03:43:03.478 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:43:03 smithi117 podman[92527]: 2024-03-19 03:43:03.234169703 +0000 UTC m=+0.660889380 container cleanup e14c366900a5afcce7dc85a3c5542c09342ca4aca90ffd51346751015653e315 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-19T03:43:03.478 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:43:03 smithi117 bash[92527]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-a 2024-03-19T03:43:03.792 INFO:journalctl@ceph.mon.a.smithi117.stdout:Mar 19 03:43:03 smithi117 podman[92538]: 2024-03-19 03:43:03.477208141 +0000 UTC m=+0.685679501 container remove e14c366900a5afcce7dc85a3c5542c09342ca4aca90ffd51346751015653e315 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T03:43:03.921 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mon.a.service' 2024-03-19T03:43:04.049 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:43:04.050 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-19T03:43:04.050 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-19T03:43:04.050 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mon.b 2024-03-19T03:43:04.391 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:43:04 smithi119 systemd[1]: Stopping Ceph mon.b for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:43:04.699 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:43:04 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-b[66857]: 2024-03-19T03:43:04.389+0000 7f514f42b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:43:04.699 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:43:04 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-b[66857]: 2024-03-19T03:43:04.389+0000 7f514f42b700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-19T03:43:04.700 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:43:04 smithi119 podman[100807]: 2024-03-19 03:43:04.411275427 +0000 UTC m=+0.138913959 container died b6d5cb3916da079d87763d159f2e5f4bea9f66bc4ed15c91dd1cc7b3087dccfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-19T03:43:04.983 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:43:04 smithi119 podman[100807]: 2024-03-19 03:43:04.80721929 +0000 UTC m=+0.534857829 container cleanup b6d5cb3916da079d87763d159f2e5f4bea9f66bc4ed15c91dd1cc7b3087dccfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True) 2024-03-19T03:43:04.984 INFO:journalctl@ceph.mon.b.smithi119.stdout:Mar 19 03:43:04 smithi119 bash[100807]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-b 2024-03-19T03:43:05.422 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mon.b.service' 2024-03-19T03:43:05.474 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:43:05.474 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-19T03:43:05.474 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-19T03:43:05.474 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mon.c 2024-03-19T03:43:05.814 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:43:05 smithi178 systemd[1]: Stopping Ceph mon.c for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:43:06.092 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:43:05 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-c[67574]: 2024-03-19T03:43:05.812+0000 7f68bafe6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:43:06.092 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:43:05 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-c[67574]: 2024-03-19T03:43:05.812+0000 7f68bafe6700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-19T03:43:06.092 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:43:05 smithi178 podman[103218]: 2024-03-19 03:43:05.834517396 +0000 UTC m=+0.164743338 container died fa156234d0fc055bf2208e77b34be92c16bb3146d9a8ef5724c397b6b6ae151f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T03:43:06.345 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:43:06 smithi178 podman[103218]: 2024-03-19 03:43:06.182781098 +0000 UTC m=+0.513007020 container cleanup fa156234d0fc055bf2208e77b34be92c16bb3146d9a8ef5724c397b6b6ae151f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-c, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-19T03:43:06.346 INFO:journalctl@ceph.mon.c.smithi178.stdout:Mar 19 03:43:06 smithi178 bash[103218]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mon-c 2024-03-19T03:43:06.650 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mon.c.service' 2024-03-19T03:43:06.703 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:43:06.703 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-19T03:43:06.703 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-19T03:43:06.703 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mgr.x 2024-03-19T03:43:07.042 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:43:06 smithi117 systemd[1]: Stopping Ceph mgr.x for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:43:07.786 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:43:07 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x[32063]: 2024-03-19T03:43:07.352+0000 7f25ae400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:43:07.786 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:43:07 smithi117 podman[92646]: 2024-03-19 03:43:07.438100975 +0000 UTC m=+0.315098056 container died fb84b6bcbbed166d71066578e1d5c4bcb6e842ee31c071ca68a87523d9e76dc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-19T03:43:08.042 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:43:07 smithi117 podman[92646]: 2024-03-19 03:43:07.963977284 +0000 UTC m=+0.840974402 container cleanup fb84b6bcbbed166d71066578e1d5c4bcb6e842ee31c071ca68a87523d9e76dc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-19T03:43:08.042 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:43:07 smithi117 bash[92646]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x 2024-03-19T03:43:08.542 INFO:journalctl@ceph.mgr.x.smithi117.stdout:Mar 19 03:43:08 smithi117 podman[92660]: 2024-03-19 03:43:08.224431203 +0000 UTC m=+0.765122751 container remove fb84b6bcbbed166d71066578e1d5c4bcb6e842ee31c071ca68a87523d9e76dc8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T03:43:08.711 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mgr.x.service' 2024-03-19T03:43:08.830 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:43:08.830 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-19T03:43:08.830 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-19T03:43:08.830 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mgr.y 2024-03-19T03:43:09.176 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:43:08 smithi119 systemd[1]: Stopping Ceph mgr.y for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:43:09.464 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:43:09 smithi119 podman[100938]: 2024-03-19 03:43:09.17631544 +0000 UTC m=+0.157798920 container died ed517e272106f9bd858316f92244da5659b60bb47255272a7873365ce93a0ff1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T03:43:09.734 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:43:09 smithi119 podman[100938]: 2024-03-19 03:43:09.563721321 +0000 UTC m=+0.545204801 container cleanup ed517e272106f9bd858316f92244da5659b60bb47255272a7873365ce93a0ff1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-19T03:43:09.734 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:43:09 smithi119 bash[100938]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y 2024-03-19T03:43:10.114 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:43:09 smithi119 podman[100952]: 2024-03-19 03:43:09.932316884 +0000 UTC m=+0.746170055 container remove ed517e272106f9bd858316f92244da5659b60bb47255272a7873365ce93a0ff1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5) 2024-03-19T03:43:10.115 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Mar 19 03:43:09 smithi119 systemd[1]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-19T03:43:10.125 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mgr.y.service' 2024-03-19T03:43:10.165 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:43:10.165 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-19T03:43:10.165 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-19T03:43:10.165 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mgr.z 2024-03-19T03:43:10.509 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:43:10 smithi178 systemd[1]: Stopping Ceph mgr.z for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:43:10.780 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:43:10 smithi178 podman[103340]: 2024-03-19 03:43:10.509092259 +0000 UTC m=+0.148225545 container died 7a6e85b68b39d5b03c427798b0afc852461803f5b1b32ddfb714e053b96b2018 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-19T03:43:11.095 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:43:10 smithi178 podman[103340]: 2024-03-19 03:43:10.888882865 +0000 UTC m=+0.528016138 container cleanup 7a6e85b68b39d5b03c427798b0afc852461803f5b1b32ddfb714e053b96b2018 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-19T03:43:11.096 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Mar 19 03:43:10 smithi178 bash[103340]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mgr-z 2024-03-19T03:43:11.353 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mgr.z.service' 2024-03-19T03:43:11.390 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:43:11.390 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-19T03:43:11.390 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-19T03:43:11.390 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.0 2024-03-19T03:43:11.792 INFO:journalctl@ceph.osd.0.smithi117.stdout:Mar 19 03:43:11 smithi117 systemd[1]: Stopping Ceph osd.0 for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:43:12.542 INFO:journalctl@ceph.osd.0.smithi117.stdout:Mar 19 03:43:12 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-0[46218]: 2024-03-19T03:43:12.282+0000 7fc8bfe00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:43:12.542 INFO:journalctl@ceph.osd.0.smithi117.stdout:Mar 19 03:43:12 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-0[46218]: 2024-03-19T03:43:12.282+0000 7fc8bfe00700 -1 osd.0 97 *** Got signal Terminated *** 2024-03-19T03:43:12.542 INFO:journalctl@ceph.osd.0.smithi117.stdout:Mar 19 03:43:12 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-0[46218]: 2024-03-19T03:43:12.282+0000 7fc8bfe00700 -1 osd.0 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T03:43:18.293 INFO:journalctl@ceph.osd.0.smithi117.stdout:Mar 19 03:43:18 smithi117 podman[92766]: 2024-03-19 03:43:18.007578463 +0000 UTC m=+6.134140211 container died 246df83a8f020077a0aa9b19fe2287a2911cd0ad96a172ffaf11e70634bf3247 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-19T03:43:18.712 INFO:journalctl@ceph.osd.0.smithi117.stdout:Mar 19 03:43:18 smithi117 podman[92766]: 2024-03-19 03:43:18.477279327 +0000 UTC m=+6.603841088 container cleanup 246df83a8f020077a0aa9b19fe2287a2911cd0ad96a172ffaf11e70634bf3247 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-19T03:43:18.712 INFO:journalctl@ceph.osd.0.smithi117.stdout:Mar 19 03:43:18 smithi117 bash[92766]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-0 2024-03-19T03:43:19.042 INFO:journalctl@ceph.osd.0.smithi117.stdout:Mar 19 03:43:18 smithi117 podman[92799]: 2024-03-19 03:43:18.711985579 +0000 UTC m=+0.688430248 container remove 246df83a8f020077a0aa9b19fe2287a2911cd0ad96a172ffaf11e70634bf3247 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T03:43:19.784 INFO:journalctl@ceph.osd.0.smithi117.stdout:Mar 19 03:43:19 smithi117 podman[92888]: 2024-03-19 03:43:19.381867563 +0000 UTC m=+0.034120637 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:43:19.785 INFO:journalctl@ceph.osd.0.smithi117.stdout:Mar 19 03:43:19 smithi117 podman[92888]: 2024-03-19 03:43:19.5047507 +0000 UTC m=+0.157003832 container create b37cce899c4334a79806edf16361f6db06b6fbbe7ff6bfd22f1792d865a6b114 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-19T03:43:19.785 INFO:journalctl@ceph.osd.0.smithi117.stdout:Mar 19 03:43:19 smithi117 podman[92888]: 2024-03-19 03:43:19.783590751 +0000 UTC m=+0.435843906 container init b37cce899c4334a79806edf16361f6db06b6fbbe7ff6bfd22f1792d865a6b114 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-19T03:43:20.042 INFO:journalctl@ceph.osd.0.smithi117.stdout:Mar 19 03:43:19 smithi117 podman[92888]: 2024-03-19 03:43:19.795654153 +0000 UTC m=+0.447907249 container start b37cce899c4334a79806edf16361f6db06b6fbbe7ff6bfd22f1792d865a6b114 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307) 2024-03-19T03:43:20.042 INFO:journalctl@ceph.osd.0.smithi117.stdout:Mar 19 03:43:19 smithi117 podman[92888]: 2024-03-19 03:43:19.890950448 +0000 UTC m=+0.543203587 container attach b37cce899c4334a79806edf16361f6db06b6fbbe7ff6bfd22f1792d865a6b114 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-19T03:43:20.542 INFO:journalctl@ceph.osd.0.smithi117.stdout:Mar 19 03:43:20 smithi117 podman[92888]: 2024-03-19 03:43:20.224830126 +0000 UTC m=+0.877083249 container died b37cce899c4334a79806edf16361f6db06b6fbbe7ff6bfd22f1792d865a6b114 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-0-deactivate, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-19T03:43:21.037 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.0.service' 2024-03-19T03:43:21.160 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:43:21.161 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-19T03:43:21.161 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-19T03:43:21.161 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.1 2024-03-19T03:43:21.484 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:43:21 smithi119 systemd[1]: Stopping Ceph osd.1 for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:43:21.984 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:43:21 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-1[71454]: 2024-03-19T03:43:21.493+0000 7f2a9ecad700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:43:21.984 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:43:21 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-1[71454]: 2024-03-19T03:43:21.493+0000 7f2a9ecad700 -1 osd.1 97 *** Got signal Terminated *** 2024-03-19T03:43:21.984 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:43:21 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-1[71454]: 2024-03-19T03:43:21.493+0000 7f2a9ecad700 -1 osd.1 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T03:43:27.221 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:43:26 smithi119 podman[101065]: 2024-03-19 03:43:26.921666175 +0000 UTC m=+5.562018856 container died daddd3056be3ca60862923560c26ee8e86da36c61af94f5e7272b2d46a847738 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-1, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-19T03:43:27.484 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:43:27 smithi119 podman[101065]: 2024-03-19 03:43:27.303976744 +0000 UTC m=+5.944329393 container cleanup daddd3056be3ca60862923560c26ee8e86da36c61af94f5e7272b2d46a847738 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-1, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-19T03:43:27.484 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:43:27 smithi119 bash[101065]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-1 2024-03-19T03:43:27.845 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:43:27 smithi119 podman[101082]: 2024-03-19 03:43:27.556556626 +0000 UTC m=+0.635164899 container remove daddd3056be3ca60862923560c26ee8e86da36c61af94f5e7272b2d46a847738 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-1, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-19T03:43:28.118 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:43:27 smithi119 podman[101170]: 2024-03-19 03:43:27.748542322 +0000 UTC m=+0.017483584 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:43:28.118 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:43:27 smithi119 podman[101170]: 2024-03-19 03:43:27.85716564 +0000 UTC m=+0.126106889 container create b4e90174e6a257102eccc8fb5301db7c4905f7d59c3fb61559d6a272368674a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-19T03:43:28.484 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:43:28 smithi119 podman[101170]: 2024-03-19 03:43:28.118885332 +0000 UTC m=+0.387826596 container init b4e90174e6a257102eccc8fb5301db7c4905f7d59c3fb61559d6a272368674a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T03:43:28.484 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:43:28 smithi119 podman[101170]: 2024-03-19 03:43:28.122184949 +0000 UTC m=+0.391126190 container start b4e90174e6a257102eccc8fb5301db7c4905f7d59c3fb61559d6a272368674a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-19T03:43:28.484 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:43:28 smithi119 podman[101170]: 2024-03-19 03:43:28.210521953 +0000 UTC m=+0.479463202 container attach b4e90174e6a257102eccc8fb5301db7c4905f7d59c3fb61559d6a272368674a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-19T03:43:28.830 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:43:28 smithi119 podman[101170]: 2024-03-19 03:43:28.514324539 +0000 UTC m=+0.783265805 container died b4e90174e6a257102eccc8fb5301db7c4905f7d59c3fb61559d6a272368674a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-19T03:43:29.071 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.1.service' 2024-03-19T03:43:29.097 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:43:29 smithi119 podman[101196]: 2024-03-19 03:43:29.056116634 +0000 UTC m=+0.532857010 container remove b4e90174e6a257102eccc8fb5301db7c4905f7d59c3fb61559d6a272368674a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:43:29.097 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:43:29 smithi119 systemd[1]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-19T03:43:29.097 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:43:29 smithi119 systemd[1]: Stopped Ceph osd.1 for a3a82456-e59f-11ee-95c9-87774f69a715. 2024-03-19T03:43:29.098 INFO:journalctl@ceph.osd.1.smithi119.stdout:Mar 19 03:43:29 smithi119 systemd[1]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.1.service: Consumed 1min 49.300s CPU time. 2024-03-19T03:43:29.114 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:43:29.114 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-19T03:43:29.114 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-19T03:43:29.114 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.2 2024-03-19T03:43:29.419 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 19 03:43:29 smithi178 systemd[1]: Stopping Ceph osd.2 for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:43:29.846 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 19 03:43:29 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-2[72716]: 2024-03-19T03:43:29.418+0000 7fc573fd7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:43:29.846 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 19 03:43:29 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-2[72716]: 2024-03-19T03:43:29.418+0000 7fc573fd7700 -1 osd.2 97 *** Got signal Terminated *** 2024-03-19T03:43:29.846 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 19 03:43:29 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-2[72716]: 2024-03-19T03:43:29.418+0000 7fc573fd7700 -1 osd.2 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T03:43:35.077 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 19 03:43:34 smithi178 podman[103463]: 2024-03-19 03:43:34.75111395 +0000 UTC m=+5.443844047 container died 9838e6b75cb7f8fe3f3a2be23f1d8d54b671102f9746c555cfd8169e43152182 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-19T03:43:35.077 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 19 03:43:35 smithi178 podman[103463]: 2024-03-19 03:43:35.077088942 +0000 UTC m=+5.769819018 container cleanup 9838e6b75cb7f8fe3f3a2be23f1d8d54b671102f9746c555cfd8169e43152182 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-19T03:43:35.346 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 19 03:43:35 smithi178 bash[103463]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-2 2024-03-19T03:43:35.346 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 19 03:43:35 smithi178 podman[103481]: 2024-03-19 03:43:35.287447766 +0000 UTC m=+0.537110902 container remove 9838e6b75cb7f8fe3f3a2be23f1d8d54b671102f9746c555cfd8169e43152182 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True) 2024-03-19T03:43:35.983 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 19 03:43:35 smithi178 podman[103574]: 2024-03-19 03:43:35.590102463 +0000 UTC m=+0.017206019 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:43:35.983 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 19 03:43:35 smithi178 podman[103574]: 2024-03-19 03:43:35.696949828 +0000 UTC m=+0.124053413 container create 7a24372a70baaf83f1880b407d133f4caffe1977f71ae885018e8695caaa6424 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-19T03:43:35.983 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 19 03:43:35 smithi178 podman[103574]: 2024-03-19 03:43:35.891965368 +0000 UTC m=+0.319068923 container init 7a24372a70baaf83f1880b407d133f4caffe1977f71ae885018e8695caaa6424 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-03-19T03:43:35.983 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 19 03:43:35 smithi178 podman[103574]: 2024-03-19 03:43:35.895344053 +0000 UTC m=+0.322447605 container start 7a24372a70baaf83f1880b407d133f4caffe1977f71ae885018e8695caaa6424 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-19T03:43:35.983 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 19 03:43:35 smithi178 podman[103574]: 2024-03-19 03:43:35.98308661 +0000 UTC m=+0.410190175 container attach 7a24372a70baaf83f1880b407d133f4caffe1977f71ae885018e8695caaa6424 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-19T03:43:36.346 INFO:journalctl@ceph.osd.2.smithi178.stdout:Mar 19 03:43:36 smithi178 podman[103574]: 2024-03-19 03:43:36.182255205 +0000 UTC m=+0.609358745 container died 7a24372a70baaf83f1880b407d133f4caffe1977f71ae885018e8695caaa6424 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-03-19T03:43:36.804 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.2.service' 2024-03-19T03:43:36.845 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:43:36.845 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-19T03:43:36.845 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-19T03:43:36.846 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.3 2024-03-19T03:43:37.291 INFO:journalctl@ceph.osd.3.smithi117.stdout:Mar 19 03:43:36 smithi117 systemd[1]: Stopping Ceph osd.3 for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:43:37.792 INFO:journalctl@ceph.osd.3.smithi117.stdout:Mar 19 03:43:37 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-3[53172]: 2024-03-19T03:43:37.419+0000 7fcaa2600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:43:37.792 INFO:journalctl@ceph.osd.3.smithi117.stdout:Mar 19 03:43:37 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-3[53172]: 2024-03-19T03:43:37.419+0000 7fcaa2600700 -1 osd.3 97 *** Got signal Terminated *** 2024-03-19T03:43:37.792 INFO:journalctl@ceph.osd.3.smithi117.stdout:Mar 19 03:43:37 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-3[53172]: 2024-03-19T03:43:37.419+0000 7fcaa2600700 -1 osd.3 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T03:43:38.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:38 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:38.319+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:43:38.596 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:38 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:38.564+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:15.765726+0000 front 2024-03-19T03:43:15.766177+0000 (oldest deadline 2024-03-19T03:43:38.065462+0000) 2024-03-19T03:43:39.042 INFO:journalctl@ceph.osd.3.smithi117.stdout:Mar 19 03:43:38 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-3[53172]: 2024-03-19T03:43:38.665+0000 7fca9de00700 -1 osd.3 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.394766+0000 front 2024-03-19T03:43:14.394285+0000 (oldest deadline 2024-03-19T03:43:38.494231+0000) 2024-03-19T03:43:39.592 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:39 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:39.323+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:43:39.846 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:39 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:39.592+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:15.765726+0000 front 2024-03-19T03:43:15.766177+0000 (oldest deadline 2024-03-19T03:43:38.065462+0000) 2024-03-19T03:43:39.963 INFO:journalctl@ceph.osd.3.smithi117.stdout:Mar 19 03:43:39 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-3[53172]: 2024-03-19T03:43:39.673+0000 7fca9de00700 -1 osd.3 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.394766+0000 front 2024-03-19T03:43:14.394285+0000 (oldest deadline 2024-03-19T03:43:38.494231+0000) 2024-03-19T03:43:40.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:39 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:39.962+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:43:40.560 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:40 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:40.277+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:43:40.846 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:40 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:40.560+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:15.765726+0000 front 2024-03-19T03:43:15.766177+0000 (oldest deadline 2024-03-19T03:43:38.065462+0000) 2024-03-19T03:43:40.943 INFO:journalctl@ceph.osd.3.smithi117.stdout:Mar 19 03:43:40 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-3[53172]: 2024-03-19T03:43:40.652+0000 7fca9de00700 -1 osd.3 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.394766+0000 front 2024-03-19T03:43:14.394285+0000 (oldest deadline 2024-03-19T03:43:38.494231+0000) 2024-03-19T03:43:41.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:40 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:40.942+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:43:41.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:41 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:41.273+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:43:41.595 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:41 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:41.228+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:43:41.914 INFO:journalctl@ceph.osd.3.smithi117.stdout:Mar 19 03:43:41 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-3[53172]: 2024-03-19T03:43:41.605+0000 7fca9de00700 -1 osd.3 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.394766+0000 front 2024-03-19T03:43:14.394285+0000 (oldest deadline 2024-03-19T03:43:38.494231+0000) 2024-03-19T03:43:42.095 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:41 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:41.602+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:15.765726+0000 front 2024-03-19T03:43:15.766177+0000 (oldest deadline 2024-03-19T03:43:38.065462+0000) 2024-03-19T03:43:42.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:41 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:41.913+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:43:42.574 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:42 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:42.210+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:43:42.734 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:42 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:42.290+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:43:42.845 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:42 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:42.574+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:15.765726+0000 front 2024-03-19T03:43:15.766177+0000 (oldest deadline 2024-03-19T03:43:38.065462+0000) 2024-03-19T03:43:43.270 INFO:journalctl@ceph.osd.3.smithi117.stdout:Mar 19 03:43:43 smithi117 podman[92993]: 2024-03-19 03:43:43.014564356 +0000 UTC m=+5.743319232 container died d52cba30da4a2638083ce376af408645b87ce379a7648939bf5666133d0f9a1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-3, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=) 2024-03-19T03:43:43.270 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:42 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:42.865+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:43:43.541 INFO:journalctl@ceph.osd.3.smithi117.stdout:Mar 19 03:43:43 smithi117 podman[92993]: 2024-03-19 03:43:43.478886835 +0000 UTC m=+6.207641675 container cleanup d52cba30da4a2638083ce376af408645b87ce379a7648939bf5666133d0f9a1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-3, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-19T03:43:43.542 INFO:journalctl@ceph.osd.3.smithi117.stdout:Mar 19 03:43:43 smithi117 bash[92993]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-3 2024-03-19T03:43:43.554 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:43 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:43.228+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:43:43.734 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:43 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:43.321+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:43:43.845 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:43 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:43.552+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:15.765726+0000 front 2024-03-19T03:43:15.766177+0000 (oldest deadline 2024-03-19T03:43:38.065462+0000) 2024-03-19T03:43:43.880 INFO:journalctl@ceph.osd.3.smithi117.stdout:Mar 19 03:43:43 smithi117 podman[93012]: 2024-03-19 03:43:43.705062761 +0000 UTC m=+0.676053057 container remove d52cba30da4a2638083ce376af408645b87ce379a7648939bf5666133d0f9a1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-3, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-19T03:43:44.291 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:43 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:43.880+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:43:44.545 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:44 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:44.208+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:43:44.734 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:44 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:44.293+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:43:44.760 INFO:journalctl@ceph.osd.3.smithi117.stdout:Mar 19 03:43:44 smithi117 podman[93108]: 2024-03-19 03:43:44.351086656 +0000 UTC m=+0.033637202 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:43:44.760 INFO:journalctl@ceph.osd.3.smithi117.stdout:Mar 19 03:43:44 smithi117 podman[93108]: 2024-03-19 03:43:44.47268095 +0000 UTC m=+0.155231531 container create 7fcf107ae552345f3d9c1f9594cb2a5689450e35233b232677ec3eafb287c3be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD) 2024-03-19T03:43:44.760 INFO:journalctl@ceph.osd.3.smithi117.stdout:Mar 19 03:43:44 smithi117 podman[93108]: 2024-03-19 03:43:44.759654379 +0000 UTC m=+0.442205003 container init 7fcf107ae552345f3d9c1f9594cb2a5689450e35233b232677ec3eafb287c3be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:43:44.845 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:44 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:44.546+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:15.765726+0000 front 2024-03-19T03:43:15.766177+0000 (oldest deadline 2024-03-19T03:43:38.065462+0000) 2024-03-19T03:43:45.042 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:44 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:44.881+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:43:45.042 INFO:journalctl@ceph.osd.3.smithi117.stdout:Mar 19 03:43:44 smithi117 podman[93108]: 2024-03-19 03:43:44.76918051 +0000 UTC m=+0.451731069 container start 7fcf107ae552345f3d9c1f9594cb2a5689450e35233b232677ec3eafb287c3be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-19T03:43:45.042 INFO:journalctl@ceph.osd.3.smithi117.stdout:Mar 19 03:43:44 smithi117 podman[93108]: 2024-03-19 03:43:44.883902262 +0000 UTC m=+0.566452890 container attach 7fcf107ae552345f3d9c1f9594cb2a5689450e35233b232677ec3eafb287c3be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-19T03:43:45.509 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:45 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:45.181+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:43:45.542 INFO:journalctl@ceph.osd.3.smithi117.stdout:Mar 19 03:43:45 smithi117 podman[93108]: 2024-03-19 03:43:45.19348191 +0000 UTC m=+0.876032524 container died 7fcf107ae552345f3d9c1f9594cb2a5689450e35233b232677ec3eafb287c3be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-19T03:43:45.734 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:45 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:45.324+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:43:45.845 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:45 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:45.508+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:15.765726+0000 front 2024-03-19T03:43:15.766177+0000 (oldest deadline 2024-03-19T03:43:38.065462+0000) 2024-03-19T03:43:45.973 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.3.service' 2024-03-19T03:43:46.095 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:43:46.095 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-19T03:43:46.095 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-19T03:43:46.096 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.4 2024-03-19T03:43:46.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:45 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:45.871+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:43:46.408 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:46 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:46.318+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:43:46.409 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 19 03:43:46 smithi119 systemd[1]: Stopping Ceph osd.4 for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:43:46.554 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:46 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:46.134+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:43:46.734 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 19 03:43:46 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-4[78276]: 2024-03-19T03:43:46.407+0000 7f9b6e0b2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:43:46.734 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 19 03:43:46 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-4[78276]: 2024-03-19T03:43:46.407+0000 7f9b6e0b2700 -1 osd.4 97 *** Got signal Terminated *** 2024-03-19T03:43:46.734 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 19 03:43:46 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-4[78276]: 2024-03-19T03:43:46.407+0000 7f9b6e0b2700 -1 osd.4 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T03:43:46.845 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:46 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:46.553+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:15.765726+0000 front 2024-03-19T03:43:15.766177+0000 (oldest deadline 2024-03-19T03:43:38.065462+0000) 2024-03-19T03:43:47.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:46 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:46.842+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:43:47.345 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:47 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:47.135+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:43:47.734 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 19 03:43:47 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-4[78276]: 2024-03-19T03:43:47.507+0000 7f9b6a6a6700 -1 osd.4 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.966325+0000 front 2024-03-19T03:43:23.966350+0000 (oldest deadline 2024-03-19T03:43:47.367291+0000) 2024-03-19T03:43:47.734 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:47 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:47.298+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:43:47.846 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:47 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:47.586+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:15.765726+0000 front 2024-03-19T03:43:15.766177+0000 (oldest deadline 2024-03-19T03:43:38.065462+0000) 2024-03-19T03:43:48.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:47 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:47.855+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:43:48.345 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:48 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:48.093+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:43:48.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:43:48 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:43:48.045+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:43:48.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:48 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:48.251+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:43:48.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:48 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:48.251+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:43:48.984 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 19 03:43:48 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-4[78276]: 2024-03-19T03:43:48.544+0000 7f9b6a6a6700 -1 osd.4 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.966325+0000 front 2024-03-19T03:43:23.966350+0000 (oldest deadline 2024-03-19T03:43:47.367291+0000) 2024-03-19T03:43:49.095 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:48 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:48.617+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:15.765726+0000 front 2024-03-19T03:43:15.766177+0000 (oldest deadline 2024-03-19T03:43:38.065462+0000) 2024-03-19T03:43:49.264 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:48 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:48.868+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:43:49.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:43:49 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:43:49.084+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:43:49.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:49 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:49.252+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:43:49.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:49 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:49.252+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:43:49.509 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:49 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:49.098+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:43:49.542 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:43:49 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:43:49.263+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.245728+0000 front 2024-03-19T03:43:26.245677+0000 (oldest deadline 2024-03-19T03:43:48.545497+0000) 2024-03-19T03:43:49.845 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:49 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:49.572+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:15.765726+0000 front 2024-03-19T03:43:15.766177+0000 (oldest deadline 2024-03-19T03:43:38.065462+0000) 2024-03-19T03:43:49.846 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:43:49 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:43:49.509+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:43:49.984 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 19 03:43:49 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-4[78276]: 2024-03-19T03:43:49.543+0000 7f9b6a6a6700 -1 osd.4 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.966325+0000 front 2024-03-19T03:43:23.966350+0000 (oldest deadline 2024-03-19T03:43:47.367291+0000) 2024-03-19T03:43:50.233 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:49 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:49.835+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:43:50.233 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:49 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:49.835+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:43:50.345 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:50 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:50.063+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:43:50.345 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:50 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:50.063+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:43:50.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:43:50 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:43:50.044+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:43:50.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:50 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:50.211+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:43:50.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:50 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:50.211+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:43:50.542 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:43:50 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:43:50.232+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.245728+0000 front 2024-03-19T03:43:26.245677+0000 (oldest deadline 2024-03-19T03:43:48.545497+0000) 2024-03-19T03:43:50.845 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:50 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:50.551+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:15.765726+0000 front 2024-03-19T03:43:15.766177+0000 (oldest deadline 2024-03-19T03:43:38.065462+0000) 2024-03-19T03:43:50.846 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:43:50 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:43:50.472+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:43:50.984 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 19 03:43:50 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-4[78276]: 2024-03-19T03:43:50.510+0000 7f9b6a6a6700 -1 osd.4 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.966325+0000 front 2024-03-19T03:43:23.966350+0000 (oldest deadline 2024-03-19T03:43:47.367291+0000) 2024-03-19T03:43:51.237 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:50 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:50.813+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:43:51.237 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:50 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:50.813+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:43:51.345 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:51 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:51.067+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:43:51.346 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:51 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:51.067+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:43:51.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:43:51 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:43:51.009+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:43:51.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:51 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:51.166+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:43:51.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:51 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:51.166+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:43:51.542 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:43:51 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:43:51.235+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.245728+0000 front 2024-03-19T03:43:26.245677+0000 (oldest deadline 2024-03-19T03:43:48.545497+0000) 2024-03-19T03:43:51.845 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:43:51 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:43:51.431+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:43:51.846 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:51 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:51.517+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:15.765726+0000 front 2024-03-19T03:43:15.766177+0000 (oldest deadline 2024-03-19T03:43:38.065462+0000) 2024-03-19T03:43:51.846 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:51 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:51.517+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.266995+0000 front 2024-03-19T03:43:26.266864+0000 (oldest deadline 2024-03-19T03:43:50.966755+0000) 2024-03-19T03:43:52.168 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:43:51 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:43:51.996+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:43:52.168 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 19 03:43:51 smithi119 podman[101291]: 2024-03-19 03:43:51.85593439 +0000 UTC m=+5.556627156 container died 6794d9b7a9b49527eea5756e7048d4023c738a2c0cc63bcb69da0ae6b1357e7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-4, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-19T03:43:52.232 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:43:52 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:43:52.231+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.245728+0000 front 2024-03-19T03:43:26.245677+0000 (oldest deadline 2024-03-19T03:43:48.545497+0000) 2024-03-19T03:43:52.232 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:51 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:51.861+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:43:52.232 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:51 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:51.861+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:43:52.345 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:52 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:52.092+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:43:52.345 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:52 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:52.092+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:43:52.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:52 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:52.206+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:43:52.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:52 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:52.206+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:43:52.484 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 19 03:43:52 smithi119 podman[101291]: 2024-03-19 03:43:52.2851534 +0000 UTC m=+5.985846158 container cleanup 6794d9b7a9b49527eea5756e7048d4023c738a2c0cc63bcb69da0ae6b1357e7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-4, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-19T03:43:52.484 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 19 03:43:52 smithi119 bash[101291]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-4 2024-03-19T03:43:52.845 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:43:52 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:43:52.387+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:43:52.846 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:52 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:52.511+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:15.765726+0000 front 2024-03-19T03:43:15.766177+0000 (oldest deadline 2024-03-19T03:43:38.065462+0000) 2024-03-19T03:43:52.846 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:52 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:52.511+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.266995+0000 front 2024-03-19T03:43:26.266864+0000 (oldest deadline 2024-03-19T03:43:50.966755+0000) 2024-03-19T03:43:52.947 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 19 03:43:52 smithi119 podman[101309]: 2024-03-19 03:43:52.545026667 +0000 UTC m=+0.678153336 container remove 6794d9b7a9b49527eea5756e7048d4023c738a2c0cc63bcb69da0ae6b1357e7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T03:43:53.198 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:43:53 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:43:53.018+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:43:53.198 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 19 03:43:52 smithi119 podman[101402]: 2024-03-19 03:43:52.850669895 +0000 UTC m=+0.017667252 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:43:53.198 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 19 03:43:52 smithi119 podman[101402]: 2024-03-19 03:43:52.970855016 +0000 UTC m=+0.137852351 container create 5d29c2e8fb58ac448e479aba3ac58b7433b9419be084cf0a93411fe25f17ffc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-19T03:43:53.237 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:52 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:52.892+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:43:53.237 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:52 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:52.892+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:43:53.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:53 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:53.197+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:43:53.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:53 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:53.197+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:43:53.484 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 19 03:43:53 smithi119 podman[101402]: 2024-03-19 03:43:53.249295821 +0000 UTC m=+0.416293178 container init 5d29c2e8fb58ac448e479aba3ac58b7433b9419be084cf0a93411fe25f17ffc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T03:43:53.484 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 19 03:43:53 smithi119 podman[101402]: 2024-03-19 03:43:53.252377677 +0000 UTC m=+0.419375021 container start 5d29c2e8fb58ac448e479aba3ac58b7433b9419be084cf0a93411fe25f17ffc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-19T03:43:53.484 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 19 03:43:53 smithi119 podman[101402]: 2024-03-19 03:43:53.349377202 +0000 UTC m=+0.516374555 container attach 5d29c2e8fb58ac448e479aba3ac58b7433b9419be084cf0a93411fe25f17ffc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-19T03:43:53.507 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:43:53 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:43:53.370+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:43:53.507 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:53 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:53.121+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:43:53.507 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:53 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:53.121+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:43:53.542 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:43:53 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:43:53.236+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.245728+0000 front 2024-03-19T03:43:26.245677+0000 (oldest deadline 2024-03-19T03:43:48.545497+0000) 2024-03-19T03:43:53.810 INFO:journalctl@ceph.osd.4.smithi119.stdout:Mar 19 03:43:53 smithi119 podman[101402]: 2024-03-19 03:43:53.547233096 +0000 UTC m=+0.714230441 container died 5d29c2e8fb58ac448e479aba3ac58b7433b9419be084cf0a93411fe25f17ffc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T03:43:53.845 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:53 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:53.506+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:15.765726+0000 front 2024-03-19T03:43:15.766177+0000 (oldest deadline 2024-03-19T03:43:38.065462+0000) 2024-03-19T03:43:53.845 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:53 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:53.506+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.266995+0000 front 2024-03-19T03:43:26.266864+0000 (oldest deadline 2024-03-19T03:43:50.966755+0000) 2024-03-19T03:43:54.078 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:43:54 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:43:54.002+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:43:54.105 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.4.service' 2024-03-19T03:43:54.147 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:43:54.147 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-19T03:43:54.147 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-19T03:43:54.147 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.5 2024-03-19T03:43:54.196 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:53 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:53.910+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:43:54.196 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:53 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:53.910+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:43:54.439 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:54 smithi178 systemd[1]: Stopping Ceph osd.5 for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:43:54.440 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:43:54 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:43:54.334+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:43:54.440 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:54 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:54.171+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:43:54.440 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:54 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:54.171+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:43:54.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:54 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:54.225+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:43:54.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:54 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:54.225+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:43:54.542 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:43:54 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:43:54.195+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.245728+0000 front 2024-03-19T03:43:26.245677+0000 (oldest deadline 2024-03-19T03:43:48.545497+0000) 2024-03-19T03:43:54.846 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:54 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:54.438+0000 7fb074086700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:43:54.846 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:54 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:54.438+0000 7fb074086700 -1 osd.5 97 *** Got signal Terminated *** 2024-03-19T03:43:54.846 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:54 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:54.438+0000 7fb074086700 -1 osd.5 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T03:43:54.846 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:54 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:54.549+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:15.765726+0000 front 2024-03-19T03:43:15.766177+0000 (oldest deadline 2024-03-19T03:43:38.065462+0000) 2024-03-19T03:43:54.846 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:54 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:54.549+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.266995+0000 front 2024-03-19T03:43:26.266864+0000 (oldest deadline 2024-03-19T03:43:50.966755+0000) 2024-03-19T03:43:55.292 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:43:55 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:43:55.160+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.245728+0000 front 2024-03-19T03:43:26.245677+0000 (oldest deadline 2024-03-19T03:43:48.545497+0000) 2024-03-19T03:43:55.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:54 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:54.954+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:43:55.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:54 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:54.954+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:43:55.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:43:55 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:43:55.038+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:43:55.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:55 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:55.199+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:43:55.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:55 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:55.199+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:43:55.518 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:43:55 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:43:55.305+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:43:55.518 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:55 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:55.192+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:43:55.518 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:55 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:55.192+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:43:55.845 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:55 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:55.517+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:15.765726+0000 front 2024-03-19T03:43:15.766177+0000 (oldest deadline 2024-03-19T03:43:38.065462+0000) 2024-03-19T03:43:55.846 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:55 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:55.517+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.266995+0000 front 2024-03-19T03:43:26.266864+0000 (oldest deadline 2024-03-19T03:43:50.966755+0000) 2024-03-19T03:43:55.846 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:55 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:55.517+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:33.168097+0000 front 2024-03-19T03:43:33.168182+0000 (oldest deadline 2024-03-19T03:43:54.868266+0000) 2024-03-19T03:43:56.292 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:43:56 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:43:56.184+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.245728+0000 front 2024-03-19T03:43:26.245677+0000 (oldest deadline 2024-03-19T03:43:48.545497+0000) 2024-03-19T03:43:56.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:55 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:55.936+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:43:56.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:55 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:55.936+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:43:56.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:43:56 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:43:56.059+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:43:56.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:56 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:56.176+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:43:56.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:56 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:56.176+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:43:56.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:56 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:56.176+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:43:56.516 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:56 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:56.211+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:43:56.516 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:56 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:56.211+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:43:56.517 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:56 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:56.211+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:43:56.517 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:43:56 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:43:56.306+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:43:56.517 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:43:56 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:43:56.306+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:43:56.846 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:56 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:56.515+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:15.765726+0000 front 2024-03-19T03:43:15.766177+0000 (oldest deadline 2024-03-19T03:43:38.065462+0000) 2024-03-19T03:43:56.846 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:56 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:56.515+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.266995+0000 front 2024-03-19T03:43:26.266864+0000 (oldest deadline 2024-03-19T03:43:50.966755+0000) 2024-03-19T03:43:56.846 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:56 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:56.515+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:33.168097+0000 front 2024-03-19T03:43:33.168182+0000 (oldest deadline 2024-03-19T03:43:54.868266+0000) 2024-03-19T03:43:57.215 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:56 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:56.952+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:43:57.215 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:56 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:56.952+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:43:57.215 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:56 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:56.952+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:43:57.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:43:57 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:43:57.036+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:43:57.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:57 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:57.141+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:43:57.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:57 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:57.141+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:43:57.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:57 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:57.141+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:43:57.484 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:43:57 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:43:57.331+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:43:57.485 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:43:57 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:43:57.331+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:43:57.485 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:57 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:57.176+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:43:57.485 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:57 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:57.176+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:43:57.485 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:57 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:57.176+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:43:57.542 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:43:57 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:43:57.214+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.245728+0000 front 2024-03-19T03:43:26.245677+0000 (oldest deadline 2024-03-19T03:43:48.545497+0000) 2024-03-19T03:43:57.845 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:57 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:57.484+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:15.765726+0000 front 2024-03-19T03:43:15.766177+0000 (oldest deadline 2024-03-19T03:43:38.065462+0000) 2024-03-19T03:43:57.846 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:57 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:57.484+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.266995+0000 front 2024-03-19T03:43:26.266864+0000 (oldest deadline 2024-03-19T03:43:50.966755+0000) 2024-03-19T03:43:57.846 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:57 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:57.484+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:33.168097+0000 front 2024-03-19T03:43:33.168182+0000 (oldest deadline 2024-03-19T03:43:54.868266+0000) 2024-03-19T03:43:58.182 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:57 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:57.930+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:43:58.182 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:57 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:57.930+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:43:58.183 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:57 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:57.930+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:43:58.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:43:58 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:43:58.076+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:43:58.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:58 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:58.169+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:43:58.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:58 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:58.169+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:43:58.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:58 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:58.169+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:43:58.510 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:43:58 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:43:58.338+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:43:58.510 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:43:58 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:43:58.338+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:43:58.510 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:58 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:58.206+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:43:58.510 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:58 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:58.206+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:43:58.510 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:58 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:58.206+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:43:58.541 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:43:58 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:43:58.181+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.245728+0000 front 2024-03-19T03:43:26.245677+0000 (oldest deadline 2024-03-19T03:43:48.545497+0000) 2024-03-19T03:43:58.846 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:58 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:58.508+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:15.765726+0000 front 2024-03-19T03:43:15.766177+0000 (oldest deadline 2024-03-19T03:43:38.065462+0000) 2024-03-19T03:43:58.846 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:58 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:58.508+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.266995+0000 front 2024-03-19T03:43:26.266864+0000 (oldest deadline 2024-03-19T03:43:50.966755+0000) 2024-03-19T03:43:58.846 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:58 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5[79264]: 2024-03-19T03:43:58.508+0000 7fb07067a700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:33.168097+0000 front 2024-03-19T03:43:33.168182+0000 (oldest deadline 2024-03-19T03:43:54.868266+0000) 2024-03-19T03:43:59.292 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:43:59 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:43:59.153+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.245728+0000 front 2024-03-19T03:43:26.245677+0000 (oldest deadline 2024-03-19T03:43:48.545497+0000) 2024-03-19T03:43:59.292 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:43:59 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:43:59.153+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.348656+0000 front 2024-03-19T03:43:34.348769+0000 (oldest deadline 2024-03-19T03:43:58.448904+0000) 2024-03-19T03:43:59.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:58 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:58.943+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:43:59.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:58 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:58.943+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:43:59.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:58 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:58.943+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:43:59.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:43:59 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:43:59.079+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:43:59.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:43:59 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:43:59.079+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.311998+0000 front 2024-03-19T03:43:34.312155+0000 (oldest deadline 2024-03-19T03:43:58.411836+0000) 2024-03-19T03:43:59.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:59 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:59.193+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:43:59.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:59 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:59.193+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:43:59.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:43:59 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:43:59.193+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:43:59.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:43:59 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:43:59.318+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:43:59.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:43:59 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:43:59.318+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:43:59.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:59 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:59.197+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:43:59.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:59 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:59.197+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:43:59.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:43:59 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:43:59.197+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:00.281 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:00 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:00.234+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:00.281 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:00 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:00.234+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:00.281 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:00 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:00.234+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:00.282 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:43:59 smithi178 podman[103691]: 2024-03-19 03:43:59.99705209 +0000 UTC m=+5.652067690 container died ece9b29409b4dc69b7da986b04588fde865a0d86c5414e8ff19d34b24d107652 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T03:44:00.292 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:00 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:44:00.178+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.245728+0000 front 2024-03-19T03:43:26.245677+0000 (oldest deadline 2024-03-19T03:43:48.545497+0000) 2024-03-19T03:44:00.292 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:00 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:44:00.178+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.348656+0000 front 2024-03-19T03:43:34.348769+0000 (oldest deadline 2024-03-19T03:43:58.448904+0000) 2024-03-19T03:44:00.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:59 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:59.991+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:00.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:59 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:59.991+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:00.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:43:59 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:43:59.991+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:00.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:00 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:00.085+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:44:00.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:00 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:00.085+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.311998+0000 front 2024-03-19T03:43:34.312155+0000 (oldest deadline 2024-03-19T03:43:58.411836+0000) 2024-03-19T03:44:00.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:00 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:00.157+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:00.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:00 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:00.157+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:00.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:00 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:00.157+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:00.595 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:00 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:00.318+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:44:00.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:00 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:00.318+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:44:00.596 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:44:00 smithi178 podman[103691]: 2024-03-19 03:44:00.281158166 +0000 UTC m=+5.936173755 container cleanup ece9b29409b4dc69b7da986b04588fde865a0d86c5414e8ff19d34b24d107652 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T03:44:00.596 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:44:00 smithi178 bash[103691]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5 2024-03-19T03:44:00.596 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:44:00 smithi178 podman[103708]: 2024-03-19 03:44:00.524433325 +0000 UTC m=+0.528391566 container remove ece9b29409b4dc69b7da986b04588fde865a0d86c5414e8ff19d34b24d107652 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T03:44:01.095 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:44:00 smithi178 podman[103796]: 2024-03-19 03:44:00.745938289 +0000 UTC m=+0.017422523 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:44:01.095 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:44:00 smithi178 podman[103796]: 2024-03-19 03:44:00.867393282 +0000 UTC m=+0.138877542 container create 4aa31c0f38f8e377e5bdf14f75e4926d5a49ad6b206421a52c1ccdc6ed24bd54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-19T03:44:01.292 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:01 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:44:01.163+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.245728+0000 front 2024-03-19T03:43:26.245677+0000 (oldest deadline 2024-03-19T03:43:48.545497+0000) 2024-03-19T03:44:01.292 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:01 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:44:01.163+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.348656+0000 front 2024-03-19T03:43:34.348769+0000 (oldest deadline 2024-03-19T03:43:58.448904+0000) 2024-03-19T03:44:01.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:01 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:01.025+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:01.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:01 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:01.025+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:01.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:01 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:01.025+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:01.346 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:44:01 smithi178 podman[103796]: 2024-03-19 03:44:01.095337512 +0000 UTC m=+0.366821742 container init 4aa31c0f38f8e377e5bdf14f75e4926d5a49ad6b206421a52c1ccdc6ed24bd54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-19T03:44:01.346 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:44:01 smithi178 podman[103796]: 2024-03-19 03:44:01.098815974 +0000 UTC m=+0.370300190 container start 4aa31c0f38f8e377e5bdf14f75e4926d5a49ad6b206421a52c1ccdc6ed24bd54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-19T03:44:01.346 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:44:01 smithi178 podman[103796]: 2024-03-19 03:44:01.161805865 +0000 UTC m=+0.433290091 container attach 4aa31c0f38f8e377e5bdf14f75e4926d5a49ad6b206421a52c1ccdc6ed24bd54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-19T03:44:01.346 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:01 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:01.271+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:44:01.346 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:01 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:01.271+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:44:01.347 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:01 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:01.241+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:01.347 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:01 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:01.241+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:01.347 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:01 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:01.241+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:01.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:01 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:01.080+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:44:01.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:01 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:01.080+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.311998+0000 front 2024-03-19T03:43:34.312155+0000 (oldest deadline 2024-03-19T03:43:58.411836+0000) 2024-03-19T03:44:01.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:01 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:01.182+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:01.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:01 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:01.182+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:01.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:01 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:01.182+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:01.706 INFO:journalctl@ceph.osd.5.smithi178.stdout:Mar 19 03:44:01 smithi178 podman[103796]: 2024-03-19 03:44:01.399643641 +0000 UTC m=+0.671127862 container died 4aa31c0f38f8e377e5bdf14f75e4926d5a49ad6b206421a52c1ccdc6ed24bd54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T03:44:02.035 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.5.service' 2024-03-19T03:44:02.077 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:44:02.077 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-19T03:44:02.077 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-19T03:44:02.077 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.6 2024-03-19T03:44:02.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:01 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:01.987+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:02.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:01 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:01.987+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:02.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:01 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:01.987+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:02.292 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:02 smithi117 systemd[1]: Stopping Ceph osd.6 for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:44:02.292 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:02 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:44:02.209+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.245728+0000 front 2024-03-19T03:43:26.245677+0000 (oldest deadline 2024-03-19T03:43:48.545497+0000) 2024-03-19T03:44:02.292 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:02 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:44:02.209+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.348656+0000 front 2024-03-19T03:43:34.348769+0000 (oldest deadline 2024-03-19T03:43:58.448904+0000) 2024-03-19T03:44:02.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:02 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:02.059+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:44:02.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:02 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:02.059+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.311998+0000 front 2024-03-19T03:43:34.312155+0000 (oldest deadline 2024-03-19T03:43:58.411836+0000) 2024-03-19T03:44:02.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:02 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:02.179+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:02.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:02 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:02.179+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:02.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:02 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:02.179+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:02.595 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:02 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:02.283+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:44:02.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:02 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:02.283+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:44:02.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:02 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:02.276+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:02.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:02 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:02.276+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:02.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:02 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:02.276+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:02.982 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:02 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:44:02.700+0000 7f36cf200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:44:02.982 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:02 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:44:02.700+0000 7f36cf200700 -1 osd.6 97 *** Got signal Terminated *** 2024-03-19T03:44:02.982 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:02 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:44:02.700+0000 7f36cf200700 -1 osd.6 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T03:44:03.292 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:03 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:44:03.198+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.245728+0000 front 2024-03-19T03:43:26.245677+0000 (oldest deadline 2024-03-19T03:43:48.545497+0000) 2024-03-19T03:44:03.292 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:03 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:44:03.198+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.348656+0000 front 2024-03-19T03:43:34.348769+0000 (oldest deadline 2024-03-19T03:43:58.448904+0000) 2024-03-19T03:44:03.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:02 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:02.981+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:03.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:02 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:02.981+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:03.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:02 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:02.981+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:03.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:03 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:03.023+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:44:03.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:03 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:03.023+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.311998+0000 front 2024-03-19T03:43:34.312155+0000 (oldest deadline 2024-03-19T03:43:58.411836+0000) 2024-03-19T03:44:03.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:03 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:03.207+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:03.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:03 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:03.207+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:03.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:03 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:03.207+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:03.595 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:03 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:03.323+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:44:03.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:03 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:03.323+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:44:03.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:03 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:03.235+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:03.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:03 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:03.235+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:03.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:03 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:03.235+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:04.292 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:04 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:44:04.240+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.245728+0000 front 2024-03-19T03:43:26.245677+0000 (oldest deadline 2024-03-19T03:43:48.545497+0000) 2024-03-19T03:44:04.292 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:04 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:44:04.240+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.348656+0000 front 2024-03-19T03:43:34.348769+0000 (oldest deadline 2024-03-19T03:43:58.448904+0000) 2024-03-19T03:44:04.292 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:04 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:44:04.240+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.852994+0000 front 2024-03-19T03:43:41.852677+0000 (oldest deadline 2024-03-19T03:44:04.152366+0000) 2024-03-19T03:44:04.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:03 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:03.993+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:04.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:03 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:03.993+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:04.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:03 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:03.993+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:04.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:04 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:04.002+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:44:04.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:04 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:04.002+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.311998+0000 front 2024-03-19T03:43:34.312155+0000 (oldest deadline 2024-03-19T03:43:58.411836+0000) 2024-03-19T03:44:04.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:04 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:04.002+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:38.412580+0000 front 2024-03-19T03:43:38.412664+0000 (oldest deadline 2024-03-19T03:44:03.712408+0000) 2024-03-19T03:44:04.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:04 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:04.215+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:04.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:04 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:04.215+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:04.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:04 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:04.215+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:04.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:04 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:04.284+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:44:04.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:04 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:04.284+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:44:04.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:04 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:04.284+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851981+0000 front 2024-03-19T03:43:41.852468+0000 (oldest deadline 2024-03-19T03:44:03.551908+0000) 2024-03-19T03:44:04.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:04 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:04.256+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:04.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:04 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:04.256+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:04.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:04 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:04.256+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:05.234 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:04 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:04.957+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:44:05.234 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:04 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:04.957+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.311998+0000 front 2024-03-19T03:43:34.312155+0000 (oldest deadline 2024-03-19T03:43:58.411836+0000) 2024-03-19T03:44:05.234 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:04 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:04.957+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:38.412580+0000 front 2024-03-19T03:43:38.412664+0000 (oldest deadline 2024-03-19T03:44:03.712408+0000) 2024-03-19T03:44:05.285 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:05 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:05.022+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:05.285 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:05 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:05.022+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:05.285 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:05 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:05.022+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:05.542 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:05 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:44:05.283+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.245728+0000 front 2024-03-19T03:43:26.245677+0000 (oldest deadline 2024-03-19T03:43:48.545497+0000) 2024-03-19T03:44:05.542 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:05 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:44:05.283+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.348656+0000 front 2024-03-19T03:43:34.348769+0000 (oldest deadline 2024-03-19T03:43:58.448904+0000) 2024-03-19T03:44:05.542 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:05 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:44:05.283+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.852994+0000 front 2024-03-19T03:43:41.852677+0000 (oldest deadline 2024-03-19T03:44:04.152366+0000) 2024-03-19T03:44:05.595 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:05 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:05.304+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:44:05.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:05 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:05.304+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:44:05.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:05 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:05.304+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851981+0000 front 2024-03-19T03:43:41.852468+0000 (oldest deadline 2024-03-19T03:44:03.551908+0000) 2024-03-19T03:44:05.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:05 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:05.289+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:05.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:05 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:05.289+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:05.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:05 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:05.289+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:05.734 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:05 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:05.251+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:05.734 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:05 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:05.251+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:05.734 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:05 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:05.251+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:06.222 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:05 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:05.966+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:44:06.222 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:05 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:05.966+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.311998+0000 front 2024-03-19T03:43:34.312155+0000 (oldest deadline 2024-03-19T03:43:58.411836+0000) 2024-03-19T03:44:06.222 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:05 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:05.966+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:38.412580+0000 front 2024-03-19T03:43:38.412664+0000 (oldest deadline 2024-03-19T03:44:03.712408+0000) 2024-03-19T03:44:06.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:06 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:06.010+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:06.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:06 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:06.010+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:06.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:06 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:06.010+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:06.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:06 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:06.221+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:06.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:06 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:06.221+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:06.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:06 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:06.221+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:06.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:06 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:06.221+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:06.595 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:06 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:06.342+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:44:06.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:06 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:06.342+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:44:06.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:06 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:06.342+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851981+0000 front 2024-03-19T03:43:41.852468+0000 (oldest deadline 2024-03-19T03:44:03.551908+0000) 2024-03-19T03:44:06.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:06 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:06.290+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:06.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:06 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:06.290+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:06.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:06 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:06.290+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:06.792 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:06 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:44:06.292+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.245728+0000 front 2024-03-19T03:43:26.245677+0000 (oldest deadline 2024-03-19T03:43:48.545497+0000) 2024-03-19T03:44:06.792 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:06 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:44:06.292+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.348656+0000 front 2024-03-19T03:43:34.348769+0000 (oldest deadline 2024-03-19T03:43:58.448904+0000) 2024-03-19T03:44:06.792 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:06 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:44:06.292+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.852994+0000 front 2024-03-19T03:43:41.852677+0000 (oldest deadline 2024-03-19T03:44:04.152366+0000) 2024-03-19T03:44:07.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:07 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:07.009+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:07.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:07 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:07.009+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:07.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:07 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:07.009+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:07.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:07 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:07.003+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:44:07.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:07 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:07.003+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.311998+0000 front 2024-03-19T03:43:34.312155+0000 (oldest deadline 2024-03-19T03:43:58.411836+0000) 2024-03-19T03:44:07.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:07 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:07.003+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:38.412580+0000 front 2024-03-19T03:43:38.412664+0000 (oldest deadline 2024-03-19T03:44:03.712408+0000) 2024-03-19T03:44:07.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:07 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:07.189+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:07.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:07 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:07.189+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:07.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:07 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:07.189+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:07.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:07 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:07.189+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:07.595 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:07 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:07.315+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:44:07.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:07 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:07.316+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:44:07.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:07 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:07.316+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851981+0000 front 2024-03-19T03:43:41.852468+0000 (oldest deadline 2024-03-19T03:44:03.551908+0000) 2024-03-19T03:44:07.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:07 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:07.243+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:07.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:07 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:07.243+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:07.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:07 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:07.243+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:07.792 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:07 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:44:07.341+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.245728+0000 front 2024-03-19T03:43:26.245677+0000 (oldest deadline 2024-03-19T03:43:48.545497+0000) 2024-03-19T03:44:07.792 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:07 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:44:07.341+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.348656+0000 front 2024-03-19T03:43:34.348769+0000 (oldest deadline 2024-03-19T03:43:58.448904+0000) 2024-03-19T03:44:07.792 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:07 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-19T03:44:07.341+0000 7f36c5a00700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.852994+0000 front 2024-03-19T03:43:41.852677+0000 (oldest deadline 2024-03-19T03:44:04.152366+0000) 2024-03-19T03:44:08.234 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:07 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:07.974+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:44:08.234 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:07 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:07.975+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.311998+0000 front 2024-03-19T03:43:34.312155+0000 (oldest deadline 2024-03-19T03:43:58.411836+0000) 2024-03-19T03:44:08.234 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:07 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:07.975+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:38.412580+0000 front 2024-03-19T03:43:38.412664+0000 (oldest deadline 2024-03-19T03:44:03.712408+0000) 2024-03-19T03:44:08.234 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:08 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:08.163+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:08.234 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:08 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:08.163+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:08.234 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:08 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:08.163+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:08.234 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:08 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:08.163+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:08.292 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:08 smithi117 podman[93214]: 2024-03-19 03:44:08.164779751 +0000 UTC m=+5.616873471 container died 8cf120542559391c07e174177a2aa26ca070f21b4feed9cd7eea74474a9dbe82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-19T03:44:08.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:08 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:08.008+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:08.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:08 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:08.008+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:08.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:08 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:08.008+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:08.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:08 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:08.008+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851051+0000 front 2024-03-19T03:43:41.850539+0000 (oldest deadline 2024-03-19T03:44:07.750871+0000) 2024-03-19T03:44:08.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:08 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:08.209+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:08.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:08 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:08.209+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:08.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:08 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:08.209+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:08.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:08 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:08.324+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:44:08.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:08 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:08.324+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:44:08.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:08 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:08.324+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851981+0000 front 2024-03-19T03:43:41.852468+0000 (oldest deadline 2024-03-19T03:44:03.551908+0000) 2024-03-19T03:44:08.792 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:08 smithi117 podman[93214]: 2024-03-19 03:44:08.54344089 +0000 UTC m=+5.995534659 container cleanup 8cf120542559391c07e174177a2aa26ca070f21b4feed9cd7eea74474a9dbe82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-19T03:44:08.792 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:08 smithi117 bash[93214]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6 2024-03-19T03:44:09.291 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:08 smithi117 podman[93234]: 2024-03-19 03:44:08.894676393 +0000 UTC m=+0.717738052 container remove 8cf120542559391c07e174177a2aa26ca070f21b4feed9cd7eea74474a9dbe82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-19T03:44:09.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:09 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:09.016+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:09.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:09 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:09.016+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:09.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:09 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:09.016+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:09.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:09 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:09.016+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851051+0000 front 2024-03-19T03:43:41.850539+0000 (oldest deadline 2024-03-19T03:44:07.750871+0000) 2024-03-19T03:44:09.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:08 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:08.983+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:44:09.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:08 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:08.983+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.311998+0000 front 2024-03-19T03:43:34.312155+0000 (oldest deadline 2024-03-19T03:43:58.411836+0000) 2024-03-19T03:44:09.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:08 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:08.983+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:38.412580+0000 front 2024-03-19T03:43:38.412664+0000 (oldest deadline 2024-03-19T03:44:03.712408+0000) 2024-03-19T03:44:09.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:09 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:09.120+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:09.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:09 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:09.120+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:09.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:09 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:09.120+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:09.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:09 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:09.120+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:09.595 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:09 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:09.296+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:44:09.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:09 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:09.296+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:44:09.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:09 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:09.296+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851981+0000 front 2024-03-19T03:43:41.852468+0000 (oldest deadline 2024-03-19T03:44:03.551908+0000) 2024-03-19T03:44:09.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:09 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:09.232+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:09.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:09 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:09.232+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:09.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:09 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:09.232+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:09.791 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:09 smithi117 podman[93320]: 2024-03-19 03:44:09.377763587 +0000 UTC m=+0.032929818 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:44:09.791 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:09 smithi117 podman[93320]: 2024-03-19 03:44:09.487029133 +0000 UTC m=+0.142195388 container create ae803188ff7ffd569fc0f59df0844dc026a84ff0bb1a1e34ad5b79bb494b1e98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-19T03:44:09.791 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:09 smithi117 podman[93320]: 2024-03-19 03:44:09.790546379 +0000 UTC m=+0.445712697 container init ae803188ff7ffd569fc0f59df0844dc026a84ff0bb1a1e34ad5b79bb494b1e98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-19T03:44:10.042 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:09 smithi117 podman[93320]: 2024-03-19 03:44:09.80195197 +0000 UTC m=+0.457118195 container start ae803188ff7ffd569fc0f59df0844dc026a84ff0bb1a1e34ad5b79bb494b1e98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T03:44:10.042 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:09 smithi117 podman[93320]: 2024-03-19 03:44:09.897817324 +0000 UTC m=+0.552983649 container attach ae803188ff7ffd569fc0f59df0844dc026a84ff0bb1a1e34ad5b79bb494b1e98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-19T03:44:10.042 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:09 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:09.981+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:10.042 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:09 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:09.981+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:10.043 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:09 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:09.981+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:10.043 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:09 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:09.981+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851051+0000 front 2024-03-19T03:43:41.850539+0000 (oldest deadline 2024-03-19T03:44:07.750871+0000) 2024-03-19T03:44:10.234 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:09 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:09.938+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:44:10.234 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:09 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:09.938+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.311998+0000 front 2024-03-19T03:43:34.312155+0000 (oldest deadline 2024-03-19T03:43:58.411836+0000) 2024-03-19T03:44:10.235 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:09 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:09.938+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:38.412580+0000 front 2024-03-19T03:43:38.412664+0000 (oldest deadline 2024-03-19T03:44:03.712408+0000) 2024-03-19T03:44:10.235 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:10 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:10.113+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:10.235 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:10 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:10.113+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:10.235 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:10 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:10.113+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:10.235 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:10 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:10.113+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:10.542 INFO:journalctl@ceph.osd.6.smithi117.stdout:Mar 19 03:44:10 smithi117 podman[93320]: 2024-03-19 03:44:10.229839138 +0000 UTC m=+0.885005365 container died ae803188ff7ffd569fc0f59df0844dc026a84ff0bb1a1e34ad5b79bb494b1e98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-19T03:44:10.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:10 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:10.321+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:44:10.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:10 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:10.321+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:44:10.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:10 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:10.321+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851981+0000 front 2024-03-19T03:43:41.852468+0000 (oldest deadline 2024-03-19T03:44:03.551908+0000) 2024-03-19T03:44:10.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:10 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:10.222+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:10.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:10 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:10.222+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:10.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:10 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:10.222+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:10.927 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.6.service' 2024-03-19T03:44:11.052 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:44:11.052 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-19T03:44:11.052 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-19T03:44:11.052 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.7 2024-03-19T03:44:11.234 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:10 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:10.935+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:44:11.234 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:10 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:10.935+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.311998+0000 front 2024-03-19T03:43:34.312155+0000 (oldest deadline 2024-03-19T03:43:58.411836+0000) 2024-03-19T03:44:11.234 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:10 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:10.935+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:38.412580+0000 front 2024-03-19T03:43:38.412664+0000 (oldest deadline 2024-03-19T03:44:03.712408+0000) 2024-03-19T03:44:11.234 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:11 smithi119 systemd[1]: Stopping Ceph osd.7 for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:44:11.234 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:11 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:11.085+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:11.235 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:11 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:11.085+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:11.235 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:11 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:11.085+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:11.235 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:11 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:11.085+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:11.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:10 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:10.997+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:11.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:10 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:10.997+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:11.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:10 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:10.997+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:11.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:10 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:10.997+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851051+0000 front 2024-03-19T03:43:41.850539+0000 (oldest deadline 2024-03-19T03:44:07.750871+0000) 2024-03-19T03:44:11.595 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:11 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:11.299+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:44:11.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:11 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:11.299+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:44:11.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:11 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:11.299+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851981+0000 front 2024-03-19T03:43:41.852468+0000 (oldest deadline 2024-03-19T03:44:03.551908+0000) 2024-03-19T03:44:11.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:11 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:11.175+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:11.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:11 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:11.175+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:11.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:11 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:11.175+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:11.734 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:11 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:11.389+0000 7fc5c558e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:44:11.734 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:11 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:11.389+0000 7fc5c558e700 -1 osd.7 97 *** Got signal Terminated *** 2024-03-19T03:44:11.734 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:11 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:11.389+0000 7fc5c558e700 -1 osd.7 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T03:44:12.234 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:11 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:11.922+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:44:12.234 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:11 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:11.922+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.311998+0000 front 2024-03-19T03:43:34.312155+0000 (oldest deadline 2024-03-19T03:43:58.411836+0000) 2024-03-19T03:44:12.234 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:11 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:11.922+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:38.412580+0000 front 2024-03-19T03:43:38.412664+0000 (oldest deadline 2024-03-19T03:44:03.712408+0000) 2024-03-19T03:44:12.234 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:12 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:12.111+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:12.234 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:12 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:12.111+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:12.235 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:12 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:12.111+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:12.235 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:12 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:12.111+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:12.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:12 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:12.016+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:12.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:12 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:12.016+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:12.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:12 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:12.016+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:12.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:12 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:12.016+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851051+0000 front 2024-03-19T03:43:41.850539+0000 (oldest deadline 2024-03-19T03:44:07.750871+0000) 2024-03-19T03:44:12.595 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:12 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:12.129+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:12.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:12 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:12.129+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:12.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:12 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:12.129+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:12.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:12 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:12.290+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:44:12.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:12 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:12.290+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:44:12.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:12 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:12.290+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851981+0000 front 2024-03-19T03:43:41.852468+0000 (oldest deadline 2024-03-19T03:44:03.551908+0000) 2024-03-19T03:44:13.234 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:12 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:12.891+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:44:13.234 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:12 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:12.891+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.311998+0000 front 2024-03-19T03:43:34.312155+0000 (oldest deadline 2024-03-19T03:43:58.411836+0000) 2024-03-19T03:44:13.234 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:12 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:12.891+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:38.412580+0000 front 2024-03-19T03:43:38.412664+0000 (oldest deadline 2024-03-19T03:44:03.712408+0000) 2024-03-19T03:44:13.234 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:13 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:13.076+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:13.234 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:13 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:13.076+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:13.234 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:13 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:13.076+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:13.235 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:13 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:13.076+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:13.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:13 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:13.043+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:13.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:13 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:13.043+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:13.543 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:13 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:13.043+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:13.543 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:13 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:13.043+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851051+0000 front 2024-03-19T03:43:41.850539+0000 (oldest deadline 2024-03-19T03:44:07.750871+0000) 2024-03-19T03:44:13.543 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:13 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:13.043+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.052823+0000 front 2024-03-19T03:43:50.052939+0000 (oldest deadline 2024-03-19T03:44:12.352894+0000) 2024-03-19T03:44:13.595 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:13 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:13.335+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:44:13.595 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:13 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:13.335+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:44:13.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:13 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:13.335+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851981+0000 front 2024-03-19T03:43:41.852468+0000 (oldest deadline 2024-03-19T03:44:03.551908+0000) 2024-03-19T03:44:13.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:13 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:13.136+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:13.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:13 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:13.136+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:13.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:13 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:13.136+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:14.234 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:13 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:13.869+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:44:14.234 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:13 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:13.869+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.311998+0000 front 2024-03-19T03:43:34.312155+0000 (oldest deadline 2024-03-19T03:43:58.411836+0000) 2024-03-19T03:44:14.234 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:13 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:13.869+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:38.412580+0000 front 2024-03-19T03:43:38.412664+0000 (oldest deadline 2024-03-19T03:44:03.712408+0000) 2024-03-19T03:44:14.234 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:14 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:14.085+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:14.234 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:14 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:14.085+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:14.234 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:14 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:14.085+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:14.234 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:14 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:14.085+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:14.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:14 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:14.042+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:14.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:14 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:14.042+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:14.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:14 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:14.042+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:14.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:14 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:14.042+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851051+0000 front 2024-03-19T03:43:41.850539+0000 (oldest deadline 2024-03-19T03:44:07.750871+0000) 2024-03-19T03:44:14.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:14 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:14.042+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.052823+0000 front 2024-03-19T03:43:50.052939+0000 (oldest deadline 2024-03-19T03:44:12.352894+0000) 2024-03-19T03:44:14.345 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:14 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:14.087+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:14.345 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:14 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:14.087+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:14.346 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:14 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:14.087+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:14.346 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:14 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:14.286+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:44:14.346 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:14 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:14.286+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:44:14.346 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:14 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:14.286+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851981+0000 front 2024-03-19T03:43:41.852468+0000 (oldest deadline 2024-03-19T03:44:03.551908+0000) 2024-03-19T03:44:15.113 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:14 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:14.843+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:44:15.113 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:14 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:14.843+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.311998+0000 front 2024-03-19T03:43:34.312155+0000 (oldest deadline 2024-03-19T03:43:58.411836+0000) 2024-03-19T03:44:15.113 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:14 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:14.843+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:38.412580+0000 front 2024-03-19T03:43:38.412664+0000 (oldest deadline 2024-03-19T03:44:03.712408+0000) 2024-03-19T03:44:15.113 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:14 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:14.843+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:48.313875+0000 front 2024-03-19T03:43:48.313963+0000 (oldest deadline 2024-03-19T03:44:14.214046+0000) 2024-03-19T03:44:15.290 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:15 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:15.037+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:15.290 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:15 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:15.037+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:15.290 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:15 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:15.037+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:15.290 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:15 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:15.037+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:15.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:15 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:15.112+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:15.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:15 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:15.112+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:15.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:15 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:15.112+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:15.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:15 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:15.112+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:15.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:15 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:15.112+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:15.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:15 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:15.054+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:15.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:15 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:15.054+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:15.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:15 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:15.054+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:15.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:15 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:15.054+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851051+0000 front 2024-03-19T03:43:41.850539+0000 (oldest deadline 2024-03-19T03:44:07.750871+0000) 2024-03-19T03:44:15.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:15 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:15.054+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.052823+0000 front 2024-03-19T03:43:50.052939+0000 (oldest deadline 2024-03-19T03:44:12.352894+0000) 2024-03-19T03:44:15.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:15 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:15.289+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:44:15.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:15 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:15.289+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:44:15.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:15 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:15.289+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851981+0000 front 2024-03-19T03:43:41.852468+0000 (oldest deadline 2024-03-19T03:44:03.551908+0000) 2024-03-19T03:44:15.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:15 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:15.289+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.854157+0000 front 2024-03-19T03:43:49.854260+0000 (oldest deadline 2024-03-19T03:44:15.154184+0000) 2024-03-19T03:44:16.234 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:16 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:16.064+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:16.234 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:16 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:16.064+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:16.234 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:16 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:16.064+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:16.234 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:16 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:16.064+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:16.234 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:16 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:16.064+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:16.234 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:15 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:15.864+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.210249+0000 front 2024-03-19T03:43:23.210454+0000 (oldest deadline 2024-03-19T03:43:47.910358+0000) 2024-03-19T03:44:16.234 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:15 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:15.864+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:34.311998+0000 front 2024-03-19T03:43:34.312155+0000 (oldest deadline 2024-03-19T03:43:58.411836+0000) 2024-03-19T03:44:16.235 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:15 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:15.864+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:38.412580+0000 front 2024-03-19T03:43:38.412664+0000 (oldest deadline 2024-03-19T03:44:03.712408+0000) 2024-03-19T03:44:16.235 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:15 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7[84891]: 2024-03-19T03:44:15.864+0000 7fc5bdb7a700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:48.313875+0000 front 2024-03-19T03:43:48.313963+0000 (oldest deadline 2024-03-19T03:44:14.214046+0000) 2024-03-19T03:44:16.311 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:16 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:16.000+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:16.312 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:16 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:16.000+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:16.312 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:16 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:16.000+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:16.312 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:16 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:16.000+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:16.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:16 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:16.070+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:16.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:16 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:16.070+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:16.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:16 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:16.070+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:16.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:16 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:16.070+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851051+0000 front 2024-03-19T03:43:41.850539+0000 (oldest deadline 2024-03-19T03:44:07.750871+0000) 2024-03-19T03:44:16.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:16 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:16.070+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.052823+0000 front 2024-03-19T03:43:50.052939+0000 (oldest deadline 2024-03-19T03:44:12.352894+0000) 2024-03-19T03:44:16.595 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:16 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:16.311+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:44:16.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:16 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:16.311+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:44:16.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:16 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:16.311+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851981+0000 front 2024-03-19T03:43:41.852468+0000 (oldest deadline 2024-03-19T03:44:03.551908+0000) 2024-03-19T03:44:16.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:16 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:16.311+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.854157+0000 front 2024-03-19T03:43:49.854260+0000 (oldest deadline 2024-03-19T03:44:15.154184+0000) 2024-03-19T03:44:17.099 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:16 smithi119 podman[101518]: 2024-03-19 03:44:16.789439828 +0000 UTC m=+5.514390105 container died f4e014e088954d275822d571676d4de44b37fe1434e389a9d4ca91e68eddce02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-19T03:44:17.100 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:17 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:17.021+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:17.100 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:17 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:17.021+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:17.100 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:17 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:17.021+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:17.100 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:17 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:17.021+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:17.100 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:17 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:17.021+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:17.269 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:16 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:16.971+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:17.269 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:16 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:16.971+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:17.269 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:16 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:16.971+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:17.269 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:16 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:16.971+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:17.449 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:17 smithi119 podman[101518]: 2024-03-19 03:44:17.200511766 +0000 UTC m=+5.925462022 container cleanup f4e014e088954d275822d571676d4de44b37fe1434e389a9d4ca91e68eddce02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-19T03:44:17.449 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:17 smithi119 bash[101518]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7 2024-03-19T03:44:17.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:17 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:17.064+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:17.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:17 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:17.064+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:17.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:17 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:17.064+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:17.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:17 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:17.064+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851051+0000 front 2024-03-19T03:43:41.850539+0000 (oldest deadline 2024-03-19T03:44:07.750871+0000) 2024-03-19T03:44:17.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:17 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:17.064+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.052823+0000 front 2024-03-19T03:43:50.052939+0000 (oldest deadline 2024-03-19T03:44:12.352894+0000) 2024-03-19T03:44:17.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:17 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:17.269+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:44:17.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:17 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:17.269+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:44:17.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:17 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:17.269+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851981+0000 front 2024-03-19T03:43:41.852468+0000 (oldest deadline 2024-03-19T03:44:03.551908+0000) 2024-03-19T03:44:17.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:17 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:17.269+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.854157+0000 front 2024-03-19T03:43:49.854260+0000 (oldest deadline 2024-03-19T03:44:15.154184+0000) 2024-03-19T03:44:17.734 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:17 smithi119 podman[101536]: 2024-03-19 03:44:17.449598396 +0000 UTC m=+0.657928237 container remove f4e014e088954d275822d571676d4de44b37fe1434e389a9d4ca91e68eddce02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-19T03:44:18.095 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:17 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:17.937+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:18.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:17 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:17.937+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:18.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:17 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:17.937+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:18.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:17 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:17.937+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:18.186 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:17 smithi119 podman[101628]: 2024-03-19 03:44:17.754552924 +0000 UTC m=+0.017514929 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:44:18.187 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:17 smithi119 podman[101628]: 2024-03-19 03:44:17.858587927 +0000 UTC m=+0.121549966 container create 97f2ff0f430ec6c6c2c8061ed0f201bfcbbd897fa5bea32128ce35507012bb28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2) 2024-03-19T03:44:18.187 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:18 smithi119 podman[101628]: 2024-03-19 03:44:18.094869823 +0000 UTC m=+0.357831822 container init 97f2ff0f430ec6c6c2c8061ed0f201bfcbbd897fa5bea32128ce35507012bb28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-19T03:44:18.187 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:18 smithi119 podman[101628]: 2024-03-19 03:44:18.09808806 +0000 UTC m=+0.361050069 container start 97f2ff0f430ec6c6c2c8061ed0f201bfcbbd897fa5bea32128ce35507012bb28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-19T03:44:18.187 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:18 smithi119 podman[101628]: 2024-03-19 03:44:18.186430217 +0000 UTC m=+0.449392205 container attach 97f2ff0f430ec6c6c2c8061ed0f201bfcbbd897fa5bea32128ce35507012bb28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T03:44:18.187 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:17 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:17.997+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:18.187 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:17 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:17.997+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:18.187 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:17 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:17.997+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:18.187 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:17 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:17.997+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:18.187 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:17 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:17.997+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:18.484 INFO:journalctl@ceph.osd.7.smithi119.stdout:Mar 19 03:44:18 smithi119 podman[101628]: 2024-03-19 03:44:18.384988698 +0000 UTC m=+0.647950716 container died 97f2ff0f430ec6c6c2c8061ed0f201bfcbbd897fa5bea32128ce35507012bb28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-19T03:44:18.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:18 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:18.108+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:18.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:18 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:18.108+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:18.543 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:18 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:18.108+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:18.543 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:18 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:18.108+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851051+0000 front 2024-03-19T03:43:41.850539+0000 (oldest deadline 2024-03-19T03:44:07.750871+0000) 2024-03-19T03:44:18.543 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:18 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:18.108+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.052823+0000 front 2024-03-19T03:43:50.052939+0000 (oldest deadline 2024-03-19T03:44:12.352894+0000) 2024-03-19T03:44:18.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:18 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:18.287+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:44:18.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:18 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:18.287+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:44:18.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:18 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:18.287+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851981+0000 front 2024-03-19T03:43:41.852468+0000 (oldest deadline 2024-03-19T03:44:03.551908+0000) 2024-03-19T03:44:18.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:18 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:18.287+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.854157+0000 front 2024-03-19T03:43:49.854260+0000 (oldest deadline 2024-03-19T03:44:15.154184+0000) 2024-03-19T03:44:19.057 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:18 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:18.988+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:19.057 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:18 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:18.988+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:19.057 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:18 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:18.988+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:19.057 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:18 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:18.988+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:19.057 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:18 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:18.988+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:19.080 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.7.service' 2024-03-19T03:44:19.125 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:44:19.125 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-19T03:44:19.126 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-19T03:44:19.126 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.8 2024-03-19T03:44:19.174 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:18 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:18.911+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:19.174 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:18 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:18.911+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:19.174 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:18 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:18.911+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:19.174 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:18 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:18.911+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:19.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:19 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:19.063+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:19.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:19 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:19.063+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:19.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:19 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:19.063+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:19.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:19 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:19.063+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851051+0000 front 2024-03-19T03:43:41.850539+0000 (oldest deadline 2024-03-19T03:44:07.750871+0000) 2024-03-19T03:44:19.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:19 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:19.063+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.052823+0000 front 2024-03-19T03:43:50.052939+0000 (oldest deadline 2024-03-19T03:44:12.352894+0000) 2024-03-19T03:44:19.595 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:19 smithi178 systemd[1]: Stopping Ceph osd.8 for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:44:19.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:19 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:19.278+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:44:19.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:19 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:19.278+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:44:19.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:19 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:19.278+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851981+0000 front 2024-03-19T03:43:41.852468+0000 (oldest deadline 2024-03-19T03:44:03.551908+0000) 2024-03-19T03:44:19.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:19 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:19.278+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.854157+0000 front 2024-03-19T03:43:49.854260+0000 (oldest deadline 2024-03-19T03:44:15.154184+0000) 2024-03-19T03:44:19.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:19 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:19.418+0000 7fcbc90c8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:44:19.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:19 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:19.418+0000 7fcbc90c8700 -1 osd.8 97 *** Got signal Terminated *** 2024-03-19T03:44:19.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:19 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:19.418+0000 7fcbc90c8700 -1 osd.8 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T03:44:20.321 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:19 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:19.940+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:20.321 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:19 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:19.940+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:20.321 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:19 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:19.940+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:20.321 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:19 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:19.940+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:20.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:20 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:20.024+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:20.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:20 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:20.024+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:20.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:20 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:20.024+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:20.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:20 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:20.024+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:20.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:20 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:20.024+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:20.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:20 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:20.069+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:20.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:20 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:20.069+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:20.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:20 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:20.069+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:20.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:20 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:20.069+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851051+0000 front 2024-03-19T03:43:41.850539+0000 (oldest deadline 2024-03-19T03:44:07.750871+0000) 2024-03-19T03:44:20.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:20 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:20.069+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.052823+0000 front 2024-03-19T03:43:50.052939+0000 (oldest deadline 2024-03-19T03:44:12.352894+0000) 2024-03-19T03:44:20.595 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:20 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:20.321+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:44:20.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:20 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:20.321+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:44:20.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:20 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:20.321+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851981+0000 front 2024-03-19T03:43:41.852468+0000 (oldest deadline 2024-03-19T03:44:03.551908+0000) 2024-03-19T03:44:20.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:20 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:20.321+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.854157+0000 front 2024-03-19T03:43:49.854260+0000 (oldest deadline 2024-03-19T03:44:15.154184+0000) 2024-03-19T03:44:21.335 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:20 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:20.906+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:21.335 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:20 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:20.906+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:21.335 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:20 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:20.906+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:21.335 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:20 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:20.906+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:21.335 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:20 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:20.906+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:21.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:21 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:21.040+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:21.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:21 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:21.040+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:21.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:21 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:21.040+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:21.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:21 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:21.040+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:21.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:21 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:21.040+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:21.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:21 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:21.076+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:21.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:21 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:21.076+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:21.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:21 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:21.076+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:21.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:21 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:21.076+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851051+0000 front 2024-03-19T03:43:41.850539+0000 (oldest deadline 2024-03-19T03:44:07.750871+0000) 2024-03-19T03:44:21.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:21 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:21.076+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.052823+0000 front 2024-03-19T03:43:50.052939+0000 (oldest deadline 2024-03-19T03:44:12.352894+0000) 2024-03-19T03:44:21.595 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:21 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:21.335+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:44:21.595 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:21 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:21.335+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:44:21.595 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:21 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:21.335+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851981+0000 front 2024-03-19T03:43:41.852468+0000 (oldest deadline 2024-03-19T03:44:03.551908+0000) 2024-03-19T03:44:21.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:21 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:21.335+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.854157+0000 front 2024-03-19T03:43:49.854260+0000 (oldest deadline 2024-03-19T03:44:15.154184+0000) 2024-03-19T03:44:21.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:21 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:21.335+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:55.154498+0000 front 2024-03-19T03:43:55.154452+0000 (oldest deadline 2024-03-19T03:44:21.054599+0000) 2024-03-19T03:44:22.345 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:21 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:21.901+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:22.346 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:21 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:21.901+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:22.346 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:21 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:21.901+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:22.346 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:21 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:21.901+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:22.346 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:21 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:21.901+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:22.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:22 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:22.017+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:22.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:22 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:22.017+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:22.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:22 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:22.017+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:22.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:22 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:22.017+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:22.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:22 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:22.017+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:22.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:22 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:22.090+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:22.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:22 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:22.090+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:22.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:22 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:22.090+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:22.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:22 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:22.090+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851051+0000 front 2024-03-19T03:43:41.850539+0000 (oldest deadline 2024-03-19T03:44:07.750871+0000) 2024-03-19T03:44:22.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:22 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:22.090+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.052823+0000 front 2024-03-19T03:43:50.052939+0000 (oldest deadline 2024-03-19T03:44:12.352894+0000) 2024-03-19T03:44:22.846 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:22 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:22.375+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:44:22.846 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:22 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:22.375+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:44:22.846 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:22 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:22.375+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851981+0000 front 2024-03-19T03:43:41.852468+0000 (oldest deadline 2024-03-19T03:44:03.551908+0000) 2024-03-19T03:44:22.846 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:22 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:22.375+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.854157+0000 front 2024-03-19T03:43:49.854260+0000 (oldest deadline 2024-03-19T03:44:15.154184+0000) 2024-03-19T03:44:22.846 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:22 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:22.375+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:55.154498+0000 front 2024-03-19T03:43:55.154452+0000 (oldest deadline 2024-03-19T03:44:21.054599+0000) 2024-03-19T03:44:23.346 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:22 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:22.859+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:23.346 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:22 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:22.859+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:23.346 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:22 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:22.859+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:23.346 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:22 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:22.859+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:23.346 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:22 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:22.859+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:23.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:23 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:23.015+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:23.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:23 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:23.015+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:23.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:23 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:23.015+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:23.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:23 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:23.015+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:23.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:23 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:23.015+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:23.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:23 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:23.048+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:23.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:23 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:23.048+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:23.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:23 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:23.048+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:23.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:23 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:23.048+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851051+0000 front 2024-03-19T03:43:41.850539+0000 (oldest deadline 2024-03-19T03:44:07.750871+0000) 2024-03-19T03:44:23.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:23 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:23.048+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.052823+0000 front 2024-03-19T03:43:50.052939+0000 (oldest deadline 2024-03-19T03:44:12.352894+0000) 2024-03-19T03:44:23.846 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:23 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:23.402+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:44:23.846 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:23 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:23.402+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:44:23.846 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:23 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:23.402+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851981+0000 front 2024-03-19T03:43:41.852468+0000 (oldest deadline 2024-03-19T03:44:03.551908+0000) 2024-03-19T03:44:23.846 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:23 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:23.402+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.854157+0000 front 2024-03-19T03:43:49.854260+0000 (oldest deadline 2024-03-19T03:44:15.154184+0000) 2024-03-19T03:44:23.846 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:23 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:23.402+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:55.154498+0000 front 2024-03-19T03:43:55.154452+0000 (oldest deadline 2024-03-19T03:44:21.054599+0000) 2024-03-19T03:44:24.346 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:23 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:23.891+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:24.346 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:23 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:23.891+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:24.346 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:23 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:23.891+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:24.346 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:23 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:23.891+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:24.346 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:23 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:23.891+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:24.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:24 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:24.031+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:24.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:24 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:24.031+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:24.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:24 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:24.031+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:24.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:24 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:24.031+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:24.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:24 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:24.031+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:24.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:24 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:24.031+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.243767+0000 front 2024-03-19T03:43:59.243815+0000 (oldest deadline 2024-03-19T03:44:23.343757+0000) 2024-03-19T03:44:24.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:24 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:24.050+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:24.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:24 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:24.050+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:24.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:24 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:24.050+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:24.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:24 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:24.050+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851051+0000 front 2024-03-19T03:43:41.850539+0000 (oldest deadline 2024-03-19T03:44:07.750871+0000) 2024-03-19T03:44:24.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:24 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:24.050+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.052823+0000 front 2024-03-19T03:43:50.052939+0000 (oldest deadline 2024-03-19T03:44:12.352894+0000) 2024-03-19T03:44:24.846 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:24 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:24.390+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.650003+0000 front 2024-03-19T03:43:23.649905+0000 (oldest deadline 2024-03-19T03:43:48.949899+0000) 2024-03-19T03:44:24.846 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:24 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:24.390+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.450982+0000 front 2024-03-19T03:43:32.451003+0000 (oldest deadline 2024-03-19T03:43:55.951083+0000) 2024-03-19T03:44:24.846 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:24 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:24.390+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851981+0000 front 2024-03-19T03:43:41.852468+0000 (oldest deadline 2024-03-19T03:44:03.551908+0000) 2024-03-19T03:44:24.846 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:24 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:24.390+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.854157+0000 front 2024-03-19T03:43:49.854260+0000 (oldest deadline 2024-03-19T03:44:15.154184+0000) 2024-03-19T03:44:24.846 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:24 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8[86312]: 2024-03-19T03:44:24.390+0000 7fcbc56bc700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:55.154498+0000 front 2024-03-19T03:43:55.154452+0000 (oldest deadline 2024-03-19T03:44:21.054599+0000) 2024-03-19T03:44:25.178 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:24 smithi178 podman[103914]: 2024-03-19 03:44:24.915332746 +0000 UTC m=+5.598081559 container died 2539095efb1b05cbfcc318242b4e1991a0b16418d76e37cf97c30410a9f8cecc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-19T03:44:25.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:24 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:24.857+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:25.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:24 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:24.857+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:25.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:24 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:24.857+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:25.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:24 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:24.857+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:25.178 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:24 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:24.857+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:25.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:25 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:25.045+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:25.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:25 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:25.045+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:25.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:25 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:25.045+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:25.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:25 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:25.045+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851051+0000 front 2024-03-19T03:43:41.850539+0000 (oldest deadline 2024-03-19T03:44:07.750871+0000) 2024-03-19T03:44:25.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:25 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:25.045+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.052823+0000 front 2024-03-19T03:43:50.052939+0000 (oldest deadline 2024-03-19T03:44:12.352894+0000) 2024-03-19T03:44:25.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:25 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:25.045+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:54.655105+0000 front 2024-03-19T03:43:54.655153+0000 (oldest deadline 2024-03-19T03:44:24.656207+0000) 2024-03-19T03:44:25.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:24 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:24.985+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:25.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:24 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:24.985+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:25.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:24 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:24.985+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:25.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:24 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:24.985+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:25.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:24 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:24.985+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:25.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:24 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:24.985+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.243767+0000 front 2024-03-19T03:43:59.243815+0000 (oldest deadline 2024-03-19T03:44:23.343757+0000) 2024-03-19T03:44:25.595 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:25 smithi178 podman[103914]: 2024-03-19 03:44:25.284730957 +0000 UTC m=+5.967479760 container cleanup 2539095efb1b05cbfcc318242b4e1991a0b16418d76e37cf97c30410a9f8cecc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-19T03:44:25.596 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:25 smithi178 bash[103914]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8 2024-03-19T03:44:25.902 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:25 smithi178 podman[103931]: 2024-03-19 03:44:25.603200618 +0000 UTC m=+0.690198408 container remove 2539095efb1b05cbfcc318242b4e1991a0b16418d76e37cf97c30410a9f8cecc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-19T03:44:25.903 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:25 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:25.816+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:25.903 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:25 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:25.816+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:25.903 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:25 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:25.817+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:25.903 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:25 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:25.817+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:25.903 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:25 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:25.817+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:26.182 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:25 smithi178 podman[104024]: 2024-03-19 03:44:25.806243114 +0000 UTC m=+0.017969891 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:44:26.182 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:25 smithi178 podman[104024]: 2024-03-19 03:44:25.92101771 +0000 UTC m=+0.132744481 container create dd48e19a91ce8978b82eb5f653704d7b7343527ddfe1b9533e1fe11eb3b8e86a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-19T03:44:26.182 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:26 smithi178 podman[104024]: 2024-03-19 03:44:26.182212114 +0000 UTC m=+0.393938874 container init dd48e19a91ce8978b82eb5f653704d7b7343527ddfe1b9533e1fe11eb3b8e86a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T03:44:26.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:26 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:26.027+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:26.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:26 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:26.027+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:26.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:26 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:26.027+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:26.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:26 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:26.027+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851051+0000 front 2024-03-19T03:43:41.850539+0000 (oldest deadline 2024-03-19T03:44:07.750871+0000) 2024-03-19T03:44:26.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:26 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:26.027+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.052823+0000 front 2024-03-19T03:43:50.052939+0000 (oldest deadline 2024-03-19T03:44:12.352894+0000) 2024-03-19T03:44:26.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:26 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:26.027+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:54.655105+0000 front 2024-03-19T03:43:54.655153+0000 (oldest deadline 2024-03-19T03:44:24.656207+0000) 2024-03-19T03:44:26.474 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:26 smithi178 podman[104024]: 2024-03-19 03:44:26.185846832 +0000 UTC m=+0.397573609 container start dd48e19a91ce8978b82eb5f653704d7b7343527ddfe1b9533e1fe11eb3b8e86a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-19T03:44:26.475 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:26 smithi178 podman[104024]: 2024-03-19 03:44:26.273871744 +0000 UTC m=+0.485598507 container attach dd48e19a91ce8978b82eb5f653704d7b7343527ddfe1b9533e1fe11eb3b8e86a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2) 2024-03-19T03:44:26.483 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:26 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:26.027+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:26.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:26 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:26.027+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:26.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:26 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:26.027+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:26.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:26 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:26.027+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:26.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:26 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:26.027+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:26.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:26 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:26.027+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.243767+0000 front 2024-03-19T03:43:59.243815+0000 (oldest deadline 2024-03-19T03:44:23.343757+0000) 2024-03-19T03:44:26.836 INFO:journalctl@ceph.osd.8.smithi178.stdout:Mar 19 03:44:26 smithi178 podman[104024]: 2024-03-19 03:44:26.475058302 +0000 UTC m=+0.686785077 container died dd48e19a91ce8978b82eb5f653704d7b7343527ddfe1b9533e1fe11eb3b8e86a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-19T03:44:27.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:26 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:26.867+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:27.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:26 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:26.867+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:27.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:26 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:26.867+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:27.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:26 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:26.867+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:27.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:26 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:26.867+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:27.170 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.8.service' 2024-03-19T03:44:27.213 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:44:27.213 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-19T03:44:27.213 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-19T03:44:27.213 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.9 2024-03-19T03:44:27.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:27 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:27.020+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:27.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:27 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:27.020+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:27.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:27 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:27.020+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:27.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:27 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:27.020+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851051+0000 front 2024-03-19T03:43:41.850539+0000 (oldest deadline 2024-03-19T03:44:07.750871+0000) 2024-03-19T03:44:27.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:27 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:27.020+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.052823+0000 front 2024-03-19T03:43:50.052939+0000 (oldest deadline 2024-03-19T03:44:12.352894+0000) 2024-03-19T03:44:27.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:27 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:27.020+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:54.655105+0000 front 2024-03-19T03:43:54.655153+0000 (oldest deadline 2024-03-19T03:44:24.656207+0000) 2024-03-19T03:44:27.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:27 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:27.049+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:27.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:27 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:27.049+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:27.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:27 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:27.049+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:27.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:27 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:27.049+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:27.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:27 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:27.049+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:27.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:27 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:27.049+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.243767+0000 front 2024-03-19T03:43:59.243815+0000 (oldest deadline 2024-03-19T03:44:23.343757+0000) 2024-03-19T03:44:27.772 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:27 smithi117 systemd[1]: Stopping Ceph osd.9 for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:44:28.042 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:27 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:27.772+0000 7fdfe0200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:44:28.042 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:27 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:27.772+0000 7fdfe0200700 -1 osd.9 97 *** Got signal Terminated *** 2024-03-19T03:44:28.043 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:27 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:27.772+0000 7fdfe0200700 -1 osd.9 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T03:44:28.043 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:27 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:27.993+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:28.043 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:27 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:27.993+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:28.043 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:27 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:27.993+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:28.043 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:27 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:27.993+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851051+0000 front 2024-03-19T03:43:41.850539+0000 (oldest deadline 2024-03-19T03:44:07.750871+0000) 2024-03-19T03:44:28.043 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:27 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:27.993+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.052823+0000 front 2024-03-19T03:43:50.052939+0000 (oldest deadline 2024-03-19T03:44:12.352894+0000) 2024-03-19T03:44:28.043 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:27 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:27.993+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:54.655105+0000 front 2024-03-19T03:43:54.655153+0000 (oldest deadline 2024-03-19T03:44:24.656207+0000) 2024-03-19T03:44:28.095 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:27 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:27.822+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:28.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:27 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:27.823+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:28.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:27 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:27.823+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:28.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:27 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:27.823+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:28.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:27 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:27.823+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:28.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:28 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:28.051+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:28.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:28 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:28.051+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:28.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:28 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:28.051+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:28.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:28 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:28.051+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:28.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:28 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:28.051+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:28.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:28 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:28.051+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.243767+0000 front 2024-03-19T03:43:59.243815+0000 (oldest deadline 2024-03-19T03:44:23.343757+0000) 2024-03-19T03:44:29.095 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:28 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:28.778+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:29.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:28 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:28.778+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:29.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:28 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:28.778+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:29.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:28 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:28.778+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:29.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:28 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:28.778+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:29.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:28 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:28.778+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.146167+0000 front 2024-03-19T03:44:06.146054+0000 (oldest deadline 2024-03-19T03:44:28.346390+0000) 2024-03-19T03:44:29.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:29 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:29.051+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:29.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:29 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:29.051+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:29.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:29 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:29.051+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:29.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:29 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:29.051+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:29.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:29 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:29.051+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:29.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:29 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:29.051+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.243767+0000 front 2024-03-19T03:43:59.243815+0000 (oldest deadline 2024-03-19T03:44:23.343757+0000) 2024-03-19T03:44:29.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:29 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:29.051+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.745477+0000 front 2024-03-19T03:44:06.745307+0000 (oldest deadline 2024-03-19T03:44:28.945608+0000) 2024-03-19T03:44:29.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:29 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:29.042+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:29.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:29 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:29.042+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:29.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:29 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:29.042+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:29.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:29 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:29.042+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851051+0000 front 2024-03-19T03:43:41.850539+0000 (oldest deadline 2024-03-19T03:44:07.750871+0000) 2024-03-19T03:44:29.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:29 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:29.043+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.052823+0000 front 2024-03-19T03:43:50.052939+0000 (oldest deadline 2024-03-19T03:44:12.352894+0000) 2024-03-19T03:44:29.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:29 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:29.043+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:54.655105+0000 front 2024-03-19T03:43:54.655153+0000 (oldest deadline 2024-03-19T03:44:24.656207+0000) 2024-03-19T03:44:30.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:29 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:29.767+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:30.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:29 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:29.767+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:30.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:29 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:29.767+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:30.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:29 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:29.767+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:30.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:29 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:29.767+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:30.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:29 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:29.767+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.146167+0000 front 2024-03-19T03:44:06.146054+0000 (oldest deadline 2024-03-19T03:44:28.346390+0000) 2024-03-19T03:44:30.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:30 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:30.096+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:30.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:30 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:30.096+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:30.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:30 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:30.097+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:30.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:30 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:30.097+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:30.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:30 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:30.097+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:30.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:30 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:30.097+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.243767+0000 front 2024-03-19T03:43:59.243815+0000 (oldest deadline 2024-03-19T03:44:23.343757+0000) 2024-03-19T03:44:30.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:30 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:30.097+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.745477+0000 front 2024-03-19T03:44:06.745307+0000 (oldest deadline 2024-03-19T03:44:28.945608+0000) 2024-03-19T03:44:30.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:30 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:30.062+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:30.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:30 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:30.062+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:30.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:30 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:30.062+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:30.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:30 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:30.062+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851051+0000 front 2024-03-19T03:43:41.850539+0000 (oldest deadline 2024-03-19T03:44:07.750871+0000) 2024-03-19T03:44:30.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:30 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:30.062+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.052823+0000 front 2024-03-19T03:43:50.052939+0000 (oldest deadline 2024-03-19T03:44:12.352894+0000) 2024-03-19T03:44:30.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:30 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:30.062+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:54.655105+0000 front 2024-03-19T03:43:54.655153+0000 (oldest deadline 2024-03-19T03:44:24.656207+0000) 2024-03-19T03:44:31.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:30 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:30.719+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:31.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:30 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:30.719+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:31.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:30 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:30.719+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:31.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:30 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:30.719+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:31.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:30 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:30.719+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:31.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:30 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:30.719+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.146167+0000 front 2024-03-19T03:44:06.146054+0000 (oldest deadline 2024-03-19T03:44:28.346390+0000) 2024-03-19T03:44:31.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:31 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:31.132+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:31.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:31 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:31.132+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:31.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:31 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:31.132+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:31.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:31 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:31.132+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:31.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:31 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:31.132+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:31.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:31 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:31.133+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.243767+0000 front 2024-03-19T03:43:59.243815+0000 (oldest deadline 2024-03-19T03:44:23.343757+0000) 2024-03-19T03:44:31.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:31 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:31.133+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.745477+0000 front 2024-03-19T03:44:06.745307+0000 (oldest deadline 2024-03-19T03:44:28.945608+0000) 2024-03-19T03:44:31.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:31 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:31.085+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:31.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:31 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:31.085+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:31.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:31 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:31.085+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:31.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:31 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:31.085+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851051+0000 front 2024-03-19T03:43:41.850539+0000 (oldest deadline 2024-03-19T03:44:07.750871+0000) 2024-03-19T03:44:31.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:31 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:31.085+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.052823+0000 front 2024-03-19T03:43:50.052939+0000 (oldest deadline 2024-03-19T03:44:12.352894+0000) 2024-03-19T03:44:31.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:31 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:31.085+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:54.655105+0000 front 2024-03-19T03:43:54.655153+0000 (oldest deadline 2024-03-19T03:44:24.656207+0000) 2024-03-19T03:44:32.095 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:31 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:31.766+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:32.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:31 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:31.766+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:32.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:31 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:31.766+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:32.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:31 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:31.766+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:32.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:31 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:31.766+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:32.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:31 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:31.766+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.146167+0000 front 2024-03-19T03:44:06.146054+0000 (oldest deadline 2024-03-19T03:44:28.346390+0000) 2024-03-19T03:44:32.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:32 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:32.134+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:32.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:32 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:32.134+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:32.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:32 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:32.134+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:32.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:32 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:32.134+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:32.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:32 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:32.134+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:32.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:32 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:32.134+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.243767+0000 front 2024-03-19T03:43:59.243815+0000 (oldest deadline 2024-03-19T03:44:23.343757+0000) 2024-03-19T03:44:32.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:32 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:32.134+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.745477+0000 front 2024-03-19T03:44:06.745307+0000 (oldest deadline 2024-03-19T03:44:28.945608+0000) 2024-03-19T03:44:32.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:32 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:32.066+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:16.340509+0000 front 2024-03-19T03:43:16.340922+0000 (oldest deadline 2024-03-19T03:43:39.239701+0000) 2024-03-19T03:44:32.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:32 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:32.066+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:26.242822+0000 front 2024-03-19T03:43:26.242876+0000 (oldest deadline 2024-03-19T03:43:49.644424+0000) 2024-03-19T03:44:32.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:32 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:32.066+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:30.746838+0000 front 2024-03-19T03:43:30.746786+0000 (oldest deadline 2024-03-19T03:43:56.646856+0000) 2024-03-19T03:44:32.542 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:32 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:32.066+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:41.851051+0000 front 2024-03-19T03:43:41.850539+0000 (oldest deadline 2024-03-19T03:44:07.750871+0000) 2024-03-19T03:44:32.543 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:32 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:32.066+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.052823+0000 front 2024-03-19T03:43:50.052939+0000 (oldest deadline 2024-03-19T03:44:12.352894+0000) 2024-03-19T03:44:32.543 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:32 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9[67061]: 2024-03-19T03:44:32.066+0000 7fdfd6a00700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:54.655105+0000 front 2024-03-19T03:43:54.655153+0000 (oldest deadline 2024-03-19T03:44:24.656207+0000) 2024-03-19T03:44:33.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:32 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:32.728+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:33.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:32 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:32.728+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:33.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:32 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:32.728+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:33.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:32 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:32.728+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:33.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:32 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:32.728+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:33.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:32 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:32.728+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.146167+0000 front 2024-03-19T03:44:06.146054+0000 (oldest deadline 2024-03-19T03:44:28.346390+0000) 2024-03-19T03:44:33.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:33 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:33.156+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:33.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:33 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:33.156+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:33.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:33 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:33.156+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:33.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:33 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:33.156+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:33.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:33 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:33.156+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:33.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:33 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:33.156+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.243767+0000 front 2024-03-19T03:43:59.243815+0000 (oldest deadline 2024-03-19T03:44:23.343757+0000) 2024-03-19T03:44:33.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:33 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:33.156+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.745477+0000 front 2024-03-19T03:44:06.745307+0000 (oldest deadline 2024-03-19T03:44:28.945608+0000) 2024-03-19T03:44:33.533 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:33 smithi117 podman[93436]: 2024-03-19 03:44:33.258148307 +0000 UTC m=+5.611422576 container died c78311a78edfd0bb14b90c15c5392b9b1298f0ebf2f66d28e6d58f62e493cffb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-19T03:44:33.792 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:33 smithi117 podman[93436]: 2024-03-19 03:44:33.65046266 +0000 UTC m=+6.003736922 container cleanup c78311a78edfd0bb14b90c15c5392b9b1298f0ebf2f66d28e6d58f62e493cffb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-19T03:44:33.792 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:33 smithi117 bash[93436]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9 2024-03-19T03:44:34.095 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:33 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:33.766+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:34.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:33 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:33.766+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:34.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:33 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:33.766+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:34.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:33 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:33.766+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:34.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:33 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:33.766+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:34.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:33 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:33.766+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.146167+0000 front 2024-03-19T03:44:06.146054+0000 (oldest deadline 2024-03-19T03:44:28.346390+0000) 2024-03-19T03:44:34.292 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:33 smithi117 podman[93453]: 2024-03-19 03:44:33.83129073 +0000 UTC m=+0.558085014 container remove c78311a78edfd0bb14b90c15c5392b9b1298f0ebf2f66d28e6d58f62e493cffb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:44:34.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:34 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:34.125+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:34.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:34 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:34.125+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:34.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:34 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:34.125+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:34.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:34 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:34.125+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:34.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:34 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:34.125+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:34.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:34 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:34.125+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.243767+0000 front 2024-03-19T03:43:59.243815+0000 (oldest deadline 2024-03-19T03:44:23.343757+0000) 2024-03-19T03:44:34.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:34 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:34.125+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.745477+0000 front 2024-03-19T03:44:06.745307+0000 (oldest deadline 2024-03-19T03:44:28.945608+0000) 2024-03-19T03:44:34.909 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:34 smithi117 podman[93542]: 2024-03-19 03:44:34.558384492 +0000 UTC m=+0.032138064 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:44:34.909 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:34 smithi117 podman[93542]: 2024-03-19 03:44:34.669302329 +0000 UTC m=+0.143055934 container create bc5222d8d4cac9f990693011358a827b6b3f59f120d4af345fedf42fb35ed141 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-19T03:44:34.910 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:34 smithi117 podman[93542]: 2024-03-19 03:44:34.898037402 +0000 UTC m=+0.371791037 container init bc5222d8d4cac9f990693011358a827b6b3f59f120d4af345fedf42fb35ed141 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-19T03:44:34.910 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:34 smithi117 podman[93542]: 2024-03-19 03:44:34.908634444 +0000 UTC m=+0.382388079 container start bc5222d8d4cac9f990693011358a827b6b3f59f120d4af345fedf42fb35ed141 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-19T03:44:35.095 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:34 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:34.773+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:35.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:34 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:34.773+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:35.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:34 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:34.773+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:35.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:34 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:34.773+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:35.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:34 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:34.773+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:35.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:34 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:34.773+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.146167+0000 front 2024-03-19T03:44:06.146054+0000 (oldest deadline 2024-03-19T03:44:28.346390+0000) 2024-03-19T03:44:35.291 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:34 smithi117 podman[93542]: 2024-03-19 03:44:34.987965698 +0000 UTC m=+0.461719316 container attach bc5222d8d4cac9f990693011358a827b6b3f59f120d4af345fedf42fb35ed141 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-19T03:44:35.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:35 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:35.173+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:35.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:35 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:35.173+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:35.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:35 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:35.173+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:35.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:35 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:35.173+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:35.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:35 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:35.173+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:35.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:35 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:35.173+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.243767+0000 front 2024-03-19T03:43:59.243815+0000 (oldest deadline 2024-03-19T03:44:23.343757+0000) 2024-03-19T03:44:35.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:35 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:35.173+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.745477+0000 front 2024-03-19T03:44:06.745307+0000 (oldest deadline 2024-03-19T03:44:28.945608+0000) 2024-03-19T03:44:35.673 INFO:journalctl@ceph.osd.9.smithi117.stdout:Mar 19 03:44:35 smithi117 podman[93568]: 2024-03-19 03:44:35.390386661 +0000 UTC m=+0.037339639 container died bc5222d8d4cac9f990693011358a827b6b3f59f120d4af345fedf42fb35ed141 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-9-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T03:44:36.010 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.9.service' 2024-03-19T03:44:36.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:35 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:35.815+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:36.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:35 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:35.815+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:36.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:35 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:35.815+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:36.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:35 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:35.815+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:36.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:35 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:35.815+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:36.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:35 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:35.815+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.146167+0000 front 2024-03-19T03:44:06.146054+0000 (oldest deadline 2024-03-19T03:44:28.346390+0000) 2024-03-19T03:44:36.112 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:44:36.112 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-19T03:44:36.113 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-19T03:44:36.113 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.10 2024-03-19T03:44:36.435 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:36 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:36.154+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:36.435 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:36 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:36.154+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:36.435 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:36 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:36.154+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:36.435 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:36 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:36.154+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:36.435 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:36 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:36.154+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:36.435 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:36 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:36.154+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.243767+0000 front 2024-03-19T03:43:59.243815+0000 (oldest deadline 2024-03-19T03:44:23.343757+0000) 2024-03-19T03:44:36.435 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:36 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:36.154+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.745477+0000 front 2024-03-19T03:44:06.745307+0000 (oldest deadline 2024-03-19T03:44:28.945608+0000) 2024-03-19T03:44:36.435 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:36 smithi119 systemd[1]: Stopping Ceph osd.10 for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:44:36.435 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:36 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:36.434+0000 7faa3cb98700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:44:36.435 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:36 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:36.434+0000 7faa3cb98700 -1 osd.10 97 *** Got signal Terminated *** 2024-03-19T03:44:36.734 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:36 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:36.434+0000 7faa3cb98700 -1 osd.10 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T03:44:37.095 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:36 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:36.799+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:37.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:36 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:36.799+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:37.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:36 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:36.799+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:37.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:36 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:36.799+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:37.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:36 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:36.799+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:37.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:36 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:36.799+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.146167+0000 front 2024-03-19T03:44:06.146054+0000 (oldest deadline 2024-03-19T03:44:28.346390+0000) 2024-03-19T03:44:37.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:37 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:37.171+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:37.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:37 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:37.171+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:37.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:37 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:37.171+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:37.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:37 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:37.171+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:37.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:37 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:37.171+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:37.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:37 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:37.171+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.243767+0000 front 2024-03-19T03:43:59.243815+0000 (oldest deadline 2024-03-19T03:44:23.343757+0000) 2024-03-19T03:44:37.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:37 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:37.171+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.745477+0000 front 2024-03-19T03:44:06.745307+0000 (oldest deadline 2024-03-19T03:44:28.945608+0000) 2024-03-19T03:44:38.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:37 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:37.754+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:38.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:37 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:37.754+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:38.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:37 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:37.754+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:38.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:37 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:37.754+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:38.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:37 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:37.754+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:38.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:37 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:37.754+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.146167+0000 front 2024-03-19T03:44:06.146054+0000 (oldest deadline 2024-03-19T03:44:28.346390+0000) 2024-03-19T03:44:38.097 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:37 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:37.754+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-03-19T03:44:12.447552+0000 front 2024-03-19T03:44:12.447470+0000 (oldest deadline 2024-03-19T03:44:37.147263+0000) 2024-03-19T03:44:38.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:38 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:38.166+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:38.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:38 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:38.166+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:38.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:38 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:38.166+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:38.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:38 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:38.166+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:38.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:38 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:38.166+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:38.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:38 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:38.166+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.243767+0000 front 2024-03-19T03:43:59.243815+0000 (oldest deadline 2024-03-19T03:44:23.343757+0000) 2024-03-19T03:44:38.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:38 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:38.166+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.745477+0000 front 2024-03-19T03:44:06.745307+0000 (oldest deadline 2024-03-19T03:44:28.945608+0000) 2024-03-19T03:44:39.095 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:38 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:38.783+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:39.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:38 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:38.783+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:39.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:38 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:38.783+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:39.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:38 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:38.783+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:39.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:38 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:38.783+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:39.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:38 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:38.783+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.146167+0000 front 2024-03-19T03:44:06.146054+0000 (oldest deadline 2024-03-19T03:44:28.346390+0000) 2024-03-19T03:44:39.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:38 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:38.783+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-03-19T03:44:12.447552+0000 front 2024-03-19T03:44:12.447470+0000 (oldest deadline 2024-03-19T03:44:37.147263+0000) 2024-03-19T03:44:39.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:39 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:39.196+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:39.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:39 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:39.196+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:39.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:39 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:39.196+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:39.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:39 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:39.196+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:39.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:39 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:39.196+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:39.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:39 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:39.196+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.243767+0000 front 2024-03-19T03:43:59.243815+0000 (oldest deadline 2024-03-19T03:44:23.343757+0000) 2024-03-19T03:44:39.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:39 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:39.196+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.745477+0000 front 2024-03-19T03:44:06.745307+0000 (oldest deadline 2024-03-19T03:44:28.945608+0000) 2024-03-19T03:44:40.095 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:39 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:39.740+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:40.095 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:39 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:39.740+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:40.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:39 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:39.740+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:40.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:39 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:39.740+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:40.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:39 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:39.740+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:40.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:39 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:39.740+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.146167+0000 front 2024-03-19T03:44:06.146054+0000 (oldest deadline 2024-03-19T03:44:28.346390+0000) 2024-03-19T03:44:40.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:39 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:39.740+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-03-19T03:44:12.447552+0000 front 2024-03-19T03:44:12.447470+0000 (oldest deadline 2024-03-19T03:44:37.147263+0000) 2024-03-19T03:44:40.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:40 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:40.215+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:40.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:40 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:40.215+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:40.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:40 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:40.215+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:40.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:40 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:40.215+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:40.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:40 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:40.215+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:40.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:40 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:40.215+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.243767+0000 front 2024-03-19T03:43:59.243815+0000 (oldest deadline 2024-03-19T03:44:23.343757+0000) 2024-03-19T03:44:40.485 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:40 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:40.215+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.745477+0000 front 2024-03-19T03:44:06.745307+0000 (oldest deadline 2024-03-19T03:44:28.945608+0000) 2024-03-19T03:44:41.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:40 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:40.789+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:41.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:40 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:40.789+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:41.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:40 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:40.789+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:41.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:40 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:40.789+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:41.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:40 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:40.789+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:41.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:40 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:40.789+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.146167+0000 front 2024-03-19T03:44:06.146054+0000 (oldest deadline 2024-03-19T03:44:28.346390+0000) 2024-03-19T03:44:41.097 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:40 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:40.789+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-03-19T03:44:12.447552+0000 front 2024-03-19T03:44:12.447470+0000 (oldest deadline 2024-03-19T03:44:37.147263+0000) 2024-03-19T03:44:41.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:41 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:41.196+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:14.134594+0000 front 2024-03-19T03:43:14.134696+0000 (oldest deadline 2024-03-19T03:43:40.535562+0000) 2024-03-19T03:44:41.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:41 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:41.196+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:24.036512+0000 front 2024-03-19T03:43:24.036446+0000 (oldest deadline 2024-03-19T03:43:48.136586+0000) 2024-03-19T03:44:41.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:41 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:41.197+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:32.138325+0000 front 2024-03-19T03:43:32.138396+0000 (oldest deadline 2024-03-19T03:43:55.638275+0000) 2024-03-19T03:44:41.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:41 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:41.197+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6814 osd.3 since back 2024-03-19T03:43:42.040035+0000 front 2024-03-19T03:43:42.040127+0000 (oldest deadline 2024-03-19T03:44:05.539885+0000) 2024-03-19T03:44:41.485 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:41 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:41.197+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:50.741266+0000 front 2024-03-19T03:43:50.741301+0000 (oldest deadline 2024-03-19T03:44:14.141790+0000) 2024-03-19T03:44:41.485 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:41 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:41.197+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.243767+0000 front 2024-03-19T03:43:59.243815+0000 (oldest deadline 2024-03-19T03:44:23.343757+0000) 2024-03-19T03:44:41.485 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:41 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10[91646]: 2024-03-19T03:44:41.197+0000 7faa3918c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.745477+0000 front 2024-03-19T03:44:06.745307+0000 (oldest deadline 2024-03-19T03:44:28.945608+0000) 2024-03-19T03:44:42.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:41 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:41.748+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:42.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:41 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:41.748+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:42.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:41 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:41.748+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:42.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:41 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:41.748+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:42.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:41 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:41.748+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:42.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:41 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:41.748+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.146167+0000 front 2024-03-19T03:44:06.146054+0000 (oldest deadline 2024-03-19T03:44:28.346390+0000) 2024-03-19T03:44:42.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:41 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:41.748+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-03-19T03:44:12.447552+0000 front 2024-03-19T03:44:12.447470+0000 (oldest deadline 2024-03-19T03:44:37.147263+0000) 2024-03-19T03:44:42.234 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:41 smithi119 podman[101732]: 2024-03-19 03:44:41.967782294 +0000 UTC m=+5.654524319 container died b65964565b8722ed1d8ceb9a783b21e4c079a6b3c033e134e9353f222339c179 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-19T03:44:42.585 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:42 smithi119 podman[101732]: 2024-03-19 03:44:42.350473773 +0000 UTC m=+6.037215783 container cleanup b65964565b8722ed1d8ceb9a783b21e4c079a6b3c033e134e9353f222339c179 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-19T03:44:42.585 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:42 smithi119 bash[101732]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10 2024-03-19T03:44:42.893 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:42 smithi119 podman[101760]: 2024-03-19 03:44:42.585066741 +0000 UTC m=+0.615391394 container remove b65964565b8722ed1d8ceb9a783b21e4c079a6b3c033e134e9353f222339c179 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-19T03:44:43.095 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:42 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:42.765+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:43.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:42 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:42.765+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:43.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:42 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:42.765+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:43.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:42 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:42.765+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:43.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:42 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:42.765+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:43.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:42 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:42.765+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.146167+0000 front 2024-03-19T03:44:06.146054+0000 (oldest deadline 2024-03-19T03:44:28.346390+0000) 2024-03-19T03:44:43.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:42 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:42.765+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-03-19T03:44:12.447552+0000 front 2024-03-19T03:44:12.447470+0000 (oldest deadline 2024-03-19T03:44:37.147263+0000) 2024-03-19T03:44:43.179 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:42 smithi119 podman[101852]: 2024-03-19 03:44:42.893835853 +0000 UTC m=+0.115038170 container create 5a4546094e66caa39ed9223718d8694d6d65548385ec4bf64ca296165fce0655 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True) 2024-03-19T03:44:43.179 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:42 smithi119 podman[101852]: 2024-03-19 03:44:42.796332582 +0000 UTC m=+0.017534909 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:44:43.179 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:43 smithi119 podman[101852]: 2024-03-19 03:44:43.105314654 +0000 UTC m=+0.326516992 container init 5a4546094e66caa39ed9223718d8694d6d65548385ec4bf64ca296165fce0655 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:44:43.180 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:43 smithi119 podman[101852]: 2024-03-19 03:44:43.108705379 +0000 UTC m=+0.329907704 container start 5a4546094e66caa39ed9223718d8694d6d65548385ec4bf64ca296165fce0655 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS) 2024-03-19T03:44:43.180 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:43 smithi119 podman[101852]: 2024-03-19 03:44:43.179503812 +0000 UTC m=+0.400706145 container attach 5a4546094e66caa39ed9223718d8694d6d65548385ec4bf64ca296165fce0655 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-19T03:44:43.484 INFO:journalctl@ceph.osd.10.smithi119.stdout:Mar 19 03:44:43 smithi119 podman[101852]: 2024-03-19 03:44:43.402028351 +0000 UTC m=+0.623230680 container died 5a4546094e66caa39ed9223718d8694d6d65548385ec4bf64ca296165fce0655 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-10-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T03:44:43.972 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.10.service' 2024-03-19T03:44:44.012 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:44:44.013 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-19T03:44:44.013 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-19T03:44:44.013 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.11 2024-03-19T03:44:44.043 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:43 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:43.763+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:44.043 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:43 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:43.763+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:44.043 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:43 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:43.763+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:44.043 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:43 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:43.763+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:44.044 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:43 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:43.763+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:44.044 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:43 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:43.763+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.146167+0000 front 2024-03-19T03:44:06.146054+0000 (oldest deadline 2024-03-19T03:44:28.346390+0000) 2024-03-19T03:44:44.044 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:43 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:43.763+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-03-19T03:44:12.447552+0000 front 2024-03-19T03:44:12.447470+0000 (oldest deadline 2024-03-19T03:44:37.147263+0000) 2024-03-19T03:44:44.298 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:44 smithi178 systemd[1]: Stopping Ceph osd.11 for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:44:44.595 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:44 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:44.296+0000 7f93f8d46700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:44:44.595 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:44 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:44.296+0000 7f93f8d46700 -1 osd.11 97 *** Got signal Terminated *** 2024-03-19T03:44:44.596 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:44 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:44.296+0000 7f93f8d46700 -1 osd.11 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T03:44:45.095 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:44 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:44.795+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:45.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:44 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:44.795+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:45.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:44 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:44.795+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:45.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:44 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:44.795+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:45.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:44 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:44.795+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:45.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:44 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:44.795+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.146167+0000 front 2024-03-19T03:44:06.146054+0000 (oldest deadline 2024-03-19T03:44:28.346390+0000) 2024-03-19T03:44:45.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:44 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:44.795+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-03-19T03:44:12.447552+0000 front 2024-03-19T03:44:12.447470+0000 (oldest deadline 2024-03-19T03:44:37.147263+0000) 2024-03-19T03:44:46.058 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:45 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:45.783+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:46.059 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:45 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:45.783+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:46.059 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:45 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:45.783+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:46.059 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:45 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:45.783+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:46.059 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:45 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:45.783+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:46.059 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:45 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:45.783+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.146167+0000 front 2024-03-19T03:44:06.146054+0000 (oldest deadline 2024-03-19T03:44:28.346390+0000) 2024-03-19T03:44:46.059 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:45 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:45.783+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-03-19T03:44:12.447552+0000 front 2024-03-19T03:44:12.447470+0000 (oldest deadline 2024-03-19T03:44:37.147263+0000) 2024-03-19T03:44:47.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:46 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:46.737+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:47.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:46 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:46.737+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:47.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:46 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:46.737+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:47.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:46 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:46.737+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:47.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:46 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:46.737+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:47.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:46 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:46.737+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.146167+0000 front 2024-03-19T03:44:06.146054+0000 (oldest deadline 2024-03-19T03:44:28.346390+0000) 2024-03-19T03:44:47.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:46 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:46.737+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-03-19T03:44:12.447552+0000 front 2024-03-19T03:44:12.447470+0000 (oldest deadline 2024-03-19T03:44:37.147263+0000) 2024-03-19T03:44:47.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:46 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:46.737+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6820 osd.8 since back 2024-03-19T03:44:22.448262+0000 front 2024-03-19T03:44:22.448281+0000 (oldest deadline 2024-03-19T03:44:45.948211+0000) 2024-03-19T03:44:48.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:47 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:47.743+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:48.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:47 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:47.743+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:48.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:47 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:47.743+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:48.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:47 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:47.743+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:48.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:47 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:47.743+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:48.097 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:47 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:47.743+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.146167+0000 front 2024-03-19T03:44:06.146054+0000 (oldest deadline 2024-03-19T03:44:28.346390+0000) 2024-03-19T03:44:48.097 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:47 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:47.743+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-03-19T03:44:12.447552+0000 front 2024-03-19T03:44:12.447470+0000 (oldest deadline 2024-03-19T03:44:37.147263+0000) 2024-03-19T03:44:48.097 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:47 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:47.743+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6820 osd.8 since back 2024-03-19T03:44:22.448262+0000 front 2024-03-19T03:44:22.448281+0000 (oldest deadline 2024-03-19T03:44:45.948211+0000) 2024-03-19T03:44:49.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:48 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:48.774+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6806 osd.0 since back 2024-03-19T03:43:13.939926+0000 front 2024-03-19T03:43:13.939948+0000 (oldest deadline 2024-03-19T03:43:38.039750+0000) 2024-03-19T03:44:49.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:48 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:48.774+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6804 osd.1 since back 2024-03-19T03:43:23.841332+0000 front 2024-03-19T03:43:23.841480+0000 (oldest deadline 2024-03-19T03:43:49.741288+0000) 2024-03-19T03:44:49.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:48 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:48.774+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2024-03-19T03:43:29.741530+0000 front 2024-03-19T03:43:29.741644+0000 (oldest deadline 2024-03-19T03:43:55.641663+0000) 2024-03-19T03:44:49.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:48 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:48.774+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6812 osd.4 since back 2024-03-19T03:43:49.743628+0000 front 2024-03-19T03:43:49.743605+0000 (oldest deadline 2024-03-19T03:44:14.443678+0000) 2024-03-19T03:44:49.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:48 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:48.774+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2024-03-19T03:43:59.144758+0000 front 2024-03-19T03:43:59.144654+0000 (oldest deadline 2024-03-19T03:44:20.244790+0000) 2024-03-19T03:44:49.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:48 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:48.774+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.117:6822 osd.6 since back 2024-03-19T03:44:06.146167+0000 front 2024-03-19T03:44:06.146054+0000 (oldest deadline 2024-03-19T03:44:28.346390+0000) 2024-03-19T03:44:49.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:48 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:48.774+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.119:6820 osd.7 since back 2024-03-19T03:44:12.447552+0000 front 2024-03-19T03:44:12.447470+0000 (oldest deadline 2024-03-19T03:44:37.147263+0000) 2024-03-19T03:44:49.096 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:48 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11[92975]: 2024-03-19T03:44:48.774+0000 7f93f1332700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.178:6820 osd.8 since back 2024-03-19T03:44:22.448262+0000 front 2024-03-19T03:44:22.448281+0000 (oldest deadline 2024-03-19T03:44:45.948211+0000) 2024-03-19T03:44:49.989 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:49 smithi178 podman[104140]: 2024-03-19 03:44:49.664242989 +0000 UTC m=+5.458561203 container died ec6bb26c6b203ec22f753a7e2c0dd5d904d83b77128a48dd27ff03dbfd6072e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-19T03:44:49.989 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:49 smithi178 podman[104140]: 2024-03-19 03:44:49.988898736 +0000 UTC m=+5.783216945 container cleanup ec6bb26c6b203ec22f753a7e2c0dd5d904d83b77128a48dd27ff03dbfd6072e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-19T03:44:50.345 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:49 smithi178 bash[104140]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11 2024-03-19T03:44:50.346 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:50 smithi178 podman[104156]: 2024-03-19 03:44:50.198567397 +0000 UTC m=+0.529120027 container remove ec6bb26c6b203ec22f753a7e2c0dd5d904d83b77128a48dd27ff03dbfd6072e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True) 2024-03-19T03:44:50.876 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:50 smithi178 podman[104248]: 2024-03-19 03:44:50.599631883 +0000 UTC m=+0.114108658 container create cf5a11379f9ef933d0cfa8f85484683b5ff6869ca00a25793a1b0146aeaeca45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True) 2024-03-19T03:44:50.877 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:50 smithi178 podman[104248]: 2024-03-19 03:44:50.503865199 +0000 UTC m=+0.018341988 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T03:44:50.877 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:50 smithi178 podman[104248]: 2024-03-19 03:44:50.802442819 +0000 UTC m=+0.316919612 container init cf5a11379f9ef933d0cfa8f85484683b5ff6869ca00a25793a1b0146aeaeca45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-19T03:44:50.877 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:50 smithi178 podman[104248]: 2024-03-19 03:44:50.80570075 +0000 UTC m=+0.320177534 container start cf5a11379f9ef933d0cfa8f85484683b5ff6869ca00a25793a1b0146aeaeca45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-19T03:44:50.877 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:50 smithi178 podman[104248]: 2024-03-19 03:44:50.876869047 +0000 UTC m=+0.391345847 container attach cf5a11379f9ef933d0cfa8f85484683b5ff6869ca00a25793a1b0146aeaeca45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True) 2024-03-19T03:44:51.345 INFO:journalctl@ceph.osd.11.smithi178.stdout:Mar 19 03:44:51 smithi178 podman[104248]: 2024-03-19 03:44:51.093399213 +0000 UTC m=+0.607875995 container died cf5a11379f9ef933d0cfa8f85484683b5ff6869ca00a25793a1b0146aeaeca45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-osd-11-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0) 2024-03-19T03:44:51.662 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@osd.11.service' 2024-03-19T03:44:51.743 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:44:51.743 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-19T03:44:51.743 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-19T03:44:51.743 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.a 2024-03-19T03:44:52.292 INFO:journalctl@ceph.mds.a.smithi117.stdout:Mar 19 03:44:51 smithi117 systemd[1]: Stopping Ceph mds.a for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:44:52.792 INFO:journalctl@ceph.mds.a.smithi117.stdout:Mar 19 03:44:52 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-a[73998]: 2024-03-19T03:44:52.338+0000 7fe172c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:44:52.792 INFO:journalctl@ceph.mds.a.smithi117.stdout:Mar 19 03:44:52 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-a[73998]: 2024-03-19T03:44:52.338+0000 7fe172c00700 -1 mds.a *** got signal Terminated *** 2024-03-19T03:44:54.292 INFO:journalctl@ceph.mds.a.smithi117.stdout:Mar 19 03:44:53 smithi117 podman[93658]: 2024-03-19 03:44:53.979117621 +0000 UTC m=+1.777534214 container stop 175e7bb211795cd8eff5e48e000853619e9cdadaaf1575e7cc494fe06a2e8e5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-19T03:44:54.292 INFO:journalctl@ceph.mds.a.smithi117.stdout:Mar 19 03:44:54 smithi117 podman[93658]: 2024-03-19 03:44:54.005573122 +0000 UTC m=+1.803989680 container died 175e7bb211795cd8eff5e48e000853619e9cdadaaf1575e7cc494fe06a2e8e5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-a, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-19T03:44:54.642 INFO:journalctl@ceph.mds.a.smithi117.stdout:Mar 19 03:44:54 smithi117 podman[93658]: 2024-03-19 03:44:54.457186066 +0000 UTC m=+2.255602652 container cleanup 175e7bb211795cd8eff5e48e000853619e9cdadaaf1575e7cc494fe06a2e8e5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-a, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-19T03:44:54.642 INFO:journalctl@ceph.mds.a.smithi117.stdout:Mar 19 03:44:54 smithi117 bash[93658]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-a 2024-03-19T03:44:55.041 INFO:journalctl@ceph.mds.a.smithi117.stdout:Mar 19 03:44:54 smithi117 podman[93672]: 2024-03-19 03:44:54.641533772 +0000 UTC m=+0.642981435 container remove 175e7bb211795cd8eff5e48e000853619e9cdadaaf1575e7cc494fe06a2e8e5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-a, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-19T03:44:55.117 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.a.service' 2024-03-19T03:44:55.234 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:44:55.234 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-19T03:44:55.234 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-19T03:44:55.234 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.d 2024-03-19T03:44:55.792 INFO:journalctl@ceph.mds.d.smithi117.stdout:Mar 19 03:44:55 smithi117 systemd[1]: Stopping Ceph mds.d for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:44:56.292 INFO:journalctl@ceph.mds.d.smithi117.stdout:Mar 19 03:44:55 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-d[73010]: 2024-03-19T03:44:55.824+0000 7fe52de00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:44:56.292 INFO:journalctl@ceph.mds.d.smithi117.stdout:Mar 19 03:44:55 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-d[73010]: 2024-03-19T03:44:55.824+0000 7fe52de00700 -1 mds.d *** got signal Terminated *** 2024-03-19T03:44:58.942 INFO:journalctl@ceph.mds.d.smithi117.stdout:Mar 19 03:44:58 smithi117 podman[93777]: 2024-03-19 03:44:58.67627549 +0000 UTC m=+2.976719880 container died 4dd4662cf2814d8597370f913f3d6ecd84044e393a4dbb0ad7de07b3650f7701 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-d, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-19T03:44:59.270 INFO:journalctl@ceph.mds.d.smithi117.stdout:Mar 19 03:44:59 smithi117 podman[93777]: 2024-03-19 03:44:59.051965948 +0000 UTC m=+3.352410328 container cleanup 4dd4662cf2814d8597370f913f3d6ecd84044e393a4dbb0ad7de07b3650f7701 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-d, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-19T03:44:59.270 INFO:journalctl@ceph.mds.d.smithi117.stdout:Mar 19 03:44:59 smithi117 bash[93777]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-d 2024-03-19T03:44:59.542 INFO:journalctl@ceph.mds.d.smithi117.stdout:Mar 19 03:44:59 smithi117 podman[93791]: 2024-03-19 03:44:59.269468672 +0000 UTC m=+0.577880139 container remove 4dd4662cf2814d8597370f913f3d6ecd84044e393a4dbb0ad7de07b3650f7701 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-19T03:44:59.768 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.d.service' 2024-03-19T03:44:59.884 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:44:59.885 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-19T03:44:59.885 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-19T03:44:59.885 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.g 2024-03-19T03:45:00.291 INFO:journalctl@ceph.mds.g.smithi117.stdout:Mar 19 03:45:00 smithi117 systemd[1]: Stopping Ceph mds.g for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:45:00.792 INFO:journalctl@ceph.mds.g.smithi117.stdout:Mar 19 03:45:00 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-g[73651]: 2024-03-19T03:45:00.393+0000 7f01f5200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:45:00.792 INFO:journalctl@ceph.mds.g.smithi117.stdout:Mar 19 03:45:00 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-g[73651]: 2024-03-19T03:45:00.393+0000 7f01f5200700 -1 mds.g *** got signal Terminated *** 2024-03-19T03:45:02.279 INFO:journalctl@ceph.mds.g.smithi117.stdout:Mar 19 03:45:01 smithi117 podman[93896]: 2024-03-19 03:45:01.993137484 +0000 UTC m=+1.724665637 container died 71a8940c83ddc18d09d7780673e86a2effda560544d1198457ea71fdaf8f566b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-g, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-19T03:45:02.541 INFO:journalctl@ceph.mds.g.smithi117.stdout:Mar 19 03:45:02 smithi117 podman[93896]: 2024-03-19 03:45:02.396710338 +0000 UTC m=+2.128238492 container cleanup 71a8940c83ddc18d09d7780673e86a2effda560544d1198457ea71fdaf8f566b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-g, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-19T03:45:02.542 INFO:journalctl@ceph.mds.g.smithi117.stdout:Mar 19 03:45:02 smithi117 bash[93896]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-g 2024-03-19T03:45:03.042 INFO:journalctl@ceph.mds.g.smithi117.stdout:Mar 19 03:45:02 smithi117 podman[93910]: 2024-03-19 03:45:02.655905092 +0000 UTC m=+0.647513724 container remove 71a8940c83ddc18d09d7780673e86a2effda560544d1198457ea71fdaf8f566b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-g, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-19T03:45:03.140 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.g.service' 2024-03-19T03:45:03.256 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:45:03.257 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-19T03:45:03.257 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-19T03:45:03.257 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.j 2024-03-19T03:45:03.773 INFO:journalctl@ceph.mds.j.smithi117.stdout:Mar 19 03:45:03 smithi117 systemd[1]: Stopping Ceph mds.j for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:45:04.042 INFO:journalctl@ceph.mds.j.smithi117.stdout:Mar 19 03:45:03 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-j[72665]: 2024-03-19T03:45:03.772+0000 7fc5df600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:45:04.042 INFO:journalctl@ceph.mds.j.smithi117.stdout:Mar 19 03:45:03 smithi117 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-j[72665]: 2024-03-19T03:45:03.772+0000 7fc5df600700 -1 mds.j *** got signal Terminated *** 2024-03-19T03:45:06.908 INFO:journalctl@ceph.mds.j.smithi117.stdout:Mar 19 03:45:06 smithi117 podman[94014]: 2024-03-19 03:45:06.65319782 +0000 UTC m=+3.000385798 container died 657e4491ff59e38c824874bf92b170d9ba9f7752c1f8e1fc453461ba0c5a4f2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-j, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-19T03:45:07.209 INFO:journalctl@ceph.mds.j.smithi117.stdout:Mar 19 03:45:07 smithi117 podman[94014]: 2024-03-19 03:45:07.007928984 +0000 UTC m=+3.355116995 container cleanup 657e4491ff59e38c824874bf92b170d9ba9f7752c1f8e1fc453461ba0c5a4f2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-j, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T03:45:07.209 INFO:journalctl@ceph.mds.j.smithi117.stdout:Mar 19 03:45:07 smithi117 bash[94014]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-j 2024-03-19T03:45:07.541 INFO:journalctl@ceph.mds.j.smithi117.stdout:Mar 19 03:45:07 smithi117 podman[94028]: 2024-03-19 03:45:07.208743805 +0000 UTC m=+0.540503175 container remove 657e4491ff59e38c824874bf92b170d9ba9f7752c1f8e1fc453461ba0c5a4f2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-j, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-19T03:45:07.671 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.j.service' 2024-03-19T03:45:07.788 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:45:07.789 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-19T03:45:07.789 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-19T03:45:07.789 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.b 2024-03-19T03:45:08.234 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 19 03:45:07 smithi119 systemd[1]: Stopping Ceph mds.b for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:45:08.234 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 19 03:45:08 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-b[97021]: 2024-03-19T03:45:08.066+0000 7fe9154bc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:45:08.234 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 19 03:45:08 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-b[97021]: 2024-03-19T03:45:08.066+0000 7fe9154bc700 -1 mds.b *** got signal Terminated *** 2024-03-19T03:45:10.210 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 19 03:45:09 smithi119 podman[101958]: 2024-03-19 03:45:09.943146205 +0000 UTC m=+1.964383809 container died de4c1844e91141545d56c0000d4caf0b17dbbe4314c83a196b3e14f4a8f8d1ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-19T03:45:10.484 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 19 03:45:10 smithi119 podman[101958]: 2024-03-19 03:45:10.325520177 +0000 UTC m=+2.346757763 container cleanup de4c1844e91141545d56c0000d4caf0b17dbbe4314c83a196b3e14f4a8f8d1ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-19T03:45:10.484 INFO:journalctl@ceph.mds.b.smithi119.stdout:Mar 19 03:45:10 smithi119 bash[101958]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-b 2024-03-19T03:45:10.712 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.b.service' 2024-03-19T03:45:10.751 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:45:10.752 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-19T03:45:10.752 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-19T03:45:10.752 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.e 2024-03-19T03:45:11.088 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 19 03:45:10 smithi119 systemd[1]: Stopping Ceph mds.e for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:45:11.484 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 19 03:45:11 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-e[96402]: 2024-03-19T03:45:11.087+0000 7f0f30ffd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:45:11.484 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 19 03:45:11 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-e[96402]: 2024-03-19T03:45:11.087+0000 7f0f30ffd700 -1 mds.e *** got signal Terminated *** 2024-03-19T03:45:14.196 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 19 03:45:13 smithi119 podman[102092]: 2024-03-19 03:45:13.944064406 +0000 UTC m=+2.961088003 container died 0a8ea9d7e2a159b06da19404d30704eeabaef44b4f412d6f24cef6389893e859 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-19T03:45:14.447 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 19 03:45:14 smithi119 podman[102092]: 2024-03-19 03:45:14.288028828 +0000 UTC m=+3.305052426 container cleanup 0a8ea9d7e2a159b06da19404d30704eeabaef44b4f412d6f24cef6389893e859 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-e, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-19T03:45:14.448 INFO:journalctl@ceph.mds.e.smithi119.stdout:Mar 19 03:45:14 smithi119 bash[102092]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-e 2024-03-19T03:45:14.660 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.e.service' 2024-03-19T03:45:14.697 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:45:14.697 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-19T03:45:14.697 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-19T03:45:14.697 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.h 2024-03-19T03:45:15.235 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 19 03:45:14 smithi119 systemd[1]: Stopping Ceph mds.h for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:45:15.235 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 19 03:45:14 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-h[96710]: 2024-03-19T03:45:14.991+0000 7fdffade9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:45:15.235 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 19 03:45:14 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-h[96710]: 2024-03-19T03:45:14.991+0000 7fdffade9700 -1 mds.h *** got signal Terminated *** 2024-03-19T03:45:18.201 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 19 03:45:17 smithi119 podman[102218]: 2024-03-19 03:45:17.937560124 +0000 UTC m=+3.029342619 container died f0cc29854e2d57f6a1359c42b62395592430fb77051ac76851893d36d2903c80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-h, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-19T03:45:18.478 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 19 03:45:18 smithi119 podman[102218]: 2024-03-19 03:45:18.291471619 +0000 UTC m=+3.383254107 container cleanup f0cc29854e2d57f6a1359c42b62395592430fb77051ac76851893d36d2903c80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-h, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:45:18.478 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 19 03:45:18 smithi119 bash[102218]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-h 2024-03-19T03:45:18.703 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.h.service' 2024-03-19T03:45:18.728 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 19 03:45:18 smithi119 podman[102232]: 2024-03-19 03:45:18.478184189 +0000 UTC m=+0.535464695 container remove f0cc29854e2d57f6a1359c42b62395592430fb77051ac76851893d36d2903c80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-h, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux ) 2024-03-19T03:45:18.728 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 19 03:45:18 smithi119 systemd[1]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.h.service: Deactivated successfully. 2024-03-19T03:45:18.728 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 19 03:45:18 smithi119 systemd[1]: Stopped Ceph mds.h for a3a82456-e59f-11ee-95c9-87774f69a715. 2024-03-19T03:45:18.729 INFO:journalctl@ceph.mds.h.smithi119.stdout:Mar 19 03:45:18 smithi119 systemd[1]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.h.service: Consumed 4min 16.960s CPU time. 2024-03-19T03:45:18.738 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:45:18.738 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-19T03:45:18.738 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-19T03:45:18.738 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.k 2024-03-19T03:45:18.984 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 19 03:45:18 smithi119 systemd[1]: Stopping Ceph mds.k for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:45:19.484 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 19 03:45:19 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-k[96080]: 2024-03-19T03:45:19.063+0000 7f30b6b6e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:45:19.484 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 19 03:45:19 smithi119 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-k[96080]: 2024-03-19T03:45:19.063+0000 7f30b6b6e700 -1 mds.k *** got signal Terminated *** 2024-03-19T03:45:22.234 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 19 03:45:21 smithi119 podman[102340]: 2024-03-19 03:45:21.938417959 +0000 UTC m=+2.982435028 container died 0ad918646f67f8c935f922f1c21663e460f4417478d6e8849188312df1a702d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-k, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-19T03:45:22.548 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 19 03:45:22 smithi119 podman[102340]: 2024-03-19 03:45:22.322638759 +0000 UTC m=+3.366655813 container cleanup 0ad918646f67f8c935f922f1c21663e460f4417478d6e8849188312df1a702d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-k, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:45:22.548 INFO:journalctl@ceph.mds.k.smithi119.stdout:Mar 19 03:45:22 smithi119 bash[102340]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-k 2024-03-19T03:45:22.662 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.k.service' 2024-03-19T03:45:22.697 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:45:22.697 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-19T03:45:22.697 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-19T03:45:22.697 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.c 2024-03-19T03:45:22.998 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 19 03:45:22 smithi178 systemd[1]: Stopping Ceph mds.c for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:45:23.345 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 19 03:45:22 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-c[98443]: 2024-03-19T03:45:22.997+0000 7f3afdee6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:45:23.345 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 19 03:45:22 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-c[98443]: 2024-03-19T03:45:22.997+0000 7f3afdee6700 -1 mds.c *** got signal Terminated *** 2024-03-19T03:45:26.222 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 19 03:45:25 smithi178 podman[104352]: 2024-03-19 03:45:25.951489756 +0000 UTC m=+3.043731993 container died 5baaacaf8c4aeb6619a24dcf956a5026d682b459fbbe14d9fb454598fc9cc514 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-c, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5) 2024-03-19T03:45:26.222 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 19 03:45:26 smithi178 podman[104352]: 2024-03-19 03:45:26.222180757 +0000 UTC m=+3.314422976 container cleanup 5baaacaf8c4aeb6619a24dcf956a5026d682b459fbbe14d9fb454598fc9cc514 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-19T03:45:26.489 INFO:journalctl@ceph.mds.c.smithi178.stdout:Mar 19 03:45:26 smithi178 bash[104352]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-c 2024-03-19T03:45:26.598 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.c.service' 2024-03-19T03:45:26.637 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:45:26.637 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-19T03:45:26.637 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-19T03:45:26.637 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.f 2024-03-19T03:45:26.845 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 19 03:45:26 smithi178 systemd[1]: Stopping Ceph mds.f for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:45:27.345 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 19 03:45:26 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-f[98131]: 2024-03-19T03:45:26.959+0000 7f6fcff28700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:45:27.346 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 19 03:45:26 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-f[98131]: 2024-03-19T03:45:26.959+0000 7f6fcff28700 -1 mds.f *** got signal Terminated *** 2024-03-19T03:45:30.217 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 19 03:45:29 smithi178 podman[104485]: 2024-03-19 03:45:29.958106759 +0000 UTC m=+3.098165246 container died 21115f9198fa374a76bfcf782607f6ccc4ab77ba39e5953e08ea9f62920b1b2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-f, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-19T03:45:30.484 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 19 03:45:30 smithi178 podman[104485]: 2024-03-19 03:45:30.29190532 +0000 UTC m=+3.431963799 container cleanup 21115f9198fa374a76bfcf782607f6ccc4ab77ba39e5953e08ea9f62920b1b2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-19T03:45:30.484 INFO:journalctl@ceph.mds.f.smithi178.stdout:Mar 19 03:45:30 smithi178 bash[104485]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-f 2024-03-19T03:45:30.707 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.f.service' 2024-03-19T03:45:30.741 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:45:30.741 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-19T03:45:30.741 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-19T03:45:30.742 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.i 2024-03-19T03:45:31.079 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 19 03:45:30 smithi178 systemd[1]: Stopping Ceph mds.i for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:45:31.345 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 19 03:45:31 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-i[98749]: 2024-03-19T03:45:31.077+0000 7ff99f5cc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:45:31.346 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 19 03:45:31 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-i[98749]: 2024-03-19T03:45:31.077+0000 7ff99f5cc700 -1 mds.i *** got signal Terminated *** 2024-03-19T03:45:34.321 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 19 03:45:33 smithi178 podman[104607]: 2024-03-19 03:45:33.947658272 +0000 UTC m=+2.993489909 container died 5ef48f5e4afc1de7477ca158fdbaa1b4bb598435a8bfe5d63bd1b2c95358ad65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-i, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:45:34.595 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 19 03:45:34 smithi178 podman[104607]: 2024-03-19 03:45:34.412301754 +0000 UTC m=+3.458133378 container cleanup 5ef48f5e4afc1de7477ca158fdbaa1b4bb598435a8bfe5d63bd1b2c95358ad65 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-i, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-19T03:45:34.595 INFO:journalctl@ceph.mds.i.smithi178.stdout:Mar 19 03:45:34 smithi178 bash[104607]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-i 2024-03-19T03:45:34.886 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.i.service' 2024-03-19T03:45:34.921 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:45:34.921 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-19T03:45:34.921 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-19T03:45:34.921 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.l 2024-03-19T03:45:35.241 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 19 03:45:35 smithi178 systemd[1]: Stopping Ceph mds.l for a3a82456-e59f-11ee-95c9-87774f69a715... 2024-03-19T03:45:35.595 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 19 03:45:35 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-l[97823]: 2024-03-19T03:45:35.239+0000 7f2c185dc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T03:45:35.596 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 19 03:45:35 smithi178 ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-l[97823]: 2024-03-19T03:45:35.239+0000 7f2c185dc700 -1 mds.l *** got signal Terminated *** 2024-03-19T03:45:38.299 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 19 03:45:37 smithi178 podman[104730]: 2024-03-19 03:45:37.943373238 +0000 UTC m=+2.810200606 container died 503c6609fe6d22ac2b098397c1cdaa9f979700f1a6467d2bf89979353f8f43be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-l, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-19T03:45:38.591 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 19 03:45:38 smithi178 podman[104730]: 2024-03-19 03:45:38.373944949 +0000 UTC m=+3.240772318 container cleanup 503c6609fe6d22ac2b098397c1cdaa9f979700f1a6467d2bf89979353f8f43be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-l, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T03:45:38.591 INFO:journalctl@ceph.mds.l.smithi178.stdout:Mar 19 03:45:38 smithi178 bash[104730]: ceph-a3a82456-e59f-11ee-95c9-87774f69a715-mds-l 2024-03-19T03:45:38.798 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a3a82456-e59f-11ee-95c9-87774f69a715@mds.l.service' 2024-03-19T03:45:38.832 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T03:45:38.832 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-19T03:45:38.832 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a3a82456-e59f-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-19T03:45:39.153 INFO:teuthology.orchestra.run.smithi117.stdout:Deleting cluster with fsid: a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:45:44.027 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a3a82456-e59f-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-19T03:45:44.218 INFO:teuthology.orchestra.run.smithi119.stdout:Deleting cluster with fsid: a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:45:46.782 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a3a82456-e59f-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-19T03:45:46.973 INFO:teuthology.orchestra.run.smithi178.stdout:Deleting cluster with fsid: a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:45:49.385 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-19T03:45:49.486 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-19T03:45:49.521 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-19T03:45:49.549 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-19T03:45:49.551 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609460/remote/smithi117/crash 2024-03-19T03:45:49.552 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/crash -- . 2024-03-19T03:45:49.645 INFO:teuthology.orchestra.run.smithi117.stderr:tar: /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-19T03:45:49.646 INFO:teuthology.orchestra.run.smithi117.stderr:tar: Error is not recoverable: exiting now 2024-03-19T03:45:49.650 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609460/remote/smithi119/crash 2024-03-19T03:45:49.651 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/crash -- . 2024-03-19T03:45:49.697 INFO:teuthology.orchestra.run.smithi119.stderr:tar: /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-19T03:45:49.698 INFO:teuthology.orchestra.run.smithi119.stderr:tar: Error is not recoverable: exiting now 2024-03-19T03:45:49.703 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609460/remote/smithi178/crash 2024-03-19T03:45:49.704 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/crash -- . 2024-03-19T03:45:49.742 INFO:teuthology.orchestra.run.smithi178.stderr:tar: /var/lib/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-19T03:45:49.742 INFO:teuthology.orchestra.run.smithi178.stderr:tar: Error is not recoverable: exiting now 2024-03-19T03:45:49.743 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-19T03:45:49.744 DEBUG:teuthology.orchestra.run.smithi117:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-19T03:45:49.846 INFO:teuthology.orchestra.run.smithi117.stdout:2024-03-19T03:23:46.094857+0000 mon.a (mon.0) 209 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-19T03:45:49.847 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-19T03:45:49.848 DEBUG:teuthology.orchestra.run.smithi117:> sudo egrep '\[SEC\]' /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-19T03:45:49.936 DEBUG:teuthology.orchestra.run.smithi117:> sudo egrep '\[ERR\]' /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-19T03:45:50.027 DEBUG:teuthology.orchestra.run.smithi117:> sudo egrep '\[WRN\]' /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-19T03:45:50.115 INFO:teuthology.orchestra.run.smithi117.stdout:2024-03-19T03:23:46.094857+0000 mon.a (mon.0) 209 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-19T03:45:50.116 INFO:tasks.cephadm:Compressing logs... 2024-03-19T03:45:50.116 DEBUG:teuthology.orchestra.run.smithi117:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-19T03:45:50.121 DEBUG:teuthology.orchestra.run.smithi119:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-19T03:45:50.124 DEBUG:teuthology.orchestra.run.smithi178:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-19T03:45:50.151 INFO:teuthology.orchestra.run.smithi178.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-19T03:45:50.151 INFO:teuthology.orchestra.run.smithi178.stderr:: No such file or directory 2024-03-19T03:45:50.152 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.l.log 2024-03-19T03:45:50.152 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-19T03:45:50.152 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.i.log 2024-03-19T03:45:50.153 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.11.log 2024-03-19T03:45:50.153 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mgr.z.log 2024-03-19T03:45:50.154 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.i.log: /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.8.log 2024-03-19T03:45:50.154 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.5.log 2024-03-19T03:45:50.154 INFO:teuthology.orchestra.run.smithi178.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-19T03:45:50.154 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.f.log 2024-03-19T03:45:50.154 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-19T03:45:50.156 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.f.log: 90.4% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-19T03:45:50.156 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.log 2024-03-19T03:45:50.156 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.cephadm.log: 82.1% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-19T03:45:50.156 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.2.log 2024-03-19T03:45:50.157 INFO:teuthology.orchestra.run.smithi119.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-19T03:45:50.159 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.log: 86.3% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-19T03:45:50.159 INFO:teuthology.orchestra.run.smithi178.stderr: 92.8% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mgr.z.log.gz 2024-03-19T03:45:50.159 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.c.log 2024-03-19T03:45:50.159 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-19T03:45:50.161 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-19T03:45:50.162 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.10.log 2024-03-19T03:45:50.162 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-19T03:45:50.163 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.b.log 2024-03-19T03:45:50.163 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.4.log 2024-03-19T03:45:50.164 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.audit.log: /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.1.log 2024-03-19T03:45:50.164 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.k.log 2024-03-19T03:45:50.166 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.h.log 2024-03-19T03:45:50.166 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.7.log 2024-03-19T03:45:50.167 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.e.log 2024-03-19T03:45:50.167 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mgr.y.log 2024-03-19T03:45:50.167 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-19T03:45:50.167 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.log 2024-03-19T03:45:50.167 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-19T03:45:50.167 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mon.b.log 2024-03-19T03:45:50.169 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mon.b.log: 92.7% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mgr.y.log.gz 2024-03-19T03:45:50.178 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mon.c.log 2024-03-19T03:45:50.182 INFO:teuthology.orchestra.run.smithi117.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-19T03:45:50.184 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-19T03:45:50.185 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-19T03:45:50.186 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.6.log 2024-03-19T03:45:50.187 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mon.a.log 2024-03-19T03:45:50.188 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-19T03:45:50.189 INFO:teuthology.orchestra.run.smithi117.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-19T03:45:50.189 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.g.log 2024-03-19T03:45:50.191 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.j.log 2024-03-19T03:45:50.192 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.3.log 2024-03-19T03:45:50.195 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.j.log: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-19T03:45:50.195 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.3.log: 90.2% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-19T03:45:50.196 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mgr.x.log 2024-03-19T03:45:50.198 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.0.log 2024-03-19T03:45:50.199 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mgr.x.log: 94.5% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.j.log.gz 2024-03-19T03:45:50.199 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-19T03:45:50.201 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mon.c.log: 91.0% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-19T03:45:50.203 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.0.log: /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.log 2024-03-19T03:45:50.203 INFO:teuthology.orchestra.run.smithi117.stderr: 82.7% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-19T03:45:50.204 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-19T03:45:50.209 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.log: /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-volume.log: 86.4% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-19T03:45:50.210 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.9.log 2024-03-19T03:45:50.212 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.d.log 2024-03-19T03:45:50.219 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.9.log: /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.a.log 2024-03-19T03:45:50.230 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.a.log: 94.5% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.d.log.gz 2024-03-19T03:45:50.243 INFO:teuthology.orchestra.run.smithi117.stderr: 90.9% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-19T03:45:50.393 INFO:teuthology.orchestra.run.smithi119.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-19T03:45:50.395 INFO:teuthology.orchestra.run.smithi119.stderr: 90.4% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-19T03:45:50.396 INFO:teuthology.orchestra.run.smithi119.stderr: 82.2% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-19T03:45:50.474 INFO:teuthology.orchestra.run.smithi178.stderr: 94.9% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.c.log.gz 2024-03-19T03:45:50.617 INFO:teuthology.orchestra.run.smithi117.stderr: 89.9% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mgr.x.log.gz 2024-03-19T03:45:50.651 INFO:teuthology.orchestra.run.smithi178.stderr: 95.0% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.f.log.gz 2024-03-19T03:45:50.669 INFO:teuthology.orchestra.run.smithi178.stderr: 95.1% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.l.log.gz 2024-03-19T03:45:50.672 INFO:teuthology.orchestra.run.smithi119.stderr: 86.3% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-19T03:45:50.684 INFO:teuthology.orchestra.run.smithi178.stderr: 91.9% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mon.c.log.gz 2024-03-19T03:45:50.765 INFO:teuthology.orchestra.run.smithi119.stderr: 90.9% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-19T03:45:50.918 INFO:teuthology.orchestra.run.smithi119.stderr: 91.9% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mon.b.log.gz 2024-03-19T03:45:51.406 INFO:teuthology.orchestra.run.smithi117.stderr: 90.4% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mon.a.log.gz 2024-03-19T03:45:52.039 INFO:teuthology.orchestra.run.smithi119.stderr: 95.0% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.e.log.gz 2024-03-19T03:45:54.156 INFO:teuthology.orchestra.run.smithi117.stderr: 94.2% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.a.log.gz 2024-03-19T03:45:57.130 INFO:teuthology.orchestra.run.smithi117.stderr: 94.2% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.g.log.gz 2024-03-19T03:46:03.352 INFO:teuthology.orchestra.run.smithi178.stderr: 94.9% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.i.log.gz 2024-03-19T03:46:04.456 INFO:teuthology.orchestra.run.smithi178.stderr: 91.6% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.8.log.gz 2024-03-19T03:46:05.989 INFO:teuthology.orchestra.run.smithi178.stderr: 91.6% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.11.log.gz 2024-03-19T03:46:08.758 INFO:teuthology.orchestra.run.smithi117.stderr: 91.4% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.0.log.gz 2024-03-19T03:46:09.556 INFO:teuthology.orchestra.run.smithi119.stderr: 94.3% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.h.log.gz 2024-03-19T03:46:10.574 INFO:teuthology.orchestra.run.smithi117.stderr: 91.4% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.3.log.gz 2024-03-19T03:46:10.641 INFO:teuthology.orchestra.run.smithi117.stderr: 91.7% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.9.log.gz 2024-03-19T03:46:10.881 INFO:teuthology.orchestra.run.smithi117.stderr: 91.6% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.6.log.gz 2024-03-19T03:46:10.884 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-19T03:46:10.884 INFO:teuthology.orchestra.run.smithi117.stderr:real 0m20.733s 2024-03-19T03:46:10.885 INFO:teuthology.orchestra.run.smithi117.stderr:user 1m25.866s 2024-03-19T03:46:10.885 INFO:teuthology.orchestra.run.smithi117.stderr:sys 0m6.304s 2024-03-19T03:46:12.723 INFO:teuthology.orchestra.run.smithi178.stderr: 91.4% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.2.log.gz 2024-03-19T03:46:13.209 INFO:teuthology.orchestra.run.smithi119.stderr: 94.2% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.k.log.gz 2024-03-19T03:46:17.872 INFO:teuthology.orchestra.run.smithi178.stderr: 91.4% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.5.log.gz 2024-03-19T03:46:17.874 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-03-19T03:46:17.874 INFO:teuthology.orchestra.run.smithi178.stderr:real 0m27.736s 2024-03-19T03:46:17.874 INFO:teuthology.orchestra.run.smithi178.stderr:user 1m32.132s 2024-03-19T03:46:17.874 INFO:teuthology.orchestra.run.smithi178.stderr:sys 0m2.883s 2024-03-19T03:46:23.501 INFO:teuthology.orchestra.run.smithi119.stderr: 91.6% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.10.log.gz 2024-03-19T03:46:25.286 INFO:teuthology.orchestra.run.smithi119.stderr: 91.8% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.1.log.gz 2024-03-19T03:46:28.881 INFO:teuthology.orchestra.run.smithi119.stderr: 91.6% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.7.log.gz 2024-03-19T03:46:41.006 INFO:teuthology.orchestra.run.smithi119.stderr: 91.5% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-osd.4.log.gz 2024-03-19T03:49:30.377 INFO:teuthology.orchestra.run.smithi119.stderr: 93.2% -- replaced with /var/log/ceph/a3a82456-e59f-11ee-95c9-87774f69a715/ceph-mds.b.log.gz 2024-03-19T03:49:30.378 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-19T03:49:30.378 INFO:teuthology.orchestra.run.smithi119.stderr:real 3m40.240s 2024-03-19T03:49:30.378 INFO:teuthology.orchestra.run.smithi119.stderr:user 4m17.042s 2024-03-19T03:49:30.378 INFO:teuthology.orchestra.run.smithi119.stderr:sys 0m8.840s 2024-03-19T03:49:30.379 INFO:tasks.cephadm:Archiving logs... 2024-03-19T03:49:30.379 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609460/remote/smithi117/log 2024-03-19T03:49:30.380 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-19T03:49:43.602 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609460/remote/smithi119/log 2024-03-19T03:49:43.603 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-19T03:50:25.701 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609460/remote/smithi178/log 2024-03-19T03:50:25.702 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-19T03:50:40.041 INFO:tasks.cephadm:Removing cluster... 2024-03-19T03:50:40.042 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a3a82456-e59f-11ee-95c9-87774f69a715 --force 2024-03-19T03:50:40.376 INFO:teuthology.orchestra.run.smithi117.stdout:Deleting cluster with fsid: a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:50:41.297 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a3a82456-e59f-11ee-95c9-87774f69a715 --force 2024-03-19T03:50:41.491 INFO:teuthology.orchestra.run.smithi119.stdout:Deleting cluster with fsid: a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:50:42.034 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a3a82456-e59f-11ee-95c9-87774f69a715 --force 2024-03-19T03:50:42.224 INFO:teuthology.orchestra.run.smithi178.stdout:Deleting cluster with fsid: a3a82456-e59f-11ee-95c9-87774f69a715 2024-03-19T03:50:42.681 INFO:tasks.cephadm:Removing cephadm ... 2024-03-19T03:50:42.682 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-19T03:50:42.740 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-19T03:50:42.758 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-19T03:50:42.773 INFO:tasks.cephadm:Teardown complete 2024-03-19T03:50:42.773 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-19T03:50:42.783 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-19T03:50:42.783 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-19T03:50:42.789 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-19T03:50:42.800 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-19T03:50:42.942 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-19T03:50:42.951 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-19T03:50:42.952 DEBUG:teuthology.orchestra.run.smithi117:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T03:50:42.960 DEBUG:teuthology.orchestra.run.smithi119:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T03:50:42.962 DEBUG:teuthology.orchestra.run.smithi178:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T03:50:42.975 INFO:teuthology.orchestra.run.smithi119.stderr:bash: line 1: ntpq: command not found 2024-03-19T03:50:42.975 INFO:teuthology.orchestra.run.smithi178.stderr:bash: line 1: ntpq: command not found 2024-03-19T03:50:42.980 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T03:50:42.980 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2024-03-19T03:50:42.980 INFO:teuthology.orchestra.run.smithi178.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 36 +3589us[+3589us] +/- 99ms 2024-03-19T03:50:42.980 INFO:teuthology.orchestra.run.smithi178.stdout:^* hv02.front.sepia.ceph.com 3 6 377 36 -1538us[-1514us] +/- 62ms 2024-03-19T03:50:42.980 INFO:teuthology.orchestra.run.smithi178.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 37 -651us[ -627us] +/- 65ms 2024-03-19T03:50:42.980 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T03:50:43.008 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-03-19T03:50:43.018 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T03:50:43.018 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-03-19T03:50:43.018 INFO:teuthology.orchestra.run.smithi117.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 40 +3662us[+3729us] +/- 99ms 2024-03-19T03:50:43.018 INFO:teuthology.orchestra.run.smithi117.stdout:^* hv02.front.sepia.ceph.com 3 6 377 39 -1518us[-1451us] +/- 62ms 2024-03-19T03:50:43.018 INFO:teuthology.orchestra.run.smithi117.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 39 -648us[ -648us] +/- 65ms 2024-03-19T03:50:43.018 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T03:50:43.030 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T03:50:43.030 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-03-19T03:50:43.030 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 38 +3589us[+3593us] +/- 99ms 2024-03-19T03:50:43.030 INFO:teuthology.orchestra.run.smithi119.stdout:^* hv02.front.sepia.ceph.com 3 6 377 37 -1538us[-1534us] +/- 62ms 2024-03-19T03:50:43.030 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 -657us[ -653us] +/- 65ms 2024-03-19T03:50:43.030 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T03:50:43.036 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-19T03:50:43.060 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-19T03:50:43.061 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-19T03:50:43.071 DEBUG:teuthology.orchestra.run.smithi117:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-19T03:50:43.306 DEBUG:teuthology.orchestra.run.smithi119:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-19T03:50:43.401 DEBUG:teuthology.orchestra.run.smithi178:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-19T03:50:43.472 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T03:50:43.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T03:50:43.579 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T03:50:43.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T03:50:43.613 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T03:50:43.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T03:50:43.643 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-19T03:50:43.653 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-19T03:50:43.665 INFO:teuthology.task.internal:Duration was 3440.799023 seconds 2024-03-19T03:50:43.665 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-19T03:50:43.676 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-19T03:50:43.676 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-19T03:50:43.685 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-19T03:50:43.688 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-19T03:50:43.731 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T03:50:43.759 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T03:50:43.834 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T03:50:44.126 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-19T03:50:44.127 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi117.front.sepia.ceph.com 2024-03-19T03:50:44.127 DEBUG:teuthology.orchestra.run.smithi117:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-19T03:50:44.195 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi119.front.sepia.ceph.com 2024-03-19T03:50:44.195 DEBUG:teuthology.orchestra.run.smithi119:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-19T03:50:44.228 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi178.front.sepia.ceph.com 2024-03-19T03:50:44.228 DEBUG:teuthology.orchestra.run.smithi178:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-19T03:50:44.250 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-19T03:50:44.250 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-19T03:50:44.257 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-19T03:50:44.271 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-19T03:50:44.351 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-19T03:50:44.352 DEBUG:teuthology.orchestra.run.smithi117:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-19T03:50:44.357 DEBUG:teuthology.orchestra.run.smithi119:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-19T03:50:44.359 DEBUG:teuthology.orchestra.run.smithi178:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-19T03:50:44.867 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-19T03:50:44.879 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-19T03:50:44.879 DEBUG:teuthology.orchestra.run.smithi117:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-19T03:50:44.970 DEBUG:teuthology.orchestra.run.smithi119:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-19T03:50:45.012 DEBUG:teuthology.orchestra.run.smithi178:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-19T03:50:45.040 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-19T03:50:45.053 DEBUG:teuthology.orchestra.run.smithi117:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-19T03:50:45.061 DEBUG:teuthology.orchestra.run.smithi119:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-19T03:50:45.063 DEBUG:teuthology.orchestra.run.smithi178:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-19T03:50:45.096 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = core 2024-03-19T03:50:45.106 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = core 2024-03-19T03:50:45.162 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = core 2024-03-19T03:50:45.214 DEBUG:teuthology.orchestra.run.smithi117:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-19T03:50:45.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T03:50:45.262 DEBUG:teuthology.orchestra.run.smithi119:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-19T03:50:45.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T03:50:45.277 DEBUG:teuthology.orchestra.run.smithi178:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-19T03:50:45.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T03:50:45.291 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-19T03:50:45.302 INFO:teuthology.task.internal:Transferring archived files... 2024-03-19T03:50:45.303 DEBUG:teuthology.misc:Transferring archived files from smithi117:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609460/remote/smithi117 2024-03-19T03:50:45.303 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-19T03:50:45.420 DEBUG:teuthology.misc:Transferring archived files from smithi119:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609460/remote/smithi119 2024-03-19T03:50:45.420 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-19T03:50:45.458 DEBUG:teuthology.misc:Transferring archived files from smithi178:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609460/remote/smithi178 2024-03-19T03:50:45.459 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-19T03:50:45.497 INFO:teuthology.task.internal:Removing archive directory... 2024-03-19T03:50:45.497 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-19T03:50:45.505 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-19T03:50:45.507 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-19T03:50:45.555 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-19T03:50:45.569 INFO:teuthology.task.internal:Not uploading archives. 2024-03-19T03:50:45.569 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-19T03:50:45.579 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-19T03:50:45.580 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-19T03:50:45.589 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-19T03:50:45.592 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-19T03:50:45.604 INFO:teuthology.orchestra.run.smithi119.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 19 03:50 /home/ubuntu/cephtest 2024-03-19T03:50:45.606 INFO:teuthology.orchestra.run.smithi178.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 19 03:50 /home/ubuntu/cephtest 2024-03-19T03:50:45.636 INFO:teuthology.orchestra.run.smithi117.stdout: 393249 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 19 03:50 /home/ubuntu/cephtest 2024-03-19T03:50:45.639 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-19T03:50:45.651 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-19T03:50:45.673 INFO:teuthology.nuke:Checking targets against current locks 2024-03-19T03:50:45.702 DEBUG:teuthology.nuke:shortname: smithi117 2024-03-19T03:50:45.702 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T03:50:45.726 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609460', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 02:36:25.658961', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJxM4ZQBmQKCjhr6XygHCic/wYzpzLj5NY6vPMZ159xdqkwGnJimG6q1LiQvc24Zm2XuqsGiqWQyEzk0c6baW/A='} 2024-03-19T03:50:45.730 DEBUG:teuthology.nuke:shortname: smithi119 2024-03-19T03:50:45.731 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T03:50:45.755 DEBUG:teuthology.nuke:shortname: smithi178 2024-03-19T03:50:45.755 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T03:50:45.764 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609460', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 02:36:25.657241', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCKrJY4oZavoFwutvIcgDonTGOpMinb+rD5M9GAzejj7pUNZs7ElO/bXEtR/QMgOFupenJhSmcRDkF7Q3n/9TPw='} 2024-03-19T03:50:45.785 INFO:teuthology.orchestra.console.smithi117:Power off 2024-03-19T03:50:45.786 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-19T03:50:45.802 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609460', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 02:36:25.660491', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCb7+ZVOTQgtPYhzbO/uemTtpEbApH0EnJo1LXCybwiksC0fg3jmBvjKY3oIJnGw9Gb9vS5VMe60EsmO0o5tEYQ='} 2024-03-19T03:50:45.814 DEBUG:teuthology.orchestra.console.smithi117:power off output: Chassis Power Control: Down/Off 2024-03-19T03:50:45.814 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:50:45.840 INFO:teuthology.orchestra.console.smithi119:Power off 2024-03-19T03:50:45.841 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-19T03:50:45.852 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-03-19T03:50:45.865 DEBUG:teuthology.orchestra.console.smithi119:power off output: Chassis Power Control: Down/Off 2024-03-19T03:50:45.865 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:50:45.882 INFO:teuthology.orchestra.console.smithi178:Power off 2024-03-19T03:50:45.882 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-19T03:50:45.893 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-03-19T03:50:45.906 DEBUG:teuthology.orchestra.console.smithi178:power off output: Chassis Power Control: Down/Off 2024-03-19T03:50:45.907 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:50:45.928 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on 2024-03-19T03:50:49.853 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:50:49.895 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:50:49.929 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:50:49.967 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-03-19T03:50:50.010 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-03-19T03:50:50.041 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on 2024-03-19T03:50:53.970 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:50:54.011 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:50:54.042 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:50:54.080 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-03-19T03:50:54.122 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-03-19T03:50:54.151 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is off 2024-03-19T03:50:54.252 INFO:teuthology.orchestra.console.smithi178:Power off completed 2024-03-19T03:50:58.085 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:50:58.122 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T03:50:58.196 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is off 2024-03-19T03:50:58.232 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is off 2024-03-19T03:50:58.297 INFO:teuthology.orchestra.console.smithi117:Power off completed 2024-03-19T03:50:58.333 INFO:teuthology.orchestra.console.smithi119:Power off completed 2024-03-19T03:50:58.438 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 908b6b276e8f14e16032b32a9770077cd37b6bd4 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} duration: 3440.7990231513977 failure_reason: '"2024-03-19T03:23:46.094857+0000 mon.a (mon.0) 209 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log' flavor: default owner: scheduled_pdonnell@teuthology success: false 2024-03-19T03:50:58.439 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-19T03:50:58.520 INFO:teuthology.run:FAIL